# What Ingest survives, flags and refuses

The honesty exhibit: what Ingest reads, what it admits it guessed at, and
what it will not read at all — measured on the golden corpus rather than
described. Every number below is computed from the corpus manifests and the
pinned oracle outputs by `tests/unit/test_honesty_exhibit.py`, which
re-renders this page and fails the build if a single figure has moved.
Nothing here is typed by hand, and nothing here is a target: it is a
measurement of what this build does with the ugliest files we could find.

<!-- Generated by checks/honesty_exhibit.py from corpus/manifest.d/*.yaml, corpus/expected/ and schema/risk-object.v0.schema.json. Do not edit by hand: run `python3 checks/honesty_exhibit.py --write`. -->

## The files this is measured on

The corpus is `corpus/`, declared in `corpus/manifest.d/*.yaml`: 16
real-world-cursed files, append-only by specification — files and their
expected outputs are added, never weakened or removed. It carries 138
declared rows across the 11 entries that state a row count; 5 state none.
The manifests name 49 curse tags on 11 entries (none of them repeated), and
5 further entries describe their curses in a prose note instead.

| Format | Files |
|---|---|
| `csv` | 8 |
| `(not declared)` | 5 |
| `fixed-width` | 1 |
| `jsonl` | 1 |
| `pdftotext-layout` | 1 |

| Source encoding | Files |
|---|---|
| `utf-8` | 10 |
| `cp1252` | 2 |
| `ascii` | 1 |
| `iso-8859-15` | 1 |
| `utf-16` | 1 |
| `utf-8-sig` | 1 |

## What it survives

13 of the 16 files ingest to canonical risk objects — 64 objects in all —
and those objects are pinned byte for byte under `corpus/expected/`. The
oracle runs on every merge and 100% is the only passing grade, so this table
is what the build enforces, not what it hopes for. It lists the whole
corpus, refused files included, with the outcome the oracle pins for each.

| File | Format | Encoding | Rows | Curses | Outcome |
|---|---|---|---|---|---|
| `Harburn_Group_SOV_2026_FINAL_v3 (2).csv` | `csv` | `utf-8` | 14 | 4 | read |
| `Pennard & Vale Estates - Loc Schedule FY26 (broker copy).csv` | `csv` | `iso-8859-15` | 12 | 4 | read |
| `Copy of Meridian RE SOV v2 FINAL FINAL.csv` | `csv` | `utf-8` | 16 | 4 | read |
| `SOV_combined FINAL (use this one) v7.xlsx.csv` | `csv` | `utf-8` | 16 | 5 | read |
| `Ashworth Portfolio – Occupancy Schedule 2026 (FINAL).csv` | `csv` | `cp1252` | 13 | 4 | read |
| `Prem Bdx Q2 2026 FINAL (2).csv` | `(not declared)` | `utf-8` | — | 1 (prose) | read |
| `Renewals 2026 v3 - Copy.csv` | `(not declared)` | `utf-8` | — | 1 (prose) | read |
| `boderaux_export_EU_v3.csv` | `(not declared)` | `cp1252` | — | 1 (prose) | read |
| `PREMWRITTEN_FY2025_legacy_extract.txt` | `(not declared)` | `utf-16` | — | 1 (prose) | refused |
| `program_summary_2026_clean.csv` | `(not declared)` | `utf-8` | — | 1 (prose) | read |
| `Drought+Weather Station Trigger Sched 2026 (AB edits) v4.csv` | `csv` | `utf-8` | 12 | 4 | read |
| `outage exposure list - cloud+grid MERGED final2.csv` | `csv` | `utf-8` | 10 | 4 | read |
| `EQPORT26_MAINFRAME.PRN` | `fixed-width` | `ascii` | 14 | 5 | refused |
| `eq_locs dump 2026-07-14T0312 prod.jsonl` | `jsonl` | `utf-8` | 9 | 4 | read |
| `NamedStorm_CatCircle_Schedule_FY26.pdf.txt` | `pdftotext-layout` | `utf-8` | 9 | 6 | refused |
| `flood_wildfire footprint expo (merged) - Sheet1.csv` | `csv` | `utf-8-sig` | 13 | 5 | read |

The curses each file declares, by name — the manifests' own words, which is
why they read as a bug list rather than a brochure:

- `Harburn_Group_SOV_2026_FINAL_v3 (2).csv` — `merged-two-row-header`,
  `units-banner-000s-spanning-three-columns`, `preamble-title-rows`,
  `zero-value-rows`
- `Pennard & Vale Estates - Loc Schedule FY26 (broker copy).csv` —
  `non-utf8-latin9-encoding`, `in-cell-currency-symbols-mixed-per-row`,
  `thousands-separator-styles-mixed-in-one-column`, `no-currency-column`
- `Copy of Meridian RE SOV v2 FINAL FINAL.csv` —
  `mixed-decimal-and-dms-coordinates`, `empty-coords-rows`,
  `geocode-instruction-column`
- `SOV_combined FINAL (use this one) v7.xlsx.csv` —
  `duplicated-header-mid-file`, `transposed-columns-in-pasted-block`,
  `total-row-not-a-location`, `trailing-footnote-lines`,
  `deceptive-double-extension`
- `Ashworth Portfolio – Occupancy Schedule 2026 (FINAL).csv` —
  `cp1252-typographic-punctuation`,
  `occupancy-codes-mixed-atc-air-freetext`, `boolean-vocabulary-zoo`,
  `blank-means-unknown`
- `Drought+Weather Station Trigger Sched 2026 (AB edits) v4.csv` —
  `station-ids-mixed-wmo-icao-plain-names`,
  `duplicate-station-under-two-id-schemes`,
  `elevation-unit-embedded-in-cell-ft-and-m`, `trigger-thresholds-in-prose`
- `outage exposure list - cloud+grid MERGED final2.csv` —
  `zone-column-mixes-cloud-az-region-and-grid-zones`,
  `exposure-column-mw-vs-monthly-spend-by-row-class`,
  `asterisk-footnotes-scoped-to-some-rows`, `abbreviated-magnitude-1.2m`
- `EQPORT26_MAINFRAME.PRN` — `no-delimiter-positional-columns`,
  `no-header-row`, `soil-column-mixes-vs30-numbers-and-nehrp-letters`,
  `cresta-present-for-some-rows-only`, `zero-padded-tiv`
- `eq_locs dump 2026-07-14T0312 prod.jsonl` —
  `field-names-inconsistent-across-lines`,
  `one-line-invalid-json-trailing-comma-must-flag`,
  `numbers-as-currency-strings`, `lat-lon-as-strings-some-lines`
- `NamedStorm_CatCircle_Schedule_FY26.pdf.txt` — `ragged-column-alignment`,
  `wrapped-cells-continue-next-line`, `form-feed-page-breaks`,
  `repeated-headers-mid-file`, `page-number-lines`, `dash-means-zero`
- `flood_wildfire footprint expo (merged) - Sheet1.csv` —
  `bom-prefixed-header`,
  `geometry-column-mixes-wkt-latlon-pair-postcode-only`,
  `wkt-lonlat-vs-pair-latlon-axis-order`,
  `peril-vocabulary-zoo-fld-flood-fluvial-scs`, `misfiled-scs-rows`

The remaining 5 entries state their curses as prose in the manifest rather
than as tags; they are counted above but not listed here, because a sentence
and a tag are not the same unit.

Not printed above: 1 of the tags counted earlier, on 1 of the files. They
name a punctuation character whose own name is one of the words the
perimeter scan will not let a product page carry (SPEC §4, P2). The
manifests hold them in full, and this sentence is here so the arithmetic
still reconciles.

## What it flags into `source_fidelity`

What Ingest could not confidently normalise goes on the object it produced,
in `exposure.source_fidelity`. The published schema makes all 3 of its lists
mandatory — `unmapped_columns`, `guessed_units`, `ambiguous_rows` — so
"nothing was guessed" has to be stated as an empty list rather than inferred
from a missing key. Nothing is silently dropped, and nothing is silently
repaired.

Across the 13 files it reads, this build records 103 distinct honesty
entries; every one of those files carries at least one. Entries are counted
once per file: a file that ingests to several objects repeats its file-level
notes on each of them, and counting the repetitions would flatter the
exhibit.

| File | Objects | `unmapped_columns` | `guessed_units` | `ambiguous_rows` |
|---|---|---|---|---|
| `Harburn_Group_SOV_2026_FINAL_v3 (2).csv` | 1 | 0 | 3 | 0 |
| `Pennard & Vale Estates - Loc Schedule FY26 (broker copy).csv` | 1 | 1 | 2 | 1 |
| `Copy of Meridian RE SOV v2 FINAL FINAL.csv` | 1 | 1 | 1 | 0 |
| `SOV_combined FINAL (use this one) v7.xlsx.csv` | 1 | 1 | 1 | 12 |
| `Ashworth Portfolio – Occupancy Schedule 2026 (FINAL).csv` | 1 | 2 | 2 | 1 |
| `Prem Bdx Q2 2026 FINAL (2).csv` | 12 | 6 | 0 | 5 |
| `Renewals 2026 v3 - Copy.csv` | 12 | 4 | 3 | 4 |
| `boderaux_export_EU_v3.csv` | 10 | 2 | 4 | 1 |
| `program_summary_2026_clean.csv` | 10 | 4 | 0 | 5 |
| `Drought+Weather Station Trigger Sched 2026 (AB edits) v4.csv` | 1 | 5 | 0 | 0 |
| `outage exposure list - cloud+grid MERGED final2.csv` | 10 | 4 | 10 | 6 |
| `eq_locs dump 2026-07-14T0312 prod.jsonl` | 1 | 3 | 2 | 1 |
| `flood_wildfire footprint expo (merged) - Sheet1.csv` | 3 | 1 | 1 | 4 |
| **All 13** | 64 | 34 | 29 | 40 |

## What it refuses

3 of the 16 files are refused outright, and the refusal is pinned in the
same oracle as every accepted output: a `.rejected.json` beside the expected
outputs, carrying the verdict and the reason. A refusal is an answer, not a
gap — it is what stops a later version quietly starting to guess.

- `PREMWRITTEN_FY2025_legacy_extract.txt` — refused, scope
  `out_of_scope_v0`, pinned in
  `corpus/expected/PREMWRITTEN_FY2025_legacy_extract.txt.rejected.json`. The
  reason, verbatim: the header names 12 months as columns: this is a wide
  period matrix (one column per month), not a schedule of risks. Each cell
  is a written-premium figure, which the v0 risk object has no field for —
  the only premium it carries is a technical price produced by Price (SPEC
  §3.3) — so Ingest v0 refuses it rather than inventing a shape for it
- `EQPORT26_MAINFRAME.PRN` — refused, scope `out_of_scope_v0`, pinned in
  `corpus/expected/EQPORT26_MAINFRAME.PRN.rejected.json`. The reason,
  verbatim: no delimiter (comma, semicolon, tab or pipe) occurs anywhere in
  the file: this is a fixed-width record format, whose column boundaries are
  external knowledge (a record layout), not something the bytes state.
  Ingest v0 reads delimited text and JSON Lines only, and guessing the
  boundaries would shred every row
- `NamedStorm_CatCircle_Schedule_FY26.pdf.txt` — refused, scope
  `out_of_scope_v0`, pinned in
  `corpus/expected/NamedStorm_CatCircle_Schedule_FY26.pdf.txt.rejected.json`.
  The reason, verbatim: the file contains form-feed page breaks: it is a
  paginated text dump of a printed document (pdftotext -layout and friends),
  where column alignment is whitespace and cells wrap onto continuation
  lines. Ingest v0 reads delimited text and JSON Lines only, and will not
  reconstruct a table from a page layout

Refusal also happens below the file: a record Ingest cannot parse is
reported, never dropped and never repaired into a total. The corpus pins
that case too — one JSON Lines file carries a malformed record, and its
expected output carries the object built from the rest of the file with the
bad record declared in `ambiguous_rows`.

What Ingest never does is decide whether a risk is worth taking. It
classifies data; a file it reads is not a risk it has accepted, and a file
it refuses is not a risk it has declined (SPEC §3.1).

## Where each number comes from

- The file list, the formats, the encodings, the row counts and the curse
  counts are read from `corpus/manifest.d/*.yaml`. Every other figure is
  computed over that same list, so a manifest edit moves this page or fails
  the build.
- Whether a file is read or refused is not stated here: it is which of
  `<file>.expected.json` and `<file>.rejected.json` is committed under
  `corpus/expected/`.
- The honesty-entry counts are read out of those expected outputs, and the
  refusal reasons are copied from the rejected ones byte for byte.
- The 3 mandatory lists come from `schema/risk-object.v0.schema.json`, the
  published schema, which you can fetch without an account.
- `POST /ingest` is the route behind all of it;
  [the quickstart](quickstart.md) shows it end to end.