This page as Markdown, byte for byte: /ingest-honesty.md

What Ingest survives, flags and refuses#

The honesty exhibit: what Ingest reads, what it admits it guessed at, and what it will not read at all — measured on the golden corpus rather than described. Every number below is computed from the corpus manifests and the pinned oracle outputs by tests/unit/test_honesty_exhibit.py, which re-renders this page and fails the build if a single figure has moved. Nothing here is typed by hand, and nothing here is a target: it is a measurement of what this build does with the ugliest files we could find.

The files this is measured on#

The corpus is corpus/, declared in corpus/manifest.d/*.yaml: 16 real-world-cursed files, append-only by specification — files and their expected outputs are added, never weakened or removed. It carries 138 declared rows across the 11 entries that state a row count; 5 state none. The manifests name 49 curse tags on 11 entries (none of them repeated), and 5 further entries describe their curses in a prose note instead.

What it survives#

13 of the 16 files ingest to canonical risk objects — 64 objects in all — and those objects are pinned byte for byte under corpus/expected/. The oracle runs on every merge and 100% is the only passing grade, so this table is what the build enforces, not what it hopes for. It lists the whole corpus, refused files included, with the outcome the oracle pins for each.

The curses each file declares, by name — the manifests' own words, which is why they read as a bug list rather than a brochure:

• Harburn_Group_SOV_2026_FINAL_v3 (2).csv — merged-two-row-header, units-banner-000s-spanning-three-columns, preamble-title-rows, zero-value-rows
• Pennard & Vale Estates - Loc Schedule FY26 (broker copy).csv — non-utf8-latin9-encoding, in-cell-currency-symbols-mixed-per-row, thousands-separator-styles-mixed-in-one-column, no-currency-column
• Copy of Meridian RE SOV v2 FINAL FINAL.csv — mixed-decimal-and-dms-coordinates, empty-coords-rows, geocode-instruction-column
• SOV_combined FINAL (use this one) v7.xlsx.csv — duplicated-header-mid-file, transposed-columns-in-pasted-block, total-row-not-a-location, trailing-footnote-lines, deceptive-double-extension
• Ashworth Portfolio – Occupancy Schedule 2026 (FINAL).csv — cp1252-typographic-punctuation, occupancy-codes-mixed-atc-air-freetext, boolean-vocabulary-zoo, blank-means-unknown
• Drought+Weather Station Trigger Sched 2026 (AB edits) v4.csv — station-ids-mixed-wmo-icao-plain-names, duplicate-station-under-two-id-schemes, elevation-unit-embedded-in-cell-ft-and-m, trigger-thresholds-in-prose
• outage exposure list - cloud+grid MERGED final2.csv — zone-column-mixes-cloud-az-region-and-grid-zones, exposure-column-mw-vs-monthly-spend-by-row-class, asterisk-footnotes-scoped-to-some-rows, abbreviated-magnitude-1.2m
• EQPORT26_MAINFRAME.PRN — no-delimiter-positional-columns, no-header-row, soil-column-mixes-vs30-numbers-and-nehrp-letters, cresta-present-for-some-rows-only, zero-padded-tiv
• eq_locs dump 2026-07-14T0312 prod.jsonl — field-names-inconsistent-across-lines, one-line-invalid-json-trailing-comma-must-flag, numbers-as-currency-strings, lat-lon-as-strings-some-lines
• NamedStorm_CatCircle_Schedule_FY26.pdf.txt — ragged-column-alignment, wrapped-cells-continue-next-line, form-feed-page-breaks, repeated-headers-mid-file, page-number-lines, dash-means-zero
• flood_wildfire footprint expo (merged) - Sheet1.csv — bom-prefixed-header, geometry-column-mixes-wkt-latlon-pair-postcode-only, wkt-lonlat-vs-pair-latlon-axis-order, peril-vocabulary-zoo-fld-flood-fluvial-scs, misfiled-scs-rows

The remaining 5 entries state their curses as prose in the manifest rather than as tags; they are counted above but not listed here, because a sentence and a tag are not the same unit.

Not printed above: 1 of the tags counted earlier, on 1 of the files. They name a punctuation character whose own name is one of the words the perimeter scan will not let a product page carry (SPEC §4, P2). The manifests hold them in full, and this sentence is here so the arithmetic still reconciles.

What it flags into source_fidelity#

What Ingest could not confidently normalise goes on the object it produced, in exposure.source_fidelity. The published schema makes all 3 of its lists mandatory — unmapped_columns, guessed_units, ambiguous_rows — so "nothing was guessed" has to be stated as an empty list rather than inferred from a missing key. Nothing is silently dropped, and nothing is silently repaired.

Across the 13 files it reads, this build records 103 distinct honesty entries; every one of those files carries at least one. Entries are counted once per file: a file that ingests to several objects repeats its file-level notes on each of them, and counting the repetitions would flatter the exhibit.

What it refuses#

3 of the 16 files are refused outright, and the refusal is pinned in the same oracle as every accepted output: a .rejected.json beside the expected outputs, carrying the verdict and the reason. A refusal is an answer, not a gap — it is what stops a later version quietly starting to guess.

• PREMWRITTEN_FY2025_legacy_extract.txt — refused, scope out_of_scope_v0, pinned in corpus/expected/PREMWRITTEN_FY2025_legacy_extract.txt.rejected.json. The reason, verbatim: the header names 12 months as columns: this is a wide period matrix (one column per month), not a schedule of risks. Each cell is a written-premium figure, which the v0 risk object has no field for — the only premium it carries is a technical price produced by Price (SPEC §3.3) — so Ingest v0 refuses it rather than inventing a shape for it
• EQPORT26_MAINFRAME.PRN — refused, scope out_of_scope_v0, pinned in corpus/expected/EQPORT26_MAINFRAME.PRN.rejected.json. The reason, verbatim: no delimiter (comma, semicolon, tab or pipe) occurs anywhere in the file: this is a fixed-width record format, whose column boundaries are external knowledge (a record layout), not something the bytes state. Ingest v0 reads delimited text and JSON Lines only, and guessing the boundaries would shred every row
• NamedStorm_CatCircle_Schedule_FY26.pdf.txt — refused, scope out_of_scope_v0, pinned in corpus/expected/NamedStorm_CatCircle_Schedule_FY26.pdf.txt.rejected.json. The reason, verbatim: the file contains form-feed page breaks: it is a paginated text dump of a printed document (pdftotext -layout and friends), where column alignment is whitespace and cells wrap onto continuation lines. Ingest v0 reads delimited text and JSON Lines only, and will not reconstruct a table from a page layout

Refusal also happens below the file: a record Ingest cannot parse is reported, never dropped and never repaired into a total. The corpus pins that case too — one JSON Lines file carries a malformed record, and its expected output carries the object built from the rest of the file with the bad record declared in ambiguous_rows.

What Ingest never does is decide whether a risk is worth taking. It classifies data; a file it reads is not a risk it has accepted, and a file it refuses is not a risk it has declined (SPEC §3.1).

Where each number comes from#

• The file list, the formats, the encodings, the row counts and the curse counts are read from corpus/manifest.d/*.yaml. Every other figure is computed over that same list, so a manifest edit moves this page or fails the build.
• Whether a file is read or refused is not stated here: it is which of <file>.expected.json and <file>.rejected.json is committed under corpus/expected/.
• The honesty-entry counts are read out of those expected outputs, and the refusal reasons are copied from the rejected ones byte for byte.
• The 3 mandatory lists come from schema/risk-object.v0.schema.json, the published schema, which you can fetch without an account.
• POST /ingest is the route behind all of it; the quickstart shows it end to end.