Sacho

The record the subject cannot read while alive.

In 1404 King Taejong of Joseon fell from his horse while hunting and told those around him not to let a recorder know. The court historians recorded it anyway. They wrote it into a sacho (사초, draft record) the king was forbidden to read, which was compiled into the sillok only after his death.

That is the whole design. Incorrigibility is enforced by structural inaccessibility, not by cryptography: that it was said cannot later be revised by the subject, and whether it was true is always revisable.

What this v0 is

A feasibility study for coherence, not soundness.

There is no ground truth about the world available to this system. So it does not try to be right. It builds a set of claims that does not contradict itself, and where it does, it surfaces the conflict rather than silently resolving it. Every output is a claim about a corpus, never a claim about the world.

The interesting measurement is how much of that apparent coherence rests on unconfirmed identity decisions. A contradiction that only exists if one specific vocabulary merge is confirmed is a different kind of object from one that holds regardless, and the report says which is which.

Running it

Two paths. The fixture path measures the engine on hand-authored claims. The end-to-end path measures the product: real source texts, a real model, the same detection code.

uv sync

# fixture path — no LLM, no network, deterministic
uv run sacho run reagan_aids -o out/reagan_aids.md

# end-to-end — replays a committed recording, so still no model call
uv run sacho e2e reagan_aids --prompt v2 --replay -o out/e2e_v2.md

# re-record from the live model (spends calls)
uv run sacho record reagan_aids --prompt v2

uv run pytest -q          # 91 tests

Both exit 1 when a pre-registered gate fails.

Gates

Thresholds live in gates.json, registered before the run. They are data rather than code so that moving a bar is a visible diff in the same commit as the result that motivated moving it. A gate with no threshold is decoration; evaluate_e2e raises rather than defaulting a missing one.

Fixture path — sacho run, currently all green

gate	result
G1.1 recall of planted contradictions	PASS — 7/7
G1.2 precision against near-misses	PASS — 0 FP on 6
G1.5 modality false-fire rate	PASS — 0
G3.1 conditionality reported	PASS
G3.2 reception findings attributed to the record	PASS

3 unconditional findings, 4 conditional. sacho run reagan_aids exits 0.

End-to-end path — sacho e2e, currently G1.4 failing

gate	v1	v2 (default)	v3 + vocabulary
G1.4 extraction quality	FAIL 0.20	FAIL 0.44	FAIL 0.28
G2.1 extraction manufactures no contradictions	FAIL 9	FAIL 2	FAIL 4

precision, need ≥0.5. Both gates fail on all three prompts. Both SPEC §8 tuning cycles are spent and neither reached the bar; gates.json was not edited.

G2.1 is registered max_findings: 0 on the premise that a correct uncanonicalized extraction produces zero findings. That premise was true while the corpus held no genuine contradiction and is now known false — see the R3 note below. The gate is left red and the premise is recorded as the finding, per the rule that moving a bar to fit a result is a contract failure.

The thresholds are set where the project needs them, not where the current prompt lands. G1.4 fails today and that is the honest state of the study. The fixture path passing and the end-to-end path failing are both true at once, which is the whole reason the two paths are gated separately.

The fixture gates do not run on the end-to-end path: gold.jsonl names hand-authored claim ids (k01..k14) that an extraction run, whose ids are content hashes, cannot produce. Running them there would report a total miss and mean nothing.

Extraction, measured

Gate G1.4 needs a model. It runs on the local codex CLI (gpt-6-luna) via codex exec --output-schema, which constrains the response shape as a property of the call rather than a request in the prompt. Every call is cached on disk keyed by prompt + schema + model + document text, so re-running the benchmark after an engine change is free and after a prompt change is not.

Response is checked into fixtures/reagan_aids/recorded/ as a recording — a fact about what the model said once. That is what makes --replay possible, and it means a fresh clone reproduces these numbers with no API key, no cache, and no assumption that the model has not since moved.

metric	prompt v1	prompt v2	prompt v3 + vocabulary
expected propositions	18	18	18
precision	0.20	0.44	0.28
recall	0.44	0.61	0.50
claims / document	5.11	3.14	3.56
distinct relation labels	35	20	20
vocabulary fragmentation	6.5x	3.7x	3.7x
findings fed to the engine	9	2	4

Cycle 1 (v1→v2) added two worked examples addressing the two dominant failure modes: a characterisation is not the subject's utterance, and a predicate is not a proposition.

Cycle 2 (v2→v3) added the two-pass vocabulary injection — pass 1 collects the relation labels the model invented, pass 2 is given that set so it can reuse its own vocabulary. It works: six spellings of one relation collapsed to one. It also regressed, and the reason is the most useful thing this project has learned about merging. See prompts/extract.v3.md, which carries the full table and the finding.

Both cycles are spent. v2 remains the default and v3 is not.

fragmentation is reported next to precision because a run can score well and still be dead downstream. Detection groups claims by relation; four labels for one concept means the engine sees four relations where there is one. This is why G1.4 registers fragmentation as its own threshold: precision 1.0 with fragmentation 5x is a pipeline that cannot produce a finding at all.

The finding that mattered more than the score

On a 7-document corpus, feeding the v2 extraction into the engine produced 0 findings. That was correct, and it invalidated part of the fixture.

The planted R1 contradiction is "Reagan held office in California and in Illinois, 1967–1975, at the same time." That window cannot come from the text:

• The New York Times source says "Governor of Illinois before he moved to the West Coast" — no dates at all. The fixture assigned the claim the window 1967–1975 by hand. Extracted honestly, the dates are None.
• v2 correctly marks both New York Times claims role=addressee, because a reporter characterising Reagan is not Reagan asserting anything. The role gate then suppresses R1, by design.

So the fixture's headline contradiction is an artefact of hand-authoring, not a fact recoverable from the corpus. Prompt v1 — which got roles wrong — produced 3 findings on the same corpus, all of them noise manufactured by the extractor's own metadata errors.

That is the project's thesis, confirmed against itself: a phantom contradiction manufactured not by identity judgement but by a metadata field being wrong.

What that exposed, and what it became: R3_reception

The 0-findings result was also wrong in the other direction. Role gating is designed to stop a journalist's error becoming a subject's contradiction, and it overcorrected into hiding the journalist's error from the reader. "The Times said California in 1987 and Illinois in 1988" is an incoherence in the record — about reception (T3), not about the subject — and the report said nothing about it. A sacho exists so that what was said survives the person who said it.

R3_reception is the third rule, and it is the one SPEC §6.4's "two rules, and only two, until a measured failure demands a third" was waiting for. It applies R1's uniqueness test to claims the subject did not make. A mixed utterer/addressee pair is refused in both orders, because the attribution would be ambiguous and guessing is worse than silence.

Its finding text is the substance of the rule, and it exists to stop the tool laundering a source's error into a fact about a person: the detail opens "RECEPTION INCOHERENCE, attributed to the record and not to {subject}" and closes "this is a fault in how the sources describe {subject}, not a claim that {subject} held both." Gate G3.2 enforces that wording, because nothing downstream re-derives attribution from claim ids.

Two documents were added to the fixture containing a real reception incoherence — each NYT piece describes Reagan as holding two governorships at once. The extractor found them without being told, from the text, each in a single document. Those are the 2 findings v2 now produces, and they are why G2.1 fails legitimately rather than by extractor error.

Three design decisions worth knowing about

The fixture is the extraction oracle. Claims are hand-authored so the detection path can be measured on its own. That seam is what makes a detection bug and an extraction bug distinguishable failures rather than one indistinguishable failure.

Modality and role are the load-bearing filters. Only stated claims can contradict each other, and only utterer-role claims are posture evidence. A hypothetical is not an assertion; a claim about who you are talking to is not a claim about you. Ignoring these does not cause identity errors — it causes modality errors that look like identity errors and send the investigation the wrong way.

Contested cardinality is reported, not resolved. Merging a multi relation with a functional one leaves the merged relation's cardinality undecided. The engine fires the finding and flags the contest rather than picking whichever declaration happens to sort last. Silently resolving it is the exact failure this project exists to prevent.

Bugs this harness caught in its own engine

Recorded because a feasibility study that cannot fail is not a study.

• Snapshot semantics silently emptied a 1967–1989 corpus. Point-in-time liveness filtering applied to a coherence scan, which asks whether two claims are simultaneously true — a question intervals_overlap already answers. Fixed by separating read_mode="scan" from "snapshot".
• Zero-width intervals conflicted with everything. [2005,2005) is the empty set, but the pairwise interval comparison returned True for it.
• Merge canonicalization orphaned cardinality entries. Keying the cardinality table by resolved relation while looking up the canonical form suppressed every finding that depended on a merge.
• A gold label was wrong, not the engine. k03×k13 fired, gold had no entry, and the finding was correct. The harness now separates "false positive" (fired on a forbidden pair) from "not covered by gold" (fixture incomplete) — collapsing them makes every harness improvement look like an engine regression.
• The extraction scorer was measuring itself. Gold said governor of, the model said holds office, and the matcher demanded substring containment, so every relation miss was reported as an extraction miss. It was not. Replaced with prefix-root matching, after suffix-stemming was tried first and turned out to mangle served into serv.
• A second gold label was wrong. The draft's "the President's silence was the policy" affirms the silence; gold had encoded it as a denial. The extractor was right. Both wrong-gold events so far have been in the fixture, never in the engine.
• -c system_instructions= is silently ignored by codex 0.162.1. Verified by probe, not assumed. The working hook is model_instructions_file.
• Two gates could not fail, and neither was known to be broken until tested. A must_not_fire gold line states "these two claims must not produce a finding" without saying which rule would fire, but both G1.2 and G1.5 compared the rule as part of the key. A false positive firing under any rule other than the empty string therefore slipped past both gates silently. Every gate had only ever been observed green, which is not evidence a gate works. Found by tests/test_gates_negative.py, which breaks one thing at a time and requires the matching gate to notice.
• A missing gate threshold defaulted to "always pass". evaluate_e2e read gates.json with dict.get(key, default), so deleting a line would have quietly relaxed a gate. It now raises. Silently defaulting a threshold is how a bar gets moved by accident.
• documents_with_zero_claims could never fire. It compared the set of documents that produced claims against itself. It needs the manifest.
• The renderer hardcoded one fixture's document ids into a general-purpose report, so it printed a sentence about d_wikipedia_reagan and d_calonico_2015 on any corpus, including extraction-derived ones that have neither.

Deferred, with reasons

• Scope for R1 and R3. Both are uniqueness constraints with no notion of scope, and corpus 2 measures exactly what that costs: 6 false positives on 11 near-misses, every one of them a pair where one side narrows or special-cases the other ("we will cut Income Tax" beside "commit to no increases"). G1.2 is red on uk_manifestos by design — that corpus exists to be the thing that says no, and a test pins the exact false-positive set by name so a future fix has to be deliberate. See SPEC-CORPORA.md.
• Atom-side canonicalisation. The vocabulary injection unifies relations and nothing unifies atoms: "Governor of Illinois" and "Illinois governorship" are one entity with two ids, so the engine reports a person holding both. This is the other half of SPEC D7 and the actual blocker on G1.4's fragmentation.
• Merges that remove findings are invisible. detect() unions the unconditional baseline with merge-conditional results, so a merge that dissolves a contradiction contributes nothing and the finding stands. An operator who confirms a merge is shown the same report as one who did not. Corpus 1 only ever had merges that added findings, so this had never been exercised. Not fixed in v0: the fix is turning a flat union into a per-merge delta, which is a design decision about what a merge is for.
• A third G1.4 tuning cycle. Both are spent and neither reached the bar. The candidate is prompts/extract.v3.md; the corrected §6 and §8 have not been re-measured, and running them is a cycle 3.
• Identity judgement. Never asserted autonomously. Proposals are made and reported; nothing acts on them (SPEC §5.3, D8).
• Vector search. Not used. If a measured retrieval failure demands it, that is a reason to adopt it.

Layout

src/sacho/
  domain.py       pure vocabulary: tiers, modality, role, cardinality
  detect.py       the engine. R1 + R2 + R3, pure functions, no I/O
  corpus.py       fixture loading, and the bridge from extraction to detection
  extract.py      sources → claims, with validation and a rejection log
  score.py        extraction vs gold
  pipeline.py     sources → extract → detect → gates → report
  gates.py        pre-registered gates, loaded from gates.json
  recordings.py   committed model output, for replay without a model
  report.py       markdown. The deliverable
  backends.py     codex subprocess + cache, and the vocabulary-primed variant
  cli.py
fixtures/reagan_aids/
  fixture.json    hand-authored claims — the extraction oracle
  gold.jsonl      pre-registered findings + near-misses
  extraction_gold.jsonl
  sources/        raw texts the extractor reads
  recorded/       committed model responses (v1, v2, v3)
fixtures/uk_manifestos/   corpus 2. DESIGNED TO FAIL G1.2. See SPEC-CORPORA.md
gates.json        registered thresholds, diffed like code

STATUS.md records what was measured, what each gate says now, and what is blocked. SPEC-CORPORA.md covers the two corpora. SPEC.md is the full feasibility study and its non-goals table.