What went wrong
Fifteen findings, and which of them were mine
This project’s premise was that a language model reading archaeological tables would need policing, and that building the harness to police it was the real contribution.
The premise half survived. The harness is the work. But across roughly 1,500 determinations, the model has produced almost no confirmed errors, and nearly every failure has been in my own schema, rules, or reference data.
That is a more useful result than the one expected, so the log is published rather than tidied away.
The pattern
Three findings share a signature that turned out to matter more than anything else here.
The dangerous failures are absences, not errors
F12, F13, F15 — and they are invisible by construction.
F12. Isotope data is often published in its own table, on a page containing no radiocarbon dates at all. The page selector required dates, so it skipped those pages — and every determination in those documents was counted as unscreenable. The site ranked first in the register turned out to have published its isotopes one page away from its dates.
F13. Records failing the schema guard were dropped with a
bare continue. They vanished with no trace. The only symptom was a
page reporting "0 records" where the model said it had seen 24.
F15. A partial write left 1,239 records with an empty report block. Nothing errored. The diagnostic printed "0 blocked", which reads exactly like a clean corpus.
A corrupted value trips a rule. A missing page, a dropped record, an empty report — none of those trip anything, and all three look like success. Verification harnesses are built to check what is present. The failures that threaten a compilation are omissions.
The one instrument that caught any of them was the model's own count of what it could see on a page, compared against what came out. That signal costs nothing and needs no ground truth.
Where the rules were wrong
The model was 100% accurate; the rule suite was 0% accurate
F4 — the finding that reframed the project.
The first live extraction was measured against an independent hand transcription: 11 of 11 correct on every field. The rule suite blocked all eleven. Every flag was a false positive.
The worst offender flagged any row containing "cal BC" as a possible BP/calendar confusion. But every properly published date table prints the uncalibrated age beside its calibrated interval — so the rule flagged every good table in the corpus. I had written the rules against synthetic data I invented myself, so they encoded my assumptions about date tables rather than real publishing conventions.
Operating rule since: when a flag fires on real data, the null hypothesis is that the rule is wrong, not the record.
A schema that cannot represent "not stated" forces fabrication
F9, F10 — twice, the same lesson.
One source prints 3590± with the error term genuinely missing
from the page. c14_error was a required integer, so the model
emitted 0 — the only thing the schema permitted.
Separately, Russian prose quotes calendar dates: «1370 ± 30 г. до н. э.» is
1370 BC, roughly 3320 BP. With only a c14_age_bp field available,
the calendar year went there. The model transcribed faithfully; the correct
value was unrepresentable.
Both are schema failures wearing the costume of model errors.
A registry that silently changed a date's country
F7 — the most dangerous single bug.
Laboratory prefixes collide across case. LU is St Petersburg
State University; Lu is Lund, Sweden. Ki is Kyiv;
KI is Kiel. My lookup folded case and rewrote the prefix to the
registry's spelling — so a Lund determination was quietly converted into a
St Petersburg one. Wrong country, wrong institution, and nothing downstream
could have caught it.
Found only by verifying the registry against the laboratory list maintained
by Radiocarbon rather than trusting my own assembly. A widely
circulated third-party lab-code list gets LU wrong the same way.
Where I misdiagnosed my own bug
A failed edit is indistinguishable from a successful one
F14 — reported wrongly, then corrected.
Every request began failing with "Grammar compilation timed out". I diagnosed it as server load, added retry and a warm-up, and said so. The warm-up then failed with a different error: Schema is too complex — the definitive verdict, not a load symptom.
The root cause was that a schema change I had committed earlier never applied. The edit did not match, silently, and I asserted the schema was unchanged based on what I had intended rather than what was on disk.
Same shape as the absence findings above, except the absent thing was my own change. The retry logic stayed: it cost one page of latency and it is what disproved my hypothesis.
What the model actually got wrong
Almost nothing that can be confirmed.
The digit-transposition check — the strongest rule here, because it needs no ground truth — compares the parsed number against the printed string the model transcribed separately. Across the corpus it has found zero genuine transcription errors. Every hit traced back to a schema that gave the model no honest way to answer.
The one place the model did produce something unusable was calendar dates in prose, and even there it recorded exactly what the page said.
What this suggests for the method
The honest headline is not “LLMs need policing before they can touch archaeological data.” It is closer to:
The model reads these tables more faithfully than a domain-naive rule suite can judge them. The engineering problem is encoding real-world publishing practice — and noticing what never arrived.
Nothing here is an argument for skipping verification. The harness caught real problems, including several of its own. But it should be pointed at absences and at its own assumptions, not primarily at the model.