Beyond Edit Distance

Reference-Free Structural-Reconstruction Fidelity for Document OCR

Abstract

Modern document-OCR systems report near-perfect text accuracy: on OmniDocBench, normalized edit distance and character error rate (CER) have largely saturated. This saturation reflects a blind spot rather than a solved problem: edit-distance-family metrics score the content a parser emits against hand-annotated references; they do not test whether that output is sufficient to reconstruct the page. We introduce Visual Roundtrip Fidelity (VRTF), a reference-free measure of structural-reconstruction fidelity: from a parser's output alone we re-typeset the page at a single, corpus-consistent typography, place each block at its detected bounding box, and score the ink overlap of the reconstruction against the source image as a pixel confusion matrix. The metric needs no ground truth, only the source image the parser already consumed. Applied to a state-of-the-art parser (dots.ocr, 1.7B) on 2,351 scored pages of real engineering literature, character accuracy is saturated (CER 0.0045 on a transcribed subset) while structural reconstruction is only 54 to 73 percent, with a heavy tail concentrated on the tables, charts and figures that carry the design information. Scored on a second parser, VRTF reproduces, with no annotation, the ordering the annotated benchmark assigns. Because the metric is reference-free, it also decomposes the gap: each capability that enriches a parser's output (e.g. formula cleanup or chart-from-data recreation) moves VRTF by a measurable amount, so delta-VRTF ranks where reconstruction fidelity is won or lost, and an expert spot-check shows such gains can be partly coarse placement, which the protocol reports rather than hides. This makes VRTF a reference-free training signal: a capability roadmap for a reconstruction-aware vision-language model, and a reward to train it against. We position reconstructability as the evaluation axis that remains open after edit distance saturates, and release the metric code and per-page corpus scores as a harder, annotation-free benchmark.