The Running Balance Is a Free Oracle for a Bank Statement Parser

Why there is no test set

The obvious move is to collect a few thousand statements, label them by hand, and measure. Three things stop it. The documents belong to other people, so keeping a corpus of real customer statements to regress against is the opposite of what anyone converting a statement wants. The layouts are effectively unbounded: there are tens of thousands of banks, most of them change their template every few years, and the same bank prints a business account differently from a current account. And labelling is expensive precisely where it matters, because the rows a human transcribes fastest are the rows a parser already gets right.

So a fixture suite built from synthetic PDFs tells you that you have not regressed on the shapes you already thought of. It tells you nothing about the statement a customer uploaded this morning from a bank you have never seen, which is the only case where being wrong has a cost. For that you need something that can check a reading it has never seen before, without a human and without a label.

The answer key is printed on the page

Almost every statement prints a running balance beside each row, and each balance is the one above it plus the credit or minus the debit. Which means the document is one long sum: take the opening balance, apply every transaction in order, and you have to land exactly on the closing balance.

That is an oracle. Not a heuristic that scores a reading, an actual check that a reading is correct. If a parse reconciles from opening to closing, it found every row in that span and it put each one in the right column, because any missing row, any doubled row and any sign flip moves the total. If it does not reconcile, something is wrong and you know the amount it is wrong by.

The important consequence is architectural. There is exactly one parser here and it knows nothing about any particular bank. The bank names exist to read a name off the page and nothing else. Writing rules per layout would have meant an unbounded amount of work validated by nothing; checking arithmetic is a fixed amount of work that validates every layout, including the ones that do not exist yet.

Reading order stops being a guess

Statements come oldest first or newest first, and there is no reliable signal in the text for which. The dates do not settle it: plenty of statements span a month boundary, plenty print dates in a format that is ambiguous for the first twelve days of the month, and plenty of rows share a date with the row above.

With an oracle you stop guessing. Compute the chain end to end with the rows as printed. If that fails, compute it with the rows reversed. If the reversed order reconciles outright and the printed order does not, then reversed is the reading, and no heuristic was involved: it is the only order in which the arithmetic works.

Keeping that rule narrow mattered more than making it clever. It acts only when the rows on hand fail and the inverted rows pass cleanly. A statement with no balances to check, or one that fails both ways, is left exactly as it came. An oracle is useful because its answer means something, and a version of this that fired on a close call would have been just another heuristic wearing a proof as a costume.

A proved reading is also a free reading

We expected the oracle to be a quality mechanism. It turned out to be the cost mechanism too.

The cheap path is a few hundred lines of column rules that run in the browser in milliseconds. The expensive path sends the transaction table to a model, which costs money and takes seconds. Choosing between them used to be a judgement call about whether the rules looked like they had coped.

It is not a judgement call any more. If the rules produce a reading that reconciles end to end, that reading is correct, and there is nothing a model could add to a correct answer. So the question the conversion asks is not which parser is better, it is whether the arithmetic already proved the cheap one. Roughly four statements in ten settle that way and convert in a second or two for nothing. The rest go to the model, including any table with too few rows for the check to mean anything, because a sum over three rows is not proof of much.

That is a nice property of building on a real oracle rather than a score. The same check that tells you the answer is right tells you that you can stop working.

Where the oracle is blind

A clean reconcile is strong evidence, and it is not evidence about the whole document. Not every row carries a running balance: banks omit it on some lines, and the parser reports nothing where it cannot read one. The opening figure has to be derived from the first row that does print a balance, and that figure already contains the effect of everything above it.

So the sum runs over the rows the balances actually span, and the rows outside that span are unchecked. They are also exactly where a wrong row hides, because nothing in the arithmetic can contradict them. We learned this from a customer who explained it in one line, on a statement of over 130 rows: the first two were in the wrong column, the rest were fine.

The fix was not to extend the check, because the balances genuinely cannot reach those rows. It was to make the result report how many rows it could not reach, and to make every caller that treats a reading as proved look at that number as well as at the verdict. An oracle that silently covers 128 rows out of 130 while reading like it covered all of them is worse than one that admits the gap.

The bug that accused honest customers

The worst bug in this system was not a parse failure. It was the oracle itself being wrong, in the one direction that is visible to a user.

The sums originally ran over every row in the statement, while the opening figure came from the first row that printed a balance. Those two do not agree. If the first row had no balance, its amount was counted in the total and also already inside the opening figure, so it was charged twice, and the statement came back as a mismatch by exactly that row’s amount. Same at the other end. The parse was perfect and the check said it was not.

That would have been a cosmetic wrong answer if a mismatch were only a mismatch. It was not: a balance that does not add up is one of the strongest signals that a statement has been edited, so the mismatch was escalated into a danger-level finding and shown to the person who uploaded the file. A clean statement, read correctly, was being presented to a broker or a lender as one that may have been doctored. The checks that actually catch an edited statement are worth something precisely because they are rare, and a false positive spends that credibility.

The lesson generalised further than the fix. The moment an oracle is wired to anything a user sees, its exact scope becomes a correctness requirement, not documentation. A check that is right about the rows it covers and vague about which rows those are will eventually accuse someone.

One verdict per statement is not enough

The first version answered for the document: it adds up, or it is out by 12.50. That is the right answer to the wrong question. A customer holding a year of statements that are each out by a few pounds still has to find the row by hand, and the whole point was to stop people checking rows by hand.

The same arithmetic gives a verdict per row if you stop at every printed balance instead of only at the last one. Each span runs from just after one balance to the next, and every row inside it stands or falls together with that balance. The output is then one of three states per row: the balances vouch for it, the balances contradict it, or no balance reaches it.

This is the version that is worth putting in front of someone. The file can point at the rows it is unsure about, the reader checks three rows against the PDF instead of three hundred, and the rows marked as uncovered are flagged as uncovered rather than quietly presented as verified. It runs on every conversion, free ones included, because a check you have to pay for is a check nobody runs. What each check looks at and where its limits are is written out rather than scored.

If you are building one of these

The short version of four things we would want to know at the start:

• Look for an oracle in the document before you write rules or reach for a model. Financial documents are full of them: running balances, subtotals, page totals, opening and closing figures. They are cheaper than labels and they work on layouts you have never seen.
• Let the oracle choose between your parsers instead of tuning a confidence threshold. A proved cheap answer needs no expensive second opinion, which is a quality decision and a cost decision at the same time.
• Make the oracle report its own coverage, and make every caller read it. The rows a check cannot reach are the rows your bug reports will be about.
• Be careful what you wire a failed check to. Arithmetic that does not add up looks like evidence of tampering, so an oracle with an off-by-one in its scope does not produce a wrong number, it produces an accusation.

What it does not solve

None of this helps with a statement that prints no balances at all, and some credit card and savings statements do not. It does not help with a scanned page where the balance column itself was misread, which fails the check without telling you that the check is what failed. And it says nothing about descriptions or categories, because no arithmetic in the document constrains them.

It does mean the open questions are the ones worth having. Not "did we read this right", which the document answers for itself on most statements, but the narrower set where it cannot: short tables, missing balance columns, and the handful of rows at each end that the chain never reaches. If you want to see what a verdict looks like on your own file, the converter prints the row-level checks on the free first page, and the API returns the same reconciliation result alongside the transactions.