Your invoice extractor scores 94 percent. It gets the vendor name right on every invoice and the tax amount wrong on one in five, and the 94 can’t tell you which.

Picture the Thursday this surfaces. Finance drops a message in the channel: three suppliers were paid the wrong GST last week. You open the eval dashboard and it says 94. It said 94 last week too. Nothing on that screen was lying. It just wasn’t answering the question finance is asking.

Issue 3 made the case for building evals before you ship; this issue is about what goes wrong when the eval you built collapses into one number.

By the end, you’ll have a field-level scorer you can drop into CI: normalisers per field type, separate counts for wrong, missing and hallucinated values, precision, recall and F1 per field, a confusion matrix for label fields, cost weights, and gates on the fields that actually carry risk.

Where the 94 comes from

Here’s a 500-invoice eval set with six extracted fields.

Field

Correct out of 500

vendor_name

500

invoice_date

490

invoice_number

480

currency

480

total

470

tax_amount

400

That’s 2,820 correct values out of 3,000, which is your 94. Five of the six fields sit between 94 and 100 percent. The sixth sits at 80, and it’s the one that moves money.

The blended number is mostly a measure of how many easy fields you have. Add a seventh field the model never gets wrong, say document_type, and the score climbs to 94.9 without a single change to the model or the prompt. I find this genuinely funny in a bleak way: you can improve your eval by adding questions the model already answers.

The average also hides three other problems.

Missing and hallucinated look identical. If the model leaves tax_amount empty, the invoice fails validation and lands in a review queue. A person spends two minutes on it. If the model invents a tax amount on a zero-rated invoice, the invoice passes validation and gets paid. Both count as one “wrong” in the 94.

Formatting counts as failure. 03 Apr 2026 and 2026-04-03 are the same date. ₹1,250.00 and 1250 are the same amount. A strict string comparison marks both as errors, and the team spends a sprint tuning the prompt to output exact formats instead of fixing the field that’s actually broken.

You can’t tell which direction a label is wrong in. For a classification field like an approval route, sending a reject to approve is a very different mistake from sending an approve to review. One number can’t hold that.

Why an LLM judge doesn’t fix this

The usual next move is a judge prompt: “Does this extracted JSON match the reference?” It feels like progress because it stops penalising formatting.

But you’ve traded one blended number for another. The judge gives you a pass or a 1-to-5 score per document, which still doesn’t say which field failed or how. It’s also non-deterministic, it costs money on every CI run, and now you need an eval for the judge.

For a field with exactly one right answer, like a tax amount or an invoice number, deterministic comparison after normalisation is cheaper, faster and reproducible. Keep judges for the fields that genuinely have more than one correct phrasing, like a line-item description or a summary. The cost of the deterministic route is that you write normalisers by hand, and they need their own tests. More on that in a minute.

Five outcomes, not two

The core change is to stop asking “right or wrong?” per field and start classifying every gold/prediction pair into one of five outcomes.

Every field comparison ends in exactly one of these five boxes. The two that look similar on a dashboard, missing and hallucinated, end up in different branches because they fail in different ways downstream.

From those counts, precision and recall fall out cleanly:

  • Precision = correct / (correct + wrong + hallucinated). Of the values the model gave you, how many were right.

  • Recall = correct / (correct + wrong + missing). Of the values that actually exist, how many the model captured.

A wrong value counts against both, and that’s deliberate. A confidently wrong tax amount is a value you can’t trust and a real value you didn’t get. Correct abstentions don’t enter either formula, so track them as their own rate. A model that always outputs something will never abstain, and you want to see that.

Now apply it to tax_amount. Of the 500 invoices, 420 have a tax line and 80 are zero-rated. The model’s breakdown looks like this: 345 correct, 45 wrong, 30 missing, 25 hallucinated, 55 correct abstentions.

That gives you precision of 83.1 percent, recall of 82.1 percent and F1 of 82.6. But the number that matters isn’t any of those. It’s the 25 hallucinated tax amounts on invoices that have no tax at all. That’s a specific, fixable failure: the schema probably doesn’t allow null, or the prompt never mentions that zero-rated invoices exist. You’d never have found it from the 94.

Normalise before you compare

Half of a good extraction eval is the normalisers. The rule is simple: run the same function on the gold value and the prediction, then compare the outputs.

  • Text: casefold, collapse whitespace, strip. "Acme Ltd " and "acme ltd" should match.

  • Money: parse to Decimal, strip currency symbols, handle both 1,250.00 and 1.250,00, quantise to two places.

  • Dates: parse to ISO 8601. Dates are the worst. 03/04/2026 is 3 April in Bangalore and 4 March in Boston, and no parser can know which you meant. Don’t let the normaliser guess. Pass the locale per vendor, and store gold dates in ISO so only one side is ambiguous.

The trick is to normalise formatting, never meaning. INV-0042 and INV-042 are probably different invoices. If your normaliser strips leading zeros, it’ll hide a real extraction bug behind a green tick. Same for rounding money to whole units: a ₹0.50 difference on a tax line is a wrong value, not a formatting quirk.

And test your normalisers. The first version of the currency parser below read Rs. 18,000 as 18.00, because the dot in “Rs.” survived the symbol strip and got treated as a decimal point. That’s a scorer bug that would have marked correct predictions as wrong on every rupee invoice. Ten unit tests would have caught it.

The scorer

Here’s the whole thing. It’s about 50 lines, has no dependencies beyond python-dateutil, and is meant to be copied and adapted. 

import re
from collections import Counter
from decimal import Decimal, InvalidOperation
from dateutil import parser as dateparser

def norm_text(v):
    return re.sub(r"\s+", " ", str(v)).strip().casefold() or None

def norm_money(v):
    m = re.search(r"-?\d[\d.,]*", str(v))
    if not m:
        return None
    s = m.group().rstrip(".,")
    if "," in s and "." in s and s.rfind(",") > s.rfind("."):
        s = s.replace(".", "").replace(",", ".")   # 1.250,00 -> 1250.00
    else:
        s = s.replace(",", "")                     # 1,250.00 -> 1250.00
    try:
        return Decimal(s).quantize(Decimal("0.01"))
    except InvalidOperation:
        return None

def norm_date(v, dayfirst=True):
    try:
        return dateparser.parse(str(v), dayfirst=dayfirst).date().isoformat()
    except (ValueError, OverflowError):
        return None

NORMALISERS = {"text": norm_text, "money": norm_money,
               "date": norm_date, "label": norm_text}

def outcome(gold, pred, kind):
    n = NORMALISERS[kind]
    g = n(gold) if gold not in (None, "") else None
    p = n(pred) if pred not in (None, "") else None
    if g is None and p is None: return "abstain_ok"
    if g is None: return "hallucinated"
    if p is None: return "missing"
    return "correct" if g == p else "wrong"

def score(pairs, schema):
    counts = {f: Counter() for f in schema}
    for gold, pred in pairs:
        for field, kind in schema.items():
            counts[field][outcome(gold.get(field), pred.get(field), kind)] += 1
    report = {}
    for field, c in counts.items():
        tp = c["correct"]
        p_den = tp + c["wrong"] + c["hallucinated"]
        r_den = tp + c["wrong"] + c["missing"]
        prec = tp / p_den if p_den else float("nan")
        rec = tp / r_den if r_den else float("nan")
        f1 = 2 * prec * rec / (prec + rec) if prec + rec else 0.0
        report[field] = {**c, "precision": prec, "recall": rec, "f1": f1}
    return report

You call it with a schema like {"vendor_name": "text", "invoice_date": "date", "tax_amount": "money"} and a list of (gold, pred) dict pairs joined on document ID.

Two design choices worth calling out. First, an unparseable prediction normalises to None and gets counted as missing, not wrong. I did that because in most pipelines a value that fails parsing also fails validation and goes to review, which is exactly how a missing value behaves. If your pipeline passes unparseable strings straight through, count them as wrong instead. Second, dayfirst=True is the right default for Indian and European invoices and the wrong one for US invoices. Make it a per-vendor setting before you trust the date numbers.

Label fields get a confusion matrix

For classification fields, per-label precision and recall tell you which classes are weak. The confusion matrix tells you where the errors go, which is usually the more useful question. 

import numpy as np
from sklearn.metrics import confusion_matrix, precision_recall_fscore_support

LABELS = ["approve", "review", "reject", "<none>"]
gold = [g.get("route") or "<none>" for g, _ in pairs]
pred = [p.get("route") or "<none>" for _, p in pairs]
prec, rec, f1, support = precision_recall_fscore_support(
    gold, pred, labels=LABELS[:3], average=None, zero_division=np.nan)
cm = confusion_matrix(gold, pred, labels=LABELS)

Two things matter here. Including <none> as a label in the matrix makes missing and hallucinated labels visible as their own row and column, instead of silently dropped. And average=None gives you per-label numbers rather than one blended F1, which would put you right back where you started.

The zero_division setting matters more than it looks. If the model never predicts a label in a small eval set, precision for that label is 0/0. scikit-learn’s default is a warning, and setting it to np.nan excludes that label from any average rather than quietly scoring it as 0 or 1.

Then decide which cells are forbidden. On an approval route, reject → approve might need to be zero, while approve → reviewcan be a few percent because it only costs a reviewer some time. Write those down as rules, not as part of an F1.

Weight by what an error costs

Once you have outcome counts per field, you can attach a cost to each outcome type and get a number the business actually cares about.

Take illustrative costs: a missing value costs about 2 minutes of review time, and a wrong or hallucinated tax amount costs about 60 minutes once you count the correction, the credit note and the awkward email to the vendor. These are made-up starting points. Ask finance for real ones.

Scaled to 1,000 invoices, tax_amount produces 140 wrong-or-hallucinated values (8,400 minutes) and 60 missing values (120 minutes). vendor_name produces zero. One field is responsible for almost all the cost, and the cost comes almost entirely from values the model gave confidently, not from ones it skipped.

If someone insists on a single number for the dashboard, make it cost per 1,000 documents instead of average accuracy. That number goes down when the business gets better, and it doesn’t go up when you add easy fields.

Gate CI on fields, not the average

Here’s how the pieces connect into a pipeline.

The gate reads per-field metrics and forbidden-cell counts, never the blended score. The cost number goes to the dashboard, not the gate.

The gate config is a short file per feature: 

fields:
  tax_amount:  { min_precision: 0.95, min_recall: 0.85, max_hallucinated_rate: 0.01, min_support: 200 }
  total:       { min_precision: 0.97, min_recall: 0.90, min_support: 200 }
  vendor_name: { min_f1: 0.97 }
labels:
  route:       { forbidden: [["reject", "approve"]] }

Money fields get a higher precision bar than recall bar because a hallucinated amount gets paid and a missing one gets reviewed. Low-risk fields can use F1, since precision and recall cost about the same there. These thresholds are starting points; tune them against your own production data.

Then there’s noise, which is where most CI gates go wrong. At 82 percent precision on 420 examples, the 95 percent confidence interval is roughly plus or minus 3.7 points. A 2-point drop between two runs is inside that. If you fail builds on it, people will learn to rerun until it’s green.

The better comparison is paired: score the old and new prompt on the same documents and count flips. If tax_amount went from correct to wrong on 14 invoices and from wrong to correct on 3, that’s a real regression even if the headline precision barely moved. That’s the idea behind McNemar’s test, and the simple version, “fail if bad flips exceed good flips by more than N”, gets you most of the way. Treat min_support as a hard floor: a gate on a field with 30 examples is a coin toss with extra steps.

Common mistakes

1. Normalising meaning away. A normaliser that strips leading zeros from IDs or rounds currency to whole units will turn real bugs into passes. The fix is to keep normalisers narrow: they remove formatting differences, and anything that could change which entity or amount you’re talking about stays in the comparison.

2. Building the gold set from model output. It’s tempting to run the model, have someone correct the output, and call that gold. Annotators anchor on what’s in front of them, and they rarely delete a plausible-looking tax amount from a zero-rated invoice. Hallucinations leak into gold and score as correct. Have annotators label from the source document, and require an explicit null for absent fields rather than an empty cell.

3. Gating money fields on F1. F1 weights precision and recall equally. For a field that gets paid, they aren’t equal, so a model that trades five missing values for five hallucinated ones keeps the same F1 and gets meaningfully more dangerous. Gate precision and recall separately on any field where the two error types have different costs.

4. Trusting per-label numbers with tiny support. A label with eight examples can swing 12 points from one relabelled invoice. Report support next to every metric, set zero_division=np.nan so empty labels don’t get a fake score, and don’t gate on labels below your support floor until you’ve collected more examples.

Summary

A single accuracy number for an extraction or classification feature measures how many easy fields you have more than how well the model works. The fix is mechanical: normalise each field by type, classify every comparison as correct, wrong, missing, hallucinated or correctly abstained, compute precision and recall per field, use a confusion matrix for label fields, attach costs to outcomes, and gate CI on the critical fields with a support floor and paired comparisons. The scorer above is enough to start, and the checklist below is the version for your planning doc.

Production checklist

  • Replace the blended accuracy number with per-field precision, recall and F1, and show support next to each.

  • Classify every field comparison into correct, wrong, missing, hallucinated or correct abstention, and track hallucination rate as its own metric.

  • Write a normaliser per field type, apply it to both gold and prediction, and cover each normaliser with unit tests that include ugly real-world inputs.

  • Make date parsing locale-aware per vendor or source, and store gold dates in ISO 8601.

  • Label the gold set from source documents, with explicit null for absent fields, rather than by correcting model output.

  • Build a confusion matrix for every label field, include a <none> label, and write down which cells must stay at zero.

  • Agree outcome costs with the team that absorbs the errors, and report cost per 1,000 documents instead of average accuracy.

  • Gate CI on per-field thresholds, with separate precision and recall bars for fields that move money.

  • Set a minimum support per gated field and compare candidate runs against the baseline as paired flips on the same documents.

  • Reserve LLM judges for genuinely free-text fields, and score those separately from the deterministic fields.