How to Know When to Trust AI Decisions

What three rounds of experiments taught us about verification, calibration and reliable enterprise automation.

Executive summary

Every AI system that books invoices, codes costs or extracts contract terms makes two decisions: what the answer is, and whether a person needs to check it. The second decision determines whether automation saves money or creates risk. It depends on one number: how sure the system is.

We tested that number in three rounds of experiments: 15 models from six vendors, synthetic contracts and invoices with known answers, simulated bookkeeping where the true odds are known, and one company's real postings. What we found:

• Don't take a model's word for how sure it is. When asked, widely used mid-tier models said 90–100% whether they were right or wrong: 104 of their 108 wrong answers came with 90% or more. The newest frontier models were rarely wrong, and their doubts were informative, but they still could not tell when the evidence itself was thin.
• A second opinion works. For the mid-tier models that run most production volume, a separate verifier shown only the document and the proposed answer separated right from wrong almost perfectly, at a fraction of a cent per check.
• How you ask matters as much as which model you use. Asking for a probability for every option was up to five times better calibrated than asking for one answer and a confidence, using the same model at the same price.
• History is both the evidence and the risk. With thin history (new vendors, first postings), every method was overconfident. When a company changed how it booked, every method stayed confident while accuracy fell.
• Reliability is designed, not prompted. What worked was a layer around the model: independent verification, the customer's own history with its reliability stated, calibration on reviewed outcomes and a review policy. Better models make this layer cheaper. They do not make it unnecessary.

1. When 95% confidence isn't 95% accuracy

On one company's real invoices this autumn, a coding model trained on that company's own history said it was 95% sure of the cost centre it suggested. It was right 64% of the time.

Nothing was wrong with the model. The company had started booking differently, so the history the model had learned from no longer described the present. The confidence number did not move, because it had no way of knowing.

That gap between how sure a system says it is and how often it is right is what decides whether invoices get booked without a person looking at them. Accuracy gets the attention, but the amount of automation is set by the threshold: the point above which the system is trusted to act alone. A threshold is only as good as the number it is compared against.

This is the part of enterprise AI that demos skip. A model that produces a good answer is now easy to buy. A system that knows which of its answers a person must look at is not, and that is the system a finance team needs. Every invoice line ends up in one of two places: booked automatically, or in front of a reviewer. Get the routing right and people spend their time on the cases that need judgement. Get it wrong and errors reach the ledger unseen, or reviewers re-check work that was fine and the automation stops paying for itself.

So we tested the number.

2. We put confidence to the test

We ran three rounds, each building on the last. They cover two different kinds of problem, and the difference matters for everything that follows.

Facts. Contract and invoice extraction has one correct answer: the notice period is 120 days or it is not. Here the question is whether a confidence score separates right answers from wrong ones.

Odds. Cost-centre coding often has no single correct answer. A vendor's invoices go to Production most of the time and to Maintenance some of the time. Even a perfect system is wrong part of the time, and the honest output is a probability. Here the question is whether that probability is accurate.

3. Asking a model how sure it is measures the question, not the answer

The obvious way to get a confidence score is to ask for one. Many products add "give your confidence from 0 to 100" to the prompt, or ask for High, Medium or Low.

Three widely used mid-tier models (gpt-4.1-nano, gpt-4o-mini and gpt-4.1-mini) answered our 441 extraction questions and stated their confidence. Of the 108 answers they got wrong, 104 came with a stated confidence of 90% or more. Ninety-nine came with 95% or more.

Asking for a label instead of a number doesn't help. Eight of ten models put 99% or more of their answers in High. The one model that made real use of Low was right 89% of the time when it said Low and 94% when it said High. The label carries almost no information.

It is tempting to read this as a weakness that a better prompt will fix, and a better prompt does help, which is worth knowing. Asking the model to list concrete ways its answer could be wrong before stating a confidence lifted ranking quality from 0.68 to 0.84 for gpt-4.1-mini and from 0.55 to 0.85 for gpt-4o-mini. For the smallest model nothing we tried helped.

Two other popular signals did worse than asking. Reading the probability the model assigned to the tokens of its answer, and asking the same question several times to see whether the answers agree, both reported near certainty regardless of whether the answer was right. They measure how fluently the model produces an answer, not whether the answer is true.

The best models are rarely wrong, and their doubts are data

The newest models changed the picture, though not the way we expected. On the same questions, gpt-5.5, gpt-5.6-sol and Kimi K3.1 answered every one correctly, and three more made one error each. On documents buried in 30,000-token packs and run through simulated OCR noise, the top models still made almost no errors. There was very little left to calibrate.

So the strongest models' self-confidence can work, and it was well ranked on the few errors they made. But there were so few errors that this tells us more about the test set than about the method. What these models left behind was more interesting. Their confidence sat at 100% on nearly everything, and where it dipped, it dipped on a small set of questions. When we looked closely, many of those questions were broken. One invoice mentioned zero-rated lines that its table did not contain. One notice clause tied termination to the end of the contract term in a way our generator had not modelled. One amendment was left unsigned with "[Date to be inserted]". Thirteen of our 454 questions were flawed, and gpt-5.5 hesitated on eleven of them.

Two lessons came out of this. The practical one: when the strongest available model disagrees with your labels, check the label before you check the model. The more important one: a model that is almost never wrong does not need a confidence score to automate. It needs a way to say "this one is odd". Those dips were the most useful signal the top models produced, and they pointed at the input, not the answer.

Accuracy did not follow price, either. The smallest and cheapest model of the newest generation was in the top group, while a larger, several times more expensive model from the previous generation made twelve errors on the same questions. Release date didn't decide it either: a new frontier model from another vendor was as accurate but could not rank its own errors at all.

4. An independent second opinion changes everything

What worked was asking someone else. We gave a separate model only the document, the question and the proposed answer, and asked whether the answer was correct. It never saw the extracting model's reasoning or its confidence.

We tested two verifiers. One was a large general-purpose model, read through the probability it assigned to "Yes". The other was Microsoft-Decision-1, a small model built to return a calibrated probability for a yes-or-no question about a document.

The difference was not subtle. The extracting models' own confidence separated right from wrong answers at 0.54 to 0.68. Decision-1 did it at 0.98 to 0.99 on the same answers.

Here is what that looks like on real fields. One notice period had been amended later in the contract. The model missed the amendment and reported the original value with 100% confidence. The verifier gave it 0.01. Two invoice totals with arithmetic errors were reported at 90% and 95%. The verifier gave them 0.13 and 0.22.

We tried to break it. We had the extracting model write a confident justification for each answer, quoting the contract, and showed it to the verifier. Ranking quality did not move. We also tested free-text fields such as payment terms, where the most common error is leaving something out. When we asked the verifier one question per required element of the field in a single call, the share of omissions caught at a 5% false-alarm rate rose from 84% to 94%, and the answers named which element was missing.

The automation trade-off

The number that matters to a finance team is not AUROC. It is how many fields can be booked without review at an acceptable error rate. Sort every answer by the verifier's probability, start booking from the top, and track the error rate among the booked answers as you go further down.

For gpt-4.1-mini, the verifier let 96% of fields through at an error rate of 1% or less. The model's own confidence managed 37% before its errors piled up.

That 96% needs a caveat, and the caveat matters more than the number. It is measured with a threshold chosen on the same answers it is evaluated on, which tells you what is possible, not what you will get. In production the threshold has to be set in advance. When we set it on half the documents and applied it to the other half, across 500 random splits, the median result held: 96% booked at a median error of 0.95%. But in 43% of the splits the error on the new half exceeded the 1% target, and in one split in ten it reached 3.2%. A threshold is an estimate, and it needs a safety margin and continued monitoring.

What you pay for is the review queue

The economics are easy to misjudge. If a person needs thirty seconds to check a field, review costs about $375 per thousand fields at $45 an hour. Gating on the model's own confidence cuts that to $237. Gating on the verifier cuts it to about $18, including the verifier itself, which costs $0.34. The verifier is a rounding error. What you pay for is the review queue, and only a confidence number you can trust shrinks the queue.

5. When the right answer is a probability

Everything so far had one correct answer. Coding work is different. A vendor's invoices go to Production most of the time and to Maintenance sometimes. A product code fits three categories. Three past bookings are too few to tell. Here the question is not whether the answer is right, but whether the stated probability is right.

Real data can't answer that, because it gives you one booked value per line, not the odds behind it. So we built simulated bookkeeping worlds where the odds are known exactly. Some vendors book to one cost centre 97% of the time, others split roughly half and half, items tilt the mix, and histories range from 0 to 30 past bookings. For every prediction we know how likely it was to be right, so we can measure calibration directly.

The way you ask turned out to matter as much as which model you ask. Here is one real item, a line from a vendor that splits its invoices, where the true probability of the most likely cost centre is 60%:

Asked for a probability for every cost centre, gpt-4.1-mini said 60%. Asked for one answer and a confidence, it said 90%. Its token probabilities said 100%, and eight samples agreed eight times out of eight. Same model, same price, same information. Only the question changed.

Across 800 items, asking for a probability per option gave gpt-4.1-mini a calibration error of 0.049. One answer plus a confidence gave 0.253, five times worse and always on the side of too sure. The gap was largest for smaller models. For gpt-5.5 it narrowed to 0.083 against 0.115. Asking for odds was never worse.

The thin-history trap

The largest errors were not spread evenly. They sat where the evidence was thinnest. With one past booking from a vendor, which says almost nothing about a vendor that splits its invoices, gpt-4.1-mini stated 90% confidence on answers that were right 63% of the time. With no history at all it stated 70% on answers right 34% of the time. Larger models did better but did not escape it: with one past booking, the best models put 74 to 78% on answers that were right 63% of the time.

A plain classifier did not fall into the trap. Given exactly the same history, a counting model that shrinks a vendor's few bookings towards the company-wide pattern was as accurate as any LLM on pure counting tasks. It was also the best-calibrated system we tested, with a calibration error of about two points. It knows nothing a language model doesn't. It simply never treats one observation as a pattern.

This is not a story about classical ML beating language models. The LLMs earned their place where text means something. In a world where cost centres had names and invoice lines described real things, the models beat the classifier on accuracy, 55–61% against 51%, almost entirely on vendors with no history. One model got general items right 60% of the time where the classifier, with nothing to count, managed 18%. With ten or more bookings in view, the models followed the company's own habits, even surprising ones like toner booked to Production, as well as the classifier did. With no history, though, no model can know a company's surprising habits. On those items the models were right 11 to 14% of the time while stating 37 to 52%.

The answer is to use both. Averaging an LLM's per-option probabilities with the classifier's, without fitting anything, gave a calibration error of 0.02–0.05 for every model we tried and kept the LLM's advantage on text.

The scale is wrong, not the signal

The single stated confidence, badly scaled as it is, is not meaningless. It rises and falls with the evidence. A simple map from stated confidence to observed accuracy, fitted on a few hundred labelled items, brought every model's calibration error to between 0.02 and 0.06.

6. Calibration needs context and history

One more experiment changed how we think about all of the above. We wrote invoices whose cost centre could only be judged from cues: the purchase-order prefix, the requester's department, the delivery address and the item. The prompt included a booking guide that stated how reliable each cue was: the order prefix is right 90% of the time, the department 70%, the address 60%, the item half the time. Cues often pointed in different directions. Because the invoices were generated from exactly those rules, the right answer is a probability that can be computed.

The strongest model computed it. Across 320 invoices, gpt-5.5's probabilities matched the exact odds with a calibration error of 0.001, measured against the known probability of each invoice rather than against single outcomes, which is why it can be that small. gpt-5.6-luna, the cheapest model in the top group, reached 0.004. Smaller models were slightly underconfident, the opposite of everything we had seen with raw history. Even the single stated confidence, which had been 20 to 30 points too high on thin history, was within a few points on average here.

Was it simply the format? We showed the same thin vendor histories as counts instead of rows ("this vendor: 2 lines to CC-296, 1 to CC-149"). It barely helped. One model's single confidence went from 25 to 24 points too high, another's from 16 to 9. Counts still leave the model to guess what two bookings are worth. The booking guide told it.

Models are not overconfident by nature. They are overconfident when they have to guess how much the evidence is worth.

This changes the problem. A confidence score is not something a model has. It is something a system produces from evidence the model can see, stated in a form it can reason about, and checked against outcomes. The customer's own history is that evidence. Shown as rows, or even as counts, it leaves the model to guess what one booking is worth, and the model guesses high. Shown with how much it can be trusted, which is what a calibrated model of that customer's history provides, it becomes arithmetic.

What real postings add: the world moves

Simulations are clean by construction. So we ran the same signals on one company's real postings: 1,722 lines coded with cost centre, VAT code and cost unit. Everything before a cut-off date was history, and the 319 postings after it were the test. A classifier was retrained before every posting date, on exactly the information the LLMs saw.

All methods were about equally accurate (around 82% on cost centre, 93 to 96% on VAT and cost unit) and ranked their own risk about equally well. They also failed the same way. Partway through the test period, the company began booking differently: the cost centre that had taken 95 to 100% of postings every month fell to about 60%. Every method stayed near certain. For vendors never seen before, every method stated 93 to 100% confidence on the cost centre and was right 57% of the time.

No choice of model fixes that. A system that learns from history cannot know the history has stopped applying until outcomes tell it. The fix is to treat outcomes as part of the system: watch how often suggestions are overridden, per vendor and per cost centre, and lower automation as soon as that rate moves.

7. From confidence to controlled automation

The fair objection to all of this is that models are improving so fast the problem will solve itself. Part of that is true. The top models made almost no errors on our extraction questions, and the strongest one reproduced exact Bayesian odds when the evidence was spelled out.

But accuracy on a test set is not the same as knowing when you are on unfamiliar ground. The same strongest model over-trusted one or two past bookings. Every model, and a well-tuned classifier, missed a company changing its habits. And when we gave models a coin flip, a document where the deciding fact was simply missing, they said "cannot determine" when allowed to, and picked an answer with high confidence when forced to. The failures that remain are not about intelligence. They are about what the model can see.

Taken together, the three rounds point to one design. Don't ask a model how sure it is and act on the answer. Build the confidence from things that can be checked, in a layer that sits between the model and the decision to automate. We call it a Decision Reliability Layer. It is not a product but a way of organising the parts:

• Independent verification. A second model that sees the document and the proposed value, not the first model's reasoning. It costs a fraction of a cent, and it was the first signal we found that reliably separates right from wrong.
• Historical evidence, with its reliability. The customer's own history, presented as odds the model can reason about rather than raw rows it must guess from. Keep a classifier on that history in the loop. It is the best-calibrated component you can have, and averaging it with the model keeps that property.
• Calibration. Ask for odds, not certainty: a probability for every option. Fit the map from stated confidence to observed accuracy on the customer's own reviewed items, per field family, and refit it.
• A review policy. An error budget per field family, with a margin, because thresholds set in advance scatter. Treat thin history as a reason for review whatever the stated confidence, and let the system answer "cannot determine".
• Outcomes back into the loop. Every reviewed item is a labelled example. Override rates per vendor and cost centre are the only early warning when a company changes how it books.

None of these is about choosing a model. Better models make every part of the layer cheaper: fewer errors to catch, fewer items to review, better use of stated evidence. They do not remove the need for it, because the remaining failures come from what the model cannot see.

8. Reliability is engineered, not prompted

We started out asking whether AI models know when they are wrong. Mostly, when asked directly, they don't. That turned out to be the less interesting finding.

The more useful finding is that reliability does not have to come from the model. It can be built around the model from parts that each do one thing well: a second opinion that checks the claim, history that states what it is worth, a calibration fitted to the customer's own outcomes, and a policy that knows its own margin of error. Each of those parts can be tested, measured and improved on its own, which is more than you can say of a number the model makes up about itself.

This matters well beyond accounting. As AI systems take on longer chains of decisions with less supervision, the question stops being "how good is the model?" and becomes "which of its decisions can we act on?". A confidence score is a promise about the future. The model can make it. Independent verification can test it, and history can keep it.

Reliable AI is not just about making better predictions. It is about building systems that know which predictions to act on, which to verify, and when to involve a person.

How we measured

Round one: 32 synthetic documents of about 8,000 tokens, 441 questions after an audit removed 13 flawed ones, answers computed in code, three GPT-4-era models, seven confidence signals. Round two: 15 models from OpenAI, xAI, DeepSeek, Moonshot, Mistral and Meta on clean, OCR-noised and 30,000-token inputs; six elicitation prompts; 134 undecidable questions; 572 free-text field values with planted defects; held-out threshold and transfer tests. Round three: three simulated bookkeeping worlds of 800 items each with known odds; 640 cue-weighing invoices; a time-split test on one company's 1,722 postings. Ranking quality is AUROC. Calibration error is the average gap between stated confidence and the true probability of being right.

Limits: the documents and simulated worlds are synthetic, and the documents' prose was written by an LLM, which may make them easier for LLM verifiers than real paperwork. The real data comes from one company and 319 test postings, and error counts per model are small enough that close differences should not be read into. Booked values may contain errors. We used one prompt per elicitation style except where noted. Coverage figures computed with in-sample thresholds show what is possible, not what is guaranteed; the held-out figures show what to expect.