Nine Models Took an Open-Book Math Test

Most LLM leaderboards grade what a model knows. FinPlanBench asks a narrower question that matters more to anyone shipping a financial copilot: given the complete rule, can the model execute it? Each task hands the model one published Worthune model specification — the full markdown, formulas, ordering rules, sentinel conventions, everything — plus one set of inputs, and asks for the exact scalar outputs as JSON. Open-book, on purpose. Knowing the rule is not the test. Executing it is.

The answer key is not another model's opinion. Every expected value is produced by our Concordance-tested engine: two independent implementations of each spec — the production TypeScript and a Python oracle rebuilt from the spec text alone — that must agree at one part in a billion in CI before anything ships. The benchmark is built entirely from the three free-sample models (emergency-fund, relocation, side-business), whose specs and eval datasets are public, so the cases, the answer key, and the runner are reproducible by anyone, no key required: 50 deterministic cases per model, 150 total, pinned by a SHA-256 over the whole case set. This week we published the first nine runs, spanning four providers. Here is what the numbers say.

The first leaderboard

Two bars per run, same answers. Strict is the Concordance harness tolerance itself: relative 1e-9 or absolute 1e-6. Practical is the loosest bar our grading accepts anywhere: within 1% relative or one dollar absolute — close enough to say out loud to a user. A case passes only if every scalar output agrees; there is no partial credit, because partial credit hides exactly the errors that matter. Unparseable means the response contained no JSON object at all, and it scores against the model. Sorted the way the live leaderboard sorts — best practical first:

All nine runs completed on 2026-08-16 against the same case set (SHA-256 717219e9…), prompt template v1.0.0, 150 cases each. The full artifacts — per-case verdicts, per-key diffs, and the exact parameters each run used — are committed in bench/results/ in the public repo; the leaderboard at worthune.com/benchmark renders those files and nothing else.

Read the two columns together and the table stops being a ranking and becomes a diagnosis. The gap between a model's strict and practical scores tells you what kind of wrong it is — and the nine runs sort cleanly into three failure modes.

Failure mode one: drift at the ninth digit

Claude Opus 5 (92.0% strict) and GPT-5.5 (89.3%) are doing the arithmetic. Their strict misses are not wrong answers so much as answers that ran out of precision. A representative GPT-5.5 miss on an emergency-fund case: expected interestEarned 1525.784740448711, got 1525.7847345641608 — agreement to eight significant figures, a $0.000006 error, and still a strict fail, because the strict bar is the same one our own CI holds the engine to. Opus's six failed cases are the same story: a savings target off by ten cents on $163,519, wealth projections off by dollars on seven-figure amounts. These models execute a 241-iteration compound-interest loop digit by digit and lose a rounding grain somewhere in the middle.

Failure mode two: always close, almost never exact

Gemini 3.7 Flash is the most interesting row on the board, and it is why the leaderboard's practical-first sort puts a 20.0%-strict model at the top. It returned parseable JSON on all 150 cases and landed within a dollar on 148 of them — and hit double precision on just 30. On 118 of its 150 cases the verdict is strict-fail, practical-pass. The misses look like this: expected interestEarned 1278.062120654975, got 1278.061987908298. Right to five or six significant figures, wrong after that — the signature of a model that approximates the iteration rather than executing it. Claude Sonnet 5 has the same shape at lower volume: every single one of its 84 strict failures passes practical, many of them because it rounds intermediate values (one artifact entry: expected 25352.39269885543, got 25352.392615 — the model reported six decimal places and paid for it at 1e-9).

A model that is always within a dollar and almost never exact is a different kind of wrong than a model that picked the wrong formula — and depending on what your product does with the number, either one can be the dealbreaker.

Whether 98.7%-within-a-dollar is good news depends entirely on what happens to the number next. Spoken to a user in chat, a dollar of slack on an emergency-fund projection is fine. Fed into anything downstream — a comparison against a threshold, a compounding projection, a ledger — approximately-right numbers stop being right at all.

Failure mode three: the wrong computation entirely

Claude Haiku 4.5 (15.3% strict, 18.0% practical) and GPT-5.4 mini (15.3%, 16.0%) show the third pattern: loosening the tolerance barely helps, because the errors are not precision errors. Both models took every strict pass they earned on emergency-fund — 23 of 50 apiece — and scored zero on the other two specs at either tier. The per-key diffs show different computations, not noisy ones: Haiku returned a wealth-at-horizon of 1800 where the answer is 18000, an order of magnitude gone; GPT-5.4 mini reported an emergency fund reaching its goal at the spec's 240-month never-reached sentinel when the fund hits its target in month 10, and answered "Infinity" where relocation's spec defines an in-domain break-even sentinel of 9999 — the spec says exactly how to say "never," and the model said it some other way. Wrong convention, wrong path, wrong stack — the failure modes a tolerance can't forgive.

Difficulty also stacks the same way for everyone. The three specs get harder in order — emergency-fund's single loop, relocation's parallel wealth projections, side-business's tax-and-growth interaction — and strict scores fall off the same cliff:

Unparseable is a verdict, not a footnote

The DeepSeek rows carry a number most leaderboards would quietly drop: V4 Pro produced no JSON object at all on 57 of 150 cases, V4 Flash on 44 — overwhelmingly on side-business, where both models went 34-for-50 unparseable. These are delivered responses with no extractable answer in them, typically a model that spent its entire completion budget reasoning. We grade them as failures, against the model, because that is what they are: a copilot that answers "…" did not answer. The extraction is deliberately lenient — fenced or bare JSON, last object wins, stray prose forgiven — so nothing scores unparseable on a technicality.

What actually ran: recording provider reality

The protocol pins temperature to 0 and a fixed output cap. The providers had other ideas, and the runner's rule is that a published artifact records what actually ran — never the settings we wished we'd used. Three deviations, all visible in the artifacts' params blocks:

One more honesty mechanism: four Opus cases died to transient network failures mid-run, after the runner's five retries. A network failure is not a model verdict, so the runner records such cases in the artifact as erroredCases, and a --heal flag re-runs exactly those cases with the same parameters and recomputes the summaries, stamping healedAt. Healing is restricted by construction to infrastructure errors: a delivered response that was wrong, or contained no JSON, is a model verdict and is never re-run. No reroll-until-it-passes, ever.

What this means if you ship a copilot

FinPlanBench is the generous setting. The model gets the entire spec — no recall, no retrieval, no ambiguity about which rule applies. A production copilot faces a strictly harder problem: it has to pick the rule, remember the current-year constants, and then execute. On the generous setting, the best run on the board still missed 12 of 150 cases at the tolerance our CI holds the engine to, and the most consistent run was exact 20% of the time. That is the measured case for not making the language model do the arithmetic. Let the model do what it did well here — read the spec, structure the problem, say the answer — and let a deterministic, dual-implemented engine produce the number, over the API or the MCP server. Then grade what ships: the same engine backs a verification endpoint that takes a copilot's claims and returns per-claim verdicts, in production, not just at eval time.

Run it, reproduce it, beat it

The benchmark page rendered an honest empty state before the first artifact landed — "No published runs yet — the harness is, though" — because no number appears there unless a committed runner artifact produced it. That rule survives contact with a populated leaderboard: results are never edited by hand, partial runs are watermarked and refused, and when a spec version bumps, the case-set hash moves and stale runs are flagged rather than silently compared. Reproducing a score takes one command and your own API key:

git clone https://github.com/CapsteraSupport/worthune && cd worthune && pnpm install
ANTHROPIC_API_KEY=sk-… pnpm exec tsx scripts/bench/run-finplanbench.ts \
  --provider anthropic --model <model-id> --label "<display name>"
# OpenAI-compatible endpoints: --provider openai [--base-url …] [--key-env …]
# The artifact lands in bench/results/ with per-case verdicts and the
# case-set hash. Open a PR to publish it — keep the raw responses.