FinPlanBench v1.0.0
Hand a language model the spec. See if it can do the math.
FinPlanBench is an open benchmark for personal-finance computation. Each task gives a model one published Worthune specification — formulas included, so knowing the rule is not the test — and one set of inputs, and asks for the exact scalar outputs. Ground truth is the Concordance-tested engine: every expected value is produced by two independent implementations that agree at one part in a billion in CI.
Published runs
| Model | Strict (harness 1e-9) | Practical (≤1% / $1) | Unparseable | Run |
|---|---|---|---|---|
| Gemini 3.7 Flashran against a previous case set | 20.0% | 98.7% | 0 | 2026-08-16 · 150 cases |
| Claude Opus 5ran against a previous case set | 92.0% | 96.0% | 6 | 2026-08-16 · 150 cases |
| GPT-5.5ran against a previous case set | 89.3% | 95.3% | 6 | 2026-08-16 · 150 cases |
| Claude Sonnet 5ran against a previous case set | 32.0% | 88.0% | 18 | 2026-08-16 · 150 cases |
| Gemini 3.1 Pro (preview)ran against a previous case set | 22.7% | 79.3% | 9 | 2026-08-16 · 150 cases |
| DeepSeek V4 Flash (effort: low)ran against a previous case set | 62.7% | 70.0% | 44 | 2026-08-16 · 150 cases |
| DeepSeek V4 Pro (effort: low)ran against a previous case set | 54.7% | 62.0% | 57 | 2026-08-16 · 150 cases |
| Claude Haiku 4.5ran against a previous case set | 15.3% | 18.0% | 0 | 2026-08-16 · 150 cases |
| GPT-5.4 miniran against a previous case set | 15.3% | 16.0% | 1 | 2026-08-16 · 150 cases |
A case passes only if every scalar output agrees — partial credit hides exactly the errors that matter. “Practical” is the loosest bar our grading accepts: within 1% relative or $1 absolute — close enough to say to a user, still graded.
Ninth-digit drift
The method is right and the arithmetic is nearly right. These land inside the practical bar and outside the strict one — harmless in conversation, wrong in a schedule that compounds.
Always close, never exact
A model can sit within a cent of the answer on almost every case and be exactly right on a fifth of them. It reads as competence and behaves as noise.
Wrong computation
The spec was in the prompt and the model still applied a different rule. No tolerance saves these, and no amount of fluency reveals them to a user.
The launch write-up walks each mode through real cases — and if you’d rather test your own engine than a leaderboard model, grade it against the same cases.
The method, in full
Open-book, on purpose
The prompt contains the entire published spec. The benchmark isolates the failure mode that matters to anyone shipping a financial copilot: not whether the model knows the rule, but whether it can execute a rule it has in front of it.
Fully reproducible
Built only from the three free-sample models, whose specs and eval datasets are fully public: emergency-fund v1.0.0, relocation v1.0.0, side-business v1.0.1 — 50 deterministic cases each, 150 total. No key needed to regenerate the set or check any expected value.
Fixed protocol
One fixed prompt template (versioned), scalar outputs only — reproducing a 30-row projection table measures transcription, not computation. Temperature is pinned to 0 where the provider’s API accepts it; where it doesn’t (reasoning models often force their default), the default runs and the artifact records exactly that. Answers are parsed leniently (fenced or bare JSON, last object wins); a response with no JSON object at all scores as unparseable, against the model.
Pinned by hash
Every published run names the exact case set it ran against. Current case-set SHA-256: 54e0715af3647b3655a23e0e805ca2b8a035647a36c1ec271d1e5379150b3997. When a spec version bumps, the hash moves, and older runs are flagged rather than silently compared.
Run it yourself
git clone https://github.com/CapsteraSupport/worthune && cd worthune && pnpm install ANTHROPIC_API_KEY=sk-… pnpm exec tsx scripts/bench/run-finplanbench.ts \ --provider anthropic --model <model-id> --label "<display name>" # OpenAI-compatible endpoints: --provider openai [--base-url …] [--key-env …]
The runner writes a result artifact with the case-set hash, the per-case verdicts, and the run parameters. Open a pull request adding the file to publish it here — keep your raw responses, we spot-check before merging. And if you’re building the copilot rather than the leaderboard: the graded fix is the verification endpoint — same engine, per-claim verdicts, in production.
Questions
Common questions
- What benchmarks exist for financial reasoning in language models?
- FinPlanBench is an open benchmark for personal-finance computation specifically: a model is handed one published specification — formulas included, so knowing the rule is not the test — plus a set of inputs, and must return the exact scalar outputs. Specs, cases, and the runner are all public, so a result is reproducible by anyone.
- How is the ground truth established?
- Every expected value comes from the Concordance-tested engine: two independent implementations, written separately from the same specification, that agree to one part in a billion. That is a stronger guarantee than a benchmark whose answers were written by hand or produced by one implementation nobody checked.
- Why give the model the formulas?
- Because recall is not what fails. Language models can usually state the amortization formula; what they get wrong is executing it over several steps without drifting. Handing over the specification removes memorization as a variable and isolates the thing being measured, which is arithmetic under composition.
- Can I submit a result for a model I ran?
- Yes. Run the public runner, which writes an artifact containing the case-set hash, the per-case verdicts, and the run parameters, then open a pull request adding that file. Keep your raw responses — submissions are spot-checked before merging. The leaderboard renders only committed artifacts, which is also why an empty leaderboard would render honestly.
- My copilot needs to be right in production, not on a leaderboard. What then?
- Use the verification endpoint. It takes the inputs and a numeric claim and returns verified, violated, or out-of-scope with the engine's computed value — the same engine the benchmark grades against, applied per claim, at runtime, before your assistant states the number.
Last reviewed