FinPlanBench v1.0.0

Hand a language model the spec. See if it can do the math.

FinPlanBench is an open benchmark for personal-finance computation. Each task gives a model one published Worthune specification — formulas included, so knowing the rule is not the test — and one set of inputs, and asks for the exact scalar outputs. Ground truth is the Concordance-tested engine: every expected value is produced by two independent implementations that agree at one part in a billion in CI.

Published runs

No published runs yet — the harness is, though.

Every number that will ever appear here must come from a runner artifact committed to the repository with the case-set hash it ran against — we don’t publish scores we didn’t run, and we don’t hand-edit results. Run it against any model with your own API key in about ten minutes (instructions below), and submit the result.

A case passes only if every scalar output agrees — partial credit hides exactly the errors that matter. “Practical” is the loosest bar our grading accepts: within 1% relative or $1 absolute — close enough to say to a user, still graded.

The method, in full

Open-book, on purpose

The prompt contains the entire published spec. The benchmark isolates the failure mode that matters to anyone shipping a financial copilot: not whether the model knows the rule, but whether it can execute a rule it has in front of it.

Fully reproducible

Built only from the three free-sample models, whose specs and eval datasets are fully public: emergency-fund v1.0.0, relocation v1.0.0, side-business v1.0.1 — 50 deterministic cases each, 150 total. No key needed to regenerate the set or check any expected value.

Fixed protocol

Temperature 0, one fixed prompt template (versioned), scalar outputs only — reproducing a 30-row projection table measures transcription, not computation. Answers are parsed leniently (fenced or bare JSON, last object wins); a response with no JSON object at all scores as unparseable, against the model.

Pinned by hash

Every published run names the exact case set it ran against. Current case-set SHA-256: 54e0715af3647b3655a23e0e805ca2b8a035647a36c1ec271d1e5379150b3997. When a spec version bumps, the hash moves, and older runs are flagged rather than silently compared.

Run it yourself

git clone https://github.com/CapsteraSupport/worthune && cd worthune && pnpm install
ANTHROPIC_API_KEY=sk-… pnpm exec tsx scripts/bench/run-finplanbench.ts \
  --provider anthropic --model <model-id> --label "<display name>"
# OpenAI-compatible endpoints: --provider openai [--base-url …] [--key-env …]

The runner writes a result artifact with the case-set hash, the per-case verdicts, and the run parameters. Open a pull request adding the file to publish it here — keep your raw responses, we spot-check before merging. And if you’re building the copilot rather than the leaderboard: the graded fix is the verification endpoint — same engine, per-claim verdicts, in production.