every model spec’d & versioned · Concordance-tested 2×changelog →
Worthune

Worthune Evals

The answer key your eval harness is missing

15,250 input/expected-output pairs across 61 financial models — the exact cases our Concordance harness runs on every change. Download them and diff offline, or post your answers and let us grade them at the tolerances we hold ourselves to. Three datasets are open; the full set comes with a paid key.

Watch the film — The Answer Key, 2½ minutes

The Answer Key · 2:31 — Free ground truth for financial AI: the 250-case verification datasets our CI runs, as JSONL with the same tolerances, pinned to spec versions.

The problem with testing money math

When your copilot answers “when does this refinance break even?”, what checks the number? Most eval stacks can grade tone, structure, and safety — but they have nothing that knows the right answer to a financial computation. So teams grade fluency and hope the arithmetic holds.

These datasets are the missing answer key. Every expected output comes from two independent implementations that agree to one part in a billion — the engine that computed it, and a second implementation written from the published spec alone, re-checked on every change. If your model, agent, or engine disagrees with a case, start by diffing against the spec: how Concordance testing works.

Your refinance breakseven in −4 months!Let's check that.−4ANSWERKEY250 open cases per model
confidence is not verification

Use it in an afternoon

# Grab a dataset (JSONL: meta line, then one case per line)
curl -s https://worthune.com/api/v1/evals/refinance | head -3

# Every case: {"id", "inputs", "expected"}
# Meta line carries specVersion + tolerances (rel 1e-9, abs 1e-6)
# The eval loop, in pseudocode
for case in dataset:
    answer = your_model_or_agent(case.inputs)   # LLM, agent, your own engine
    for key, expected in case.expected.items():
        assert close(answer[key], expected,
                     rel=1e-9, abs=1e-6)        # same bar our CI holds

Datasets are deterministic for a given spec version — pin one in CI and it never shifts under you. When a spec changes, the changelog says exactly what moved and why. Sentinel and non-finite values are encoded as strings (“Infinity”, “NaN”) per the spec convention.

61 datasets, 250 cases each

ModelSpecDownload
relocationv1.0.0JSONL (250 cases) — open
rent-vs-buyv1.1.0250 cases — with your paid key
firev1.0.1250 cases — with your paid key
roth-vs-traditionalv1.0.1250 cases — with your paid key
refinancev1.0.0250 cases — with your paid key
windfallv1.0.1250 cases — with your paid key
sabbaticalv1.0.0250 cases — with your paid key
one-incomev1.0.0250 cases — with your paid key
second-homev1.1.0250 cases — with your paid key
divorcev1.2.0250 cases — with your paid key
down-paymentv1.0.0250 cases — with your paid key
rsuv1.1.0250 cases — with your paid key
debt-payoffv1.1.0250 cases — with your paid key
student-loan-payoffv1.0.0250 cases — with your paid key
emergency-fundv1.0.0JSONL (250 cases) — open
salary-jumpv1.0.1250 cases — with your paid key
having-a-childv1.1.0250 cases — with your paid key
weddingv1.1.0250 cases — with your paid key
renovationv1.1.0250 cases — with your paid key
side-businessv1.0.1JSONL (250 cases) — open
mega-backdoor-rothv1.0.0250 cases — with your paid key
529-college-savingsv1.0.0250 cases — with your paid key
private-schoolv1.2.0250 cases — with your paid key
social-security-timingv1.1.0250 cases — with your paid key
long-term-carev1.1.0250 cases — with your paid key
eldercarev1.1.0250 cases — with your paid key
estate-planningv1.1.0250 cases — with your paid key
auto-loanv1.0.0250 cases — with your paid key
loan-comparisonv1.0.0250 cases — with your paid key
extra-paymentv1.0.0250 cases — with your paid key
biweekly-mortgagev1.0.0250 cases — with your paid key
points-break-evenv1.0.0250 cases — with your paid key
pmi-removalv1.0.0250 cases — with your paid key
arm-vs-fixedv1.0.0250 cases — with your paid key
credit-card-minimumv1.0.0250 cases — with your paid key
rmdv1.0.0250 cases — with your paid key
roth-conversion-ladderv1.0.0250 cases — with your paid key
backdoor-rothv1.0.0250 cases — with your paid key
employer-matchv1.0.0250 cases — with your paid key
capital-gainsv1.0.0250 cases — with your paid key
ssa-piav1.0.0250 cases — with your paid key
state-income-taxv1.1.0250 cases — with your paid key
quarterly-estimatesv1.0.0250 cases — with your paid key
esppv1.0.0250 cases — with your paid key
burn-rate-runwayv1.0.0250 cases — with your paid key
safe-dilutionv1.0.0250 cases — with your paid key
option-poolv1.0.0250 cases — with your paid key
equipment-lease-vs-buyv1.0.0250 cases — with your paid key
sba-loan-costv1.0.0250 cases — with your paid key
iso-exercisev1.0.0250 cases — with your paid key
qsbs-exclusionv1.0.0250 cases — with your paid key
83b-electionv1.0.0250 cases — with your paid key
wash-salev1.0.0250 cases — with your paid key
retirement-income-cliffsv1.0.0250 cases — with your paid key
longevity-horizonv1.0.0250 cases — with your paid key
annuity-payoutv1.0.0250 cases — with your paid key
life-insurance-needsv1.0.0250 cases — with your paid key
step-up-vs-gift-nowv1.0.0250 cases — with your paid key
inherited-ira-10-yearv1.0.0250 cases — with your paid key
qcdv1.0.0250 cases — with your paid key
plan529-superfundingv1.0.0250 cases — with your paid key

Don’t just download the answer key — send us your answers: POST /api/v1/grade/{model} grades your copilot’s outputs against the engine at the same tolerances (see the grading docs). The open case sets also power FinPlanBench, our reproducible benchmark for how well language models execute published financial math. Full index, machine-readable: /api/v1/evals. Open datasets are free with attribution; the full catalog downloads with a paid key (Authorization: Bearer wk_…), licensed for your internal testing per the terms.

Need evals we haven’t built?

We can build custom datasets for your product’s exact scenarios — written to a spec, Concordance-tested, and kept current across tax years, the same way we build our own. That’s an Enterprise conversation.

Talk to us

Questions

Common questions

Are there eval datasets for testing financial calculations in LLMs?
Yes. Worthune publishes deterministic input and expected-output pairs for its financial models — the same cases its own verification harness runs. Three datasets are open downloads with no signup; the full catalog comes with a paid key. Each expected value is produced by two independent implementations that agree to one part in a billion.
How do I test whether my finance copilot's math is correct?
Two ways. Download the datasets and diff your model's answers offline in your own harness, or post your answers to the grading endpoint and get back per-case pass or fail at the tolerances we hold ourselves to. The second is faster to try; the first is what you want in CI.
Where do the cases come from, and can I trust them?
They are generated deterministically per model: automatically derived corner cases at the edges of the valid domain, hand-written cases each pinning a specific boundary such as a bracket edge or a compensation cap, and seeded random fill. Expected outputs come from the production engine and are re-checked against an independent Python implementation in CI on every change.
What tolerance should I grade at?
The same one the harness holds: one part in a billion relative, with a small absolute floor for values near zero. Grading looser than that and calling it a pass is how a rounding bug survives to production. The grading endpoint accepts a tolerance override but clamps it, so nobody can grade sloppily and cite our bar.
Can I use these datasets in my own published benchmark?
Yes, free with attribution — credit Worthune when publishing results built on them. If you need private or custom datasets for models outside the open sample, that is a paid conversation.

Last reviewed