Guide
The best eval datasets for financial AI
Financial eval datasets mostly test comprehension: FinanceBench asks questions over company filings, and FinQA pairs financial text and tables with multi-step arithmetic programs. They measure whether a model can find and reason about a figure. Very little tests pure computation — whether the arithmetic is exactly right given the inputs. Worthune's eval datasets sit in that gap, with expected values produced by two independent implementations agreeing at one part in a billion.
Written by Worthune. Last reviewed . We sell one of the things described here — the caveats section says where that shows.
Public datasets for evaluating financial AI, and what each measures:
- 01
FinQA
Best for numerical reasoning over reports
Pairs financial text and tables with multi-step arithmetic programs, so it tests both finding the numbers and operating on them. The most downloaded of the three public sets here.
Free to download. · 14,278 downloads, 28 likes
Best for retrieval-plus-reasoning over filings
Open-book question answering over public company filings — closest to what a research assistant actually does, and widely used.
Free to download, but noncommercial: CC BY-NC 4.0. · CC BY-NC 4.0 · 5,721 downloads, 138 likes
Best if you already run FLARE
A FinQA variant packaged for the FLARE evaluation suite, which saves the harness work if that is the framework you use.
Free to download. · 3,655 downloads, 3 likes
Best for testing computation rather than comprehension
Deterministic input and expected-output pairs per planning model, with no document to read — only whether the arithmetic is exactly right. Expected values come from two independent implementations that agree to one part in a billion.
Free with attribution for the open sets; subscription for the full catalog. · Free with attribution · 250 cases per model
Method
How we compared these
A list without a method is an opinion wearing a table’s clothes. These are the axes, including the one we are biased toward.
What capability it isolates
Retrieval, reasoning over a document, and raw computation fail differently and need different tests. A dataset that mixes them tells you something went wrong but not what.
Where the ground truth comes from
Answers written by hand, answers produced by one implementation, and answers produced by two independent implementations that agree are three different levels of confidence in the key you are grading against.
The license
This is the detail commercial teams miss most often. A noncommercial license constrains use inside a product's test suite, and it is not always mentioned where the dataset is discussed.
Whether it is reproducible
A case set that regenerates identically is one you can diff across releases. One that does not is a snapshot.
Read this part
Where this list is weakest
- One of these is ours, and we published the list. The categorization is the useful part; every figure is read from the source named below and dated.
- Download and like counts move. They are here because they are published and checkable, not as a ranking.
- This is not a complete survey of financial NLP benchmarks — it covers datasets a team choosing an eval set for a finance product would plausibly reach for.
- Nothing here tests whether an assistant gives good financial advice, which is a different and much harder question that we do not claim to have solved either.
Questions
Common questions
- Are there eval datasets for testing financial calculations in LLMs?
- Most public financial datasets test comprehension over documents rather than pure calculation. FinQA comes closest among the widely used sets, pairing tables and text with arithmetic programs. For isolating computation alone, Worthune publishes deterministic input and expected-output pairs per planning model with no document involved.
- Which financial eval datasets can I use commercially?
- Check each license rather than assuming. FinanceBench is published under CC BY-NC 4.0, which is noncommercial and constrains use inside a commercial product's test suite. Worthune's open sets are free with attribution. Where a dataset does not state a license clearly, treat that as a question for your counsel.
- How do I test whether my finance copilot's math is correct?
- Separate the two failure modes. Test comprehension with a document-based set like FinQA or FinanceBench. Test computation with deterministic input and expected-output pairs, graded at a tight tolerance — a copilot can read a filing correctly and still get the arithmetic wrong, and one number tells you which.
- What tolerance should I grade arithmetic at?
- Tight enough that a rounding bug fails. Worthune grades at one part in a billion relative with a small absolute floor for values near zero, which is the same bar its own verification harness holds. Grading loosely and calling it a pass is how a real error survives to production.
Sources
- FinQA — huggingface.co/api/datasets/dreamerdeo/finqa, checked
- FinanceBench (Patronus AI) — huggingface.co/api/datasets/PatronusAI/financebench, checked — The noncommercial license is the detail teams miss: it constrains use inside a commercial product's test suite. Check it against your situation.
- FLARE FinQA (ChanceFocus) — huggingface.co/api/datasets/ChanceFocus/flare-finqa, checked
- Worthune eval datasets (Worthune) — worthune.com/evals, checked — Ours. Computation rather than comprehension: there is no document to read, only whether the arithmetic is exactly right.
Product names and trademarks belong to their owners and are used here only to identify what is being described. Nothing here implies endorsement or affiliation. Found something out of date? Tell us at support@worthune.com and we will re-check it.