Eval Datasets for AI Teams: Ground Truth You Did Not Have to Build

The scarcest resource in AI evaluation is trustworthy ground truth. For financial computation, it is sitting at an endpoint: the three sample models' datasets are fully public — free, no key, no license negotiation — and the rest of the catalog ships with a Pro key.

Every team building a financial assistant eventually needs the same artifact: a set of questions with answers they can grade against without arguing about the answers. Building one in-house means becoming a financial-math shop with a verification problem — the exact burden the assistant was supposed to avoid. Worthune's eval datasets are that artifact, already built: GET /api/v1/evals/{model} returns the eval vectors the platform's own Concordance release gate runs. The three sample models — emergency-fund, relocation, and side-business — are fully public, no key required; the rest of the catalog's datasets ship with a Pro key, and every model's input contract is public either way. This piece is about what is in the datasets and how an AI team, specifically, puts them to work.

What is in a dataset, and why the answers are trustworthy

Each dataset carries its model name, the specVersion the vectors describe, the case count — 250 per model — and the numeric tolerance (relative 1e-9, absolute 1e-6 per field), followed by the cases: an inputs object and its expected outputs, including sentinel cases where the honest answer is an annotated extreme. The trust argument is structural, not reputational. The expected outputs were computed by the production engine and independently reproduced by a second implementation written from the published spec alone; releases gate on their agreement — the Concordance test every model must pass. The vectors are generated deterministically from each model's published input domains, so the dataset is reproducible, and the composition — seeded fuzz across the domain plus curated boundary and sentinel cases — means it exercises the edges an assistant will actually stumble on, not just the comfortable middle.

The terms are short: the sample datasets are free and public today, covered by the written free-tier commitment, and the ask is credit when you publish results built on them; the full catalog's datasets come with a Pro key.

From vectors to an assistant benchmark

The datasets grade implementations out of the box; grading an assistant takes one transformation. A vector's inputs become a natural-language scenario — the refinance case's balance, rates, and closing costs become a user asking whether refinancing makes sense — and the expected outputs become the grading key. Run the scenario through your assistant and grade three things separately: extraction (did the tool call's inputs match the vector's inputs), computation (did the assistant use the tool at all — with the tool, the outputs match by construction), and fidelity (did the reply's numbers match the envelope's). The decomposition matters because the three failures have three different fixes — prompt work, routing work, and template work — and a single pass-fail score hides which one you have.

Regression discipline for a moving stack

An assistant stack changes constantly — model upgrades, prompt edits, tool-schema tweaks — and each change can silently move numeric behavior. The datasets make the regression suite cheap: pin the specVersion from each dataset's metadata, re-run your scenario set on every stack change, and diff the three grading layers. Because the ground truth is external, versioned, and shared, scores are comparable across time and across teams — when the dataset's specVersion moves, the changelog explains why, and your benchmark updates with a citation instead of a shrug. Teams that publish assistant evaluations get the same property in public: results built on named, versioned vectors can be reproduced by a skeptical reader — with no gate at all when the benchmark is built on the public sample datasets — which is what separates a benchmark from a brag.

• ✓Pull the datasets per model — metadata, tolerance, and 250 cases in one request (sample models keyless; the rest with a Pro key).
• ✓Grade extraction, tool use, and reply fidelity as separate scores.
• ✓Weight the sentinel cases — they are where assistants soften the truth.
• ✓Pin the specVersion and re-run on every stack change.
• ✓Credit the datasets when you publish results.

The quiet strategic point

Ground truth this cheap changes who can be rigorous. Assistant evaluation with trustworthy financial answers used to require either a quant team or a leap of faith; a versioned, Concordance-tested vector set — fully public for the sample models — moves the rigor within reach of any team with an afternoon. The reproduction guide (/writing/reproducing-a-spec-evals) covers what the datasets prove about implementations; this piece's claim is narrower and more practical — for AI teams, the hardest part of the benchmark was already built. The rest is your scenario writing.