Eval Datasets for AI Teams: Ground Truth You Did Not Have to Build

The scarcest resource in AI evaluation is trustworthy ground truth. For financial computation, it is sitting at a public endpoint — free today, no key, no license negotiation.

Every team building a financial assistant eventually needs the same artifact: a set of questions with answers they can grade against without arguing about the answers. Building one in-house means becoming a financial-math shop with a verification problem — the exact burden the assistant was supposed to avoid. Worthune's eval datasets are that artifact, already built and already public: for every model in the catalog, GET /api/v1/evals/{model} returns the verification vectors the platform's own release gate runs. This piece is about what is in them and how an AI team, specifically, puts them to work.

What is in a dataset, and why the answers are trustworthy

Each dataset carries its model name, the specVersion the vectors describe, the case count — 250 per model — and the numeric tolerance (relative 1e-9, absolute 1e-6 per field), followed by the cases: an inputs object and its expected outputs, including sentinel cases where the honest answer is an annotated extreme. The trust argument is structural, not reputational. The expected outputs were computed by the production engine and independently reproduced by a second implementation written from the published spec alone; releases gate on their agreement. The vectors are generated deterministically from each model's published input domains, so the dataset is reproducible, and the composition — seeded fuzz across the domain plus curated boundary and sentinel cases — means it exercises the edges an assistant will actually stumble on, not just the comfortable middle.

The terms take one sentence: the datasets are free and public today, covered by the written free-tier commitment, and the ask is credit when you publish results built on them.

From vectors to an assistant benchmark

The datasets grade implementations out of the box; grading an assistant takes one transformation. A vector's inputs become a natural-language scenario — the refinance case's balance, rates, and closing costs become a user asking whether refinancing makes sense — and the expected outputs become the grading key. Run the scenario through your assistant and grade three things separately: extraction (did the tool call's inputs match the vector's inputs), computation (did the assistant use the tool at all — with the tool, the outputs match by construction), and fidelity (did the reply's numbers match the envelope's). The decomposition matters because the three failures have three different fixes — prompt work, routing work, and template work — and a single pass-fail score hides which one you have.

Regression discipline for a moving stack

An assistant stack changes constantly — model upgrades, prompt edits, tool-schema tweaks — and each change can silently move numeric behavior. The datasets make the regression suite cheap: pin the specVersion from each dataset's metadata, re-run your scenario set on every stack change, and diff the three grading layers. Because the ground truth is external, versioned, and shared, scores are comparable across time and across teams — when the dataset's specVersion moves, the changelog explains why, and your benchmark updates with a citation instead of a shrug. Teams that publish assistant evaluations get the same property in public: results built on named, versioned, downloadable vectors can be reproduced by a skeptical reader, which is what separates a benchmark from a brag.

• ✓Pull the datasets per model — metadata, tolerance, and 250 cases in one request.
• ✓Grade extraction, tool use, and reply fidelity as separate scores.
• ✓Weight the sentinel cases — they are where assistants soften the truth.
• ✓Pin the specVersion and re-run on every stack change.
• ✓Credit the datasets when you publish results.

The quiet strategic point

Ground truth this cheap changes who can be rigorous. Assistant evaluation with trustworthy financial answers used to require either a quant team or a leap of faith; a public, versioned, dual-implementation-verified vector set moves the rigor within reach of any team with an afternoon. The reproduction guide (/writing/reproducing-a-spec-evals) covers what the datasets prove about implementations; this piece's claim is narrower and more practical — for AI teams, the hardest part of the benchmark was already built. The rest is your scenario writing.