Reproducing a Model From Its Spec: The Eval Datasets

"Verified" is a checkable claim, not a badge. The eval datasets are the check: free, public, and byte-identical to what the platform's own harness verifies on every release.

The strongest thing a computational platform can say is not "trust our math" but "here is everything you need to catch us being wrong." Worthune's version of that sentence is the eval datasets: for every model in the catalog, GET /api/v1/evals/{model} returns the verification vectors — inputs paired with expected outputs — that the platform's own dual-implementation harness runs. This piece explains what is in a dataset, how to use one, and precisely what a successful reproduction does and does not prove.

What a dataset contains

Each dataset carries the model name, the specVersion the vectors were generated under, the case count, and the tolerance — then the cases themselves, each one an inputs object and its expected outputs. The vectors are 250 per model: curated boundary and sentinel cases the spec calls out, corner cases at the domain edges, and deterministic fuzz cases generated from the model's published domains filling the rest. The dataset's metadata states its own provenance: expected outputs computed by the verified engine and independently reproduced by a second implementation written from the published spec alone. That sentence is the claim; the download is what lets you test it.

The generation matters as much as the count. Because the fuzz vectors come from a deterministic generator seeded per model from a fixed string hash, re-running generation for a given spec version reproduces the identical dataset — which is what makes "the public datasets are byte-identical to what CI verified" a mechanical fact rather than a release-notes claim.

for each case: yours = your_implementation(case.inputs);  assert |yours − case.outputs| within tolerance (rel 1e-9, abs 1e-6, per numeric field)

Three uses, in ascending ambition

The first use is acceptance testing: before integrating a model, run a handful of vectors through the live API and confirm the answers match the dataset — a five-minute check that the model you evaluated is the model that is running. The second is independent reimplementation: build the model from its public spec in your own stack, verify against the full dataset, and you hold a validated internal implementation plus documented evidence that it matches a published contract — the reference-consumer pattern, and the strongest possible form of vendor-exit insurance. The third is auditing the platform itself: a reviewer who reimplements even one model from spec and hits agreement has direct evidence about the whole platform's discipline, because every model ships under the same regime.

The terms

The datasets are free and public today — no key, no signup, and the free tier's written commitment covers them. The one ask is credit: if you publish results built on the datasets — a benchmark, a reproduction study, a comparison — say where the vectors came from. Attribution is the entire price of the evidence.

• ✓Fetch /api/v1/evals/{model} — metadata plus 250 cases, one request.
• ✓Pin the specVersion in the metadata — vectors describe that version's behavior, not eternity.
• ✓Hold results to the stated tolerance, not to exact string equality.
• ✓Include the curated sentinel cases — they verify the annotations, not just the arithmetic.
• ✓Credit the source when you publish results — that is the whole license conversation.

Why give the test away

A skeptic might ask what a platform gains by publishing the means of checking it. The answer is that the alternative is worse for everyone: verification claims that cannot be independently exercised converge on marketing, and buyers discount them accordingly. Publishing the vectors converts "verified" from an adjective into a procedure — one that any integrator, auditor, or competitor can run on any Tuesday. A claim that survives that standing invitation is the only kind worth printing on a homepage.