Your Fintech Copilot Has No Answer Key

A user asks your copilot: "We're six years into a 30-year mortgage at 6.9% — if we refinance the remaining balance at 5.6% with $6,000 in closing costs, when do we break even?" The copilot answers in two confident sentences and quotes a month count. Here is the eval question that matters: which test in your suite fails if that number is wrong?

For most teams shipping financial copilots, the honest answer is none. The eval stack is real — format checks, safety and refusal graders, groundedness scoring against retrieved context, an LLM-as-judge pass for helpfulness. All of it grades the text. None of it computes anything. The gap is structural: every layer judges properties you can assess by reading the answer, and arithmetic correctness is not one of them. The only way to grade a computed number is to compute it independently.

This post is about closing that gap with an answer key you can wire into an existing eval harness in an afternoon. Two paths: download deterministic input/expected-output datasets from GET /api/v1/evals/{model} and diff locally, or the newer, shorter path — POST /api/v1/grade/{model} your copilot's answers and let the server grade them at the same tolerances our own CI holds.

What the stack grades, and what it can't

Lay out a typical eval pipeline for an assistant that talks about money, and ask each layer the refinance question:

Every grader in a typical eval stack reads the answer. None of them can compute it.

The last row is the missing piece, and it has hard requirements: deterministic inputs, expected outputs somebody actually stands behind, an explicit numeric tolerance, and a machine-checkable encoding for the edge cases where the right answer is "never" rather than a number. That is a different artifact from a golden-transcript file, and you cannot produce it by having annotators label responses.

Where the expected outputs come from

An answer key is only as good as its answers, so here is exactly how ours are produced. Every Worthune model is written to a versioned published spec and released through Concordance testing: two implementations, written independently from the same spec — the production engine, and a second implementation built from the spec text alone — must agree on 250 cases per model at a relative tolerance of 1e-9 (one part in a billion) and an absolute tolerance of 1e-6 before anything ships. When the two disagree, the release is blocked until we find out which one misread the spec — or whether the spec itself was ambiguous.

The eval datasets are those same 250 cases per model, not a marketing export of them. The route that serves GET /api/v1/evals/{model} calls the identical generateVectors() code path the CI exporter uses: a PRNG seeded per model from a fixed string, so generation is deterministic and a download is byte-stable for a given spec version. For models with scalar input domains, the first two cases are the domain corners (every input at its minimum, then every input at its maximum), followed by hand-curated edge cases — each with a written rationale, like refinance's "negative savings (rate goes up) → Infinity break-even" — and then seeded random cases filling out the 250. Pin a dataset in CI and it will not shift under you; when a spec version changes, the public changelog says what moved and why.

Path one: download the dataset and diff locally

The dataset format is JSONL. Line 1 is a meta record carrying the model name, spec version, case count, the harness tolerances, a provenance statement, and the spec URL. Lines 2 through 251 are cases: {"id": "emergency-fund-0007", "inputs": {...}, "expected": {...}}. Feed each case's inputs to your copilot (the input contract tells you what every field means), parse the numbers out of its answer, and compare at tolerance:

import json, urllib.request

REL, ABS = 1e-9, 1e-6  # the dataset's meta line carries these too

def close(exp, got):
    # Non-finite expected values arrive as strings per the spec convention.
    if isinstance(exp, str) and exp in ("Infinity", "-Infinity", "NaN"):
        return got == exp
    if isinstance(exp, (int, float)) and not isinstance(exp, bool):
        return (isinstance(got, (int, float))
                and abs(exp - got) <= ABS + REL * abs(exp))
    if isinstance(exp, list):
        return (isinstance(got, list) and len(exp) == len(got)
                and all(close(e, g) for e, g in zip(exp, got)))
    if isinstance(exp, dict):
        return (isinstance(got, dict)
                and all(close(v, got.get(k)) for k, v in exp.items()))
    return exp == got  # strings, booleans: exact

url = "https://worthune.com/api/v1/evals/emergency-fund"
lines = urllib.request.urlopen(url).read().decode().splitlines()
meta, cases = json.loads(lines[0]), [json.loads(l) for l in lines[1:]]
assert meta["count"] == len(cases) == 250

failed = [c["id"] for c in cases
          if not close(c["expected"], my_copilot(c["inputs"]))]  # your model here
print(f"{len(cases) - len(failed)}/250 pass "
      f"(specVersion {meta['specVersion']}); first failures: {failed[:5]}")

That is the whole harness. The interesting engineering is on your side of my_copilot — whether you have the copilot answer in free text and extract the numbers, or force a structured output that mirrors the model's output contract. We'd argue for the structured route: if your copilot quotes a break-even to users, it should be able to state that number as a field, not just as prose.

Path two: POST your copilot's answers for grading

The newer, shorter path skips the local comparator entirely. POST /api/v1/grade/{model} takes your cases — each one an inputs object and your copilot's answer object — and grades them server-side against the production engine, computed at request time, at the harness tolerances. Up to 500 cases per request; the three sample models grade without a key, the rest with Pro.

curl -s -X POST https://worthune.com/api/v1/grade/relocation \
  -H 'content-type: application/json' \
  -d '{
    "cases": [
      { "inputs": { "currentSalary": 95000, "newSalary": 120000,
                    "currentMonthlyExpenses": 2800, "newMonthlyExpenses": 4100,
                    "movingCosts": 12000, "currentSavings": 40000,
                    "annualReturn": 0.05, "yearsHorizon": 10 },
        "answer": { "breakEvenMonths": 16, "annualNetDelta": 9400 } },
      { "inputs": { "currentSalary": 150000, "newSalary": 110000,
                    "currentMonthlyExpenses": 3000, "newMonthlyExpenses": 4000,
                    "movingCosts": 15000, "currentSavings": 80000,
                    "annualReturn": 0.05, "yearsHorizon": 10 },
        "answer": { "breakEvenMonths": 84 } }
    ],
    "tolerance": { "rel": 1e-6 }
  }'

Each case comes back with a verdict: pass, fail, or invalid-inputs. A fail carries per-key diffs — a dot/bracket path into the outputs like yearlyData[3].move, the expected value, your value, and a note: value-mismatch, missing, or type-mismatch (capped at 20 diffs per case so responses stay small). invalid-inputs means your inputs violated the model's published contract — out-of-domain values are rejected with the validation errors, never silently clamped — and those cases are excluded from the batch passRate, which is passed / (passed + failed). The response echoes the specVersion it graded against and the exact tolerance it applied.

In the request above, both cases fail — instructively. The first copilot got its two numbers right (the break-even genuinely is 16 months) but answered only two of the nine keys relocation's contract defines, so the grader returns seven missing diffs: an answer key grades completeness, not just the headline number. The second case is a pay cut into a more expensive city — the monthly delta is negative, the expected breakEvenMonths is the documented 9999 sentinel, and the confident-sounding "84 months" comes back as a value-mismatch. That second failure mode — a plausible number where the right answer is "never" — is precisely the one no text-side grader will ever catch.

Two boundaries worth knowing. Extra keys in your answer are ignored — we grade what the spec defines, so your copilot can return richer objects without penalty. And the tolerance override is clamped on both ends: you can loosen grading for a copilot that rounds for humans, but never looser than 1e-2 relative (or 1 absolute), and never tighter than the 1e-9 / 1e-6 the harness itself runs. Past 1e-2, "graded against the spec" stops meaning anything, so the server won't let anyone claim it. String outputs — recommendation fields like "invest" versus "split" — are graded exact-match.

What this grades, and what it doesn't

Grading covers numeric agreement with a published, versioned model spec — every grade response says so in its disclaimer field. It does not grade whether refinancing was the right thing to recommend to that user, whether the copilot's tone was appropriate, or whether it should have answered at all. Keep your existing eval layers for those; this is the one layer they cannot provide. The provenance statement in every dataset makes a deliberately structural claim, too: the served cases are generated by the same code path the Concordance harness exports for CI. It does not assert a live CI status, because a static "green" would keep asserting itself even on a red run.

The afternoon version: pick an open dataset, run the 250 cases through your copilot, and diff at tolerance — or skip the comparator and POST the answers to the grade endpoint. Either way, the next time your copilot quotes a break-even, some test in your suite actually knows whether it's right.