Two Implementations or It Doesn't Ship

How do you know a financial calculator is right? Not looks-right — compiles, passes the demo, nobody has complained — but right, in the sense that the number it prints is the number its documentation says it computes. Most teams can't answer that, and the reason is structural: a single codebase produces exactly one kind of evidence about itself. It produced that number. A model that's wrong once in a thousand runs is indistinguishable from one that's never wrong, until the miss lands in front of a customer.

Worthune's answer is a rule we apply to all 47 models in the catalog: two implementations, or it doesn't ship. This post walks through the machinery behind that rule — what the specs look like, where the test cases come from, how the diff works, what checks both implementations against the outside world, and the finding from our first runs that we didn't expect.

Concordance testing, defined

We call the method Concordance testing: two implementations of every model, written independently from the same published spec, must agree on 250 cases per model at one-part-in-a-billion tolerance before any release. One implementation is the TypeScript engine that serves production (src/lib/engine.ts). The other is a Python oracle, rebuilt from the spec text alone, with no access to the engine's code. Across the catalog that's 47 models × 250 cases = 11,750 cases, re-diffed on every change. When the two disagree, the release stops.

The name is literal. Concordance is agreement between independent witnesses — and independence is the property that makes the agreement worth something. Two implementations that share code, or share an author who peeked, are one witness wearing two hats.

The spec is the contract

The spec — not the TypeScript — is the contract. Each model has one markdown file defining its inputs with units and valid domains, the exact computation, the output keys, and the assumptions and exclusions that are part of the deal. "Exact" means exact. The emergency-fund spec doesn't say "simulate monthly growth until the target is reached." It says: months 0 through 240 inclusive, 241 iterations; each iteration records the projected balance, then the coverage ratio, then tests the pre-update balance against the target with a first-hit latch, then applies interest and the deposit — in that order. Deposits land end-of-month after interest, so a deposit that crosses the target counts in the following month. That ordering is observable in the outputs, so it's in the contract.

Specs are written as-implemented, quirks included. The same emergency-fund spec has a Known model issues section that states, among other things, that monthsToGoal = 240 is a sentinel collision — it means either "goal first reached at exactly month 240" or "never reached," and the two are indistinguishable — and that the chart arrays are hard-truncated at 61 elements. Documenting a quirk is not blessing it; it's refusing to fix it silently. Quirks are fixed through a versioned spec change first, then code, then a fresh harness run — never by an unannounced edit to the arithmetic.

Why this level of precision? Because the spec has a second reader with no fallback. A human integrating our API can ask us what we meant. The oracle can't.

An implementation that never saw the code

The Python side lives in oracle/models/, one module per model, each exposing compute(inputs, meta). These were written from the spec text alone — the authors were barred from reading the TypeScript engine and from reading the exported vectors. Different language, different author, different standard library. That's deliberate: a Python reimplementation can't accidentally inherit a TypeScript bug through copy-paste, and it can't paper over a gap in the spec by glancing at what the engine "really" does. If the spec is ambiguous, the two builds diverge. That divergence is the product working as designed.

Where the 250 cases come from

Each model's 250 cases are generated by one shared code path (src/lib/model-api/vectors.ts) and come from three sources, in a fixed order:

The fuzz is deterministic: a mulberry32 PRNG seeded per model from a fixed string hash, so re-running the generator reproduces identical vectors for a given spec version. There is no wall-clock dependence anywhere in the vector set (the last of it was removed when divorce v1.2.0 and eldercare v1.1.0 pinned the projection year). Determinism is what lets the case set double as a published artifact: the eval datasets we serve at /api/v1/evals/{model} are generated by the same code path the harness exports from, so the datasets you can download are byte-identical to the cases the harness diffs. They aren't a sample of our testing; they are the testing.

Curated cases carry their rationale as data

The curated cases are where the interesting engineering lives, and each one records why it exists in a note that ships alongside the inputs. Two RSU cases pin bracket-edge behavior: 100 shares at $54 on a $100,000 salary puts total income at $105,400, inside the 22% bracket, so the entire proceeds slice is taxed at 0.22; the sibling case moves the share price to $58, total income $105,800, and only the $100 above the $105,700 boundary is taxed at 0.24. The pair checks continuity across the bracket edge — the exact property the model's v1.1.0 move from flat to progressive taxation had to get right.

Discontinuities get pinned from both sides. The SBA loan-cost model has fee tiers that step at statutory thresholds, so its curated set includes straddle pairs one dollar apart:

// src/lib/model-api/vectors.ts — sba-loan-cost, trimmed
{
  note: "Double discontinuity low side: $150,000 → 85% guaranty, 2% fee",
  inputs: { loanAmount: 150000, annualRate: 0.105, termMonths: 120,
            financeUpfrontFee: "no" },
},
{
  note: "…high side: $150,001 → 75% guaranty, 3% fee (fee jumps ~$825 up)",
  inputs: { loanAmount: 150001, annualRate: 0.105, termMonths: 120,
            financeUpfrontFee: "no" },
},

One more dollar of loan changes the guaranty percentage and the fee rate at once, and the fee jumps about $825 upward. A uniform sampler over a domain that runs to $5,000,000 would essentially never land on both sides of that dollar; the curated pair guarantees both implementations handle the step identically, forever. Other favorites: the debt-payoff case where minimum payments can't cover interest, which must produce the 600-month sentinel with exactly 601 schedule entries; the emergency-fund case that never reaches its goal, pinning the 240 sentinel and the 61-element array truncation; and a wedding pair where only debt-related fields differ under savings funding — two distinct inputs whose outputs must be identical, because the spec says those fields are dead in that branch.

The diff

The harness (oracle/compare.py) loads each model's exported vectors, runs the oracle's compute on every case, and compares recursively: dicts key by key — a missing or extra key is a failure, not a skip — lists element-wise including every chart array, and numbers through math.isclose:

# oracle/compare.py
REL_TOL = 1e-9
ABS_TOL = 1e-6

if isinstance(expected, (int, float)) and not isinstance(expected, bool) and \
   isinstance(actual, (int, float)) and not isinstance(actual, bool):
    if not math.isclose(expected, actual, rel_tol=REL_TOL, abs_tol=ABS_TOL):
        out.append(f"{path}: expected {expected!r}, got {actual!r}")
    return

Relative 1e-9 or absolute 1e-6, whichever is looser. Both implementations compute in IEEE-754 doubles, so genuine agreement is usually bit-exact; the tolerance exists to absorb benign operation-order differences, not to hide disagreement. Two conventions make the comparison well-defined at the edges. JSON has no representation for infinities or NaN, so exported vectors carry them as the strings "Infinity", "-Infinity", and "NaN"; the oracle returns math.inf and friends, and both sides sanitize to the string form before comparing. And formula fields — display strings for the UI — are stripped on both sides, because prose is presentation, not contract.

Two implementations can still both be wrong

Agreement between the engine and the oracle proves the engine matches its spec. It does not prove the spec matches the world — both implementations could faithfully share a misconception. So the harness also runs external anchors: closed-form textbook values that check the shared conventions against something neither implementation produced. A $300,000 thirty-year mortgage at 6% must amortize to $1,798.65 a month, within a cent. A $100 monthly annuity at 12% nominal, compounded monthly for ten years, must reach a future value of $23,003.87. These are small, but they tie the amortization and annuity conventions that nearly every model leans on to numbers you can find in any finance textbook — an authority outside the repo entirely.

CI is the gate

None of this depends on someone remembering to run it. The oracle.yml workflow runs on every pull request and every push to main: install, pnpm oracle:export to regenerate all 11,750 vectors through the production engine, then python3 -m oracle.compare to diff them against the oracle. The harness exits nonzero on any mismatch or any failed anchor, and a nonzero exit blocks the merge. There's no override path, no "known failures" list, no quarantine tier. The workflow's own header states the invariant: any mismatch means engine, spec, and oracle no longer tell the same story.

Every mismatch our first harness runs produced traced to spec ambiguity, never to arithmetic. The diff wasn't measuring the math. It was measuring the contract.

What the first runs actually found

Here's the finding we didn't fully expect, from the Phase 0 accuracy report covering the first ten models: every initial mismatch in the first harness run traced to spec ambiguity — mostly missing output-key enumerations — and never to math divergence. Not one arithmetic disagreement, in any model. The failures looked like an oracle emitting a key the engine didn't, or omitting an intermediate the engine exposed, because the spec hadn't said precisely which keys the output contains. Each time, the fix was to tighten the spec and re-run, and each pass through that loop left the contract more precise than the code review that preceded it.

In hindsight this is what the setup predicts. Two careful implementations of a precisely stated computation tend to agree; what independent reimplementation actually stress-tests is whether the statement is precise. That reframing is the real return on the method. The artifact that improves under Concordance testing is the spec — and the spec is the artifact our customers read, pin versions against, and hold us to. Arithmetic disagreements remain possible, and catching one is why the gate exists. But if you adopt this method expecting to catch subtle floating-point bugs, expect instead to discover every place your documentation was quietly relying on the reader's goodwill.

Run the diff yourself

A method like this is only worth writing about if you can check it. Three sample models — emergency-fund, relocation, and side-business — have fully public specs and open eval datasets, no key required. Every model in the catalog has a public input contract. The remaining specs and datasets come with a Pro key. Each dataset is JSONL: a meta line with the model and spec version, then one line per case with inputs and expected outputs, non-finite values string-encoded as described above.

Which means you can be the third implementation. Pull a dataset, run the cases through your own integration — or your own from-scratch build of the spec, which is the fun version — and diff at the same tolerances our CI uses: relative 1e-9, absolute 1e-6. If you find a case where the published vectors and the published spec disagree, that's a bug in our contract, and we want it. That's the posture the rule commits us to: not "trust us," but "here is everything you need to check."