The Annual Model Review, Run on Public Artifacts

Adoption-time review answers "should we use this?" The annual review answers a harder question: "is everything we decided then still true?" — and for a verified-model dependency, every fact needed to answer it is published.

Model governance traditions — supervisory guidance among them — converge on periodic review: models in use get re-examined on a cadence, not just at adoption. For internally built calculators that means budgeting real analyst time. For a dependency whose specs, change history, validation vectors, and constants are public, the annual review collapses into a structured read of artifacts plus a short memo — light enough that teams actually do it yearly, substantial enough that the memo means something. Here is the agenda.

The six questions and where their answers live

The known-issues row is the one reviewers from other vendor relationships find surprising. Worthune specs document their models' known issues in the spec itself — including unflattering ones — because a limitation stated in the contract is reviewable, while a limitation discovered in production is an incident. The annual review reads that section against your user population: an issue irrelevant at adoption can become material when your users change, and the review is where that intersection gets checked deliberately.

Running it, step by step

Start from your inventory entry — model, pinned spec version, date of last review. Read the changelog delta since that date; for each entry, note whether your product surfaced the changed outputs. Diff the current contract against the inputs your integration sends (the machine contract makes this scriptable). Check each consumed registry row's verifiedOn against its document cycle — fall figures re-verified after the fall wave, spring HSA figures after theirs. Re-read the known-issues section with fresh eyes. For models that carry material weight in your product, re-run the eval dataset against the live API and file the pass. Then sample your own traces for scope drift. None of these steps requires contacting the vendor, which is the property that makes the review durable: it cannot be blocked, softened, or scheduled around anyone's availability.

What the annual review is not

It is not re-validation of arithmetic — the platform's harness re-verifies every model against its spec on every release, which is a stronger cadence than annual; your optional dataset re-run is a spot-check of the pipeline between you and it, not a re-derivation. It is not a substitute for the event-driven checks either: the CI fixture catches version drift the week it happens (/writing/version-pinning-ci), and the annual pass assumes those alarms worked — its job is the slower questions alarms cannot ask, about fit, scope, and accumulated change. And for supervised firms it is an input to the model-risk calendar, not the whole of it: the SR 11-7 mapping (/writing/sr-11-7-mapping) marks where the firm's own effective challenge stands apart from anything a reading list can supply.

• ✓Calendar one review per model family per year, owner named.
• ✓Read the changelog delta first — it frames everything else.
• ✓Script the contract diff; eyeball the known-issues section.
• ✓Check verifiedOn dates against each constant's document cycle.
• ✓Re-run eval datasets for material models; file the pass.
• ✓End with the one-page memo, signed and dated, filed with its predecessors.

The asymmetry worth appreciating

An annual review of a black-box vendor is a questionnaire and a hope. An annual review of a published-artifact vendor is a reading assignment with a paper trail — same governance obligation, entirely different evidentiary texture. The firm's memo cites URLs a reviewer can open; the vendor's discipline is checked rather than attested; and the whole exercise costs an afternoon because the artifacts were maintained all year by the party best positioned to maintain them. That is the trade the verified-model architecture offers governance teams: your review gets shallower to run precisely because the thing under review got deeper to inspect.