Verify benchmark

How the test works

What the 820 test items of Verify’s benchmark are, how each was built, what “flawed” means, how the test set was kept apart from the work it measures, and how to rerun it yourself. The homepage’s figure comes from a newer, fresh test: Verify compared with other checkers.

Full results, including everything Verify gets wrong: /verify/benchmark

What the 820 test items are

Each item is a customer-style question, a one-sentence answer that makes one checkable statement, and two passages: the one the answer should rest on, and a distractor on the same subject (as a real search system would also return).

• 631 come from U.S. federal regulations (the eCFR, 1 October 2026 edition): consumer credit, payments, mortgage servicing, air travel, family leave, benefit claims, health privacy, workplace safety and similar rules from the FTC, CFPB, DOT, FAA, DOL, HHS and others. A script built them.
• 189 were written by hand about fictional companies: return pages, pricing pages, product documentation and help articles.

Every item starts from a sentence the passage states, then makes exactly one change whose effect is known in advance:

The hand-written items add the harder cases: paraphrases, swapped names, comparisons and predictions the passages cannot settle, two passages that disagree, a newer passage that replaces an older one, and answers that need a step the passages never give.

Labels come from how each item was built, not from anyone’s judgement. We know the right answer because we made the change. No AI model labels anything, and the generator uses none of Verify’s own code. When a change might be ambiguous (for example, the next sentence may limit the statement), the item goes to a review queue and is left out of every score: 632 items across both halves of the benchmark are there now. The full construction rules ship with the benchmark package.

What “flawed” means

A flawed answer is one the supplied passages do not support as written. 612 of the 820 test answers are flawed:

• 282 contradict the passages: 126 give a different number, 125 flip a “may” or “must”, 21 give a different date, 10 other.
• 154 drop a condition, exception or time limit that the passage attaches.
• 154 say something the passages do not: 144 on a point the passages never address, 10 that need a missing step or apply a rule to the wrong group.
• 22 cannot be settled from the passages: comparisons, predictions, or two passages that disagree.

“Wrongly approved” means Verify marked a flawed answer as supported by the passages. How many it wrongly approved at each rules version is on /verify/benchmark. The other 208 answers are supported; Verify let 143 of them through and sent the rest to a person. That caution is its main cost: overall it sent 264 of 820 answers (32.2%) to a person rather than decide.

Supported means supported by the supplied evidence, not true in the world.

How the test set was locked

• Split by a hash, not by hand. All items made from the same sentence form a family, and a hash of the family’s name puts it on the development side (60%, 1,220 items) or the test side (40%, 820 items). Variants of one sentence never straddle the two.
• Method fixed first. Before any system read the test side, the method write-up was committed and its fingerprint recorded (SHA-256 4c8cb6050ea42e1f106a9945fe181b1472497e7193258c15654a1ae357a56448, commit cb842bb, 2026-10-18). The code that loads the test items refuses to run without that fingerprint. The test file’s own fingerprint is 0ee369192836b1704d10f7fae10291e625a7985e48b91f94461a9d5ed820dcb0, so any change to it would show.
• Rules are developed on the development side only. Each new rules version is frozen and committed, then scored on the test side once. A version is never adjusted after seeing its test score; a fix becomes a new version.
• Scored 8 times so far, once per rules version: 1.0.0, 1.1.0, 1.2.0, 1.3.0, 2.0.0, 2.1.0, 3.0.0, 4.0.0. Separately, a regression check compares Verify’s responses on these items with a stored snapshot (it computes no score); for 2.1.0 it ran once before the freeze, which the changelog records.

8scorings is enough that this test set is no longer untouched: each rules version was written by people who had seen the earlier versions’ test scores, even though no version was tuned on its own. That is why its items and labels are now public, and why the next measurement needs new items.

Fresh test sets

The first fresh set is 300 answers built the same way from regulations and made-up support policies not used before. Its fingerprint was committed before any system ran on it, and it was scored once, on 2026-10-19. Code-only Verify let through 8 of 220 flawed answers (3.6%) there, far more than on the 820-item set (figures on /verify/benchmark). That is over the 2% failure line we set for Verify in advance. The full result, beside AI judges and other checkers, with every loss: /verify/benchmark/compare.

Benchmark v1.2 adds a second fresh test set. Its fingerprint is committed before any system is run on it, its labels are not published, and it is scored once by the rules version current at the time.

Rerun it yourself

One command, no account, no API key, no AI model. It takes a few seconds on a laptop (Node.js 20 or later):

npx -y -p @xyntherium/verify-core@1.1.0 -p @xyntherium/verify-bench@1.1.0 verify-bench-reproduce

The packages are being published to npm. Until they are, this command will not resolve; this note goes away when it does.

It runs the published rules (verify-core 1.1.0 is rules 2.1.0) on all 2,040 items, scores them with the published scorer, and checks every prediction against the published run files. It prints the held-out figures for the installed rules version (the same figures as on /verify/benchmark), then one line per published rules version saying how many predictions differ, and ends with “Reproduced: rules 2.1.0 matches its published run exactly.” when they all match.

It exits with an error if the installed rules do not reproduce their published run. The scorer, the items and the run files ship inside the package.