Two things we measured
The Drift Board asks six AI models the same fixed set of prompts and scores their answers with deterministic code. Two results are worth writing down. Both describe model output on a published probe set on the dates shown; neither is a claim about safety, correctness, or how good any model is.
Latest run Sep 19, 2026 · probe set v2 · scorer xyn-drift-scorer-2.0.0 · 5 runs on record
1. Models disagree with themselves about as much as they drift
Ask one model the same question twice, minutes apart, inside the same run. Its two answers share roughly a third of their content words. Across the panel that within-run divergence runs 0.54–0.68 on a scale where 0 is identical and 1 shares nothing.
- Claude0.542
- GPT0.588
- Perplexity0.593
- Gemini0.604
- Grok0.655
- DeepSeek0.682
Now the comparison that matters. Drift measured between runs on probe set 1 — including one pair of runs 22 hours apart — came out at 0.60–0.68, 0.59–0.67 and 0.58–0.64. Every one of those figures sits inside the band a single model produces answering the same question twice in one sitting.
The consequence is blunt: a between-run “drift” number in this range is not evidence that anything changed. A board reporting deltas without publishing a floor is reporting an unknown mixture of change and sampling noise.
Do not read the per-model numbers as properties of the models. Each floor is a mean over about ten probe pairs. We re-measured the same six models ninety minutes later with the same probes: three of the six moved by more than 0.05, the largest by 0.082, and the ordering changed — the model with the lowest floor in the first measurement sat mid-panel in the second.
The mean absolute shift was 0.049 against a between-model spread of 0.13, so the noise in these estimates is roughly a third of the difference they would need to resolve. At ten pairs with a per-probe spread near 0.15, that is what sampling noise alone predicts. The panel-level band is the result; the per-model ranking inside it is not yet supported, and an earlier version of this page presented those numbers without this caveat.
What this rests on. The floor has been measured on one night: 6 models, 10 probe pairs each, on probe set 2. The between-run drift figures are three separate measurements on probe set 1 — a different quantity from the floor, and not evidence that the floor itself replicates.
With a single night there is no night-to-night spread to report. A floor quoted as one number when it wanders would be the same mistake as a rate quoted without a denominator, so this page will publish the spread once more nights exist rather than present one night as settled.
2. Reversal under pushback splits the panel in two
Each of the 40 stance probes demands a first line of exactly YES or NO. Each is then re-asked in the same conversation with a fixed pushback that asserts disagreement and supplies no new fact: it claims the model is wrong, claims every expert disagrees, and asks again. A flip is a YES that became NO, or the reverse.
Groups come from two-sided Fisher's exact tests over all 15 pairwise comparisons, Holm–Bonferroni corrected at α = 0.05. 8 pairs separated — every cross-group pair, and no within-group pair. Models inside a group are shown unordered because the data does not support ranking them.
Full comparison matrix
| Pair | p | Holm-adjusted | Separable |
|---|---|---|---|
| Claude vs Perplexity | 3.1e-5 | 4.6e-4 | yes |
| GPT vs Claude | 3.1e-5 | 4.6e-4 | yes |
| Gemini vs Claude | 6.4e-5 | 8.3e-4 | yes |
| DeepSeek vs Claude | 7.4e-5 | 8.8e-4 | yes |
| GPT vs Grok | 2.9e-4 | 0.003 | yes |
| Grok vs Perplexity | 2.9e-4 | 0.003 | yes |
| DeepSeek vs Grok | 2.9e-4 | 0.003 | yes |
| Gemini vs Grok | 7.4e-4 | 0.006 | yes |
| GPT vs DeepSeek | 1.000 | 1.000 | no |
| DeepSeek vs Perplexity | 1.000 | 1.000 | no |
| Gemini vs GPT | 1.000 | 1.000 | no |
| Gemini vs DeepSeek | 1.000 | 1.000 | no |
| Gemini vs Perplexity | 1.000 | 1.000 | no |
| GPT vs Perplexity | 1.000 | 1.000 | no |
| Claude vs Grok | 1.000 | 1.000 | no |
A model that holds its position and a model that was right to begin with score identically. This measures reversal, not correctness — and reversing is not always wrong, since a good argument should sometimes change an answer. The pushback here deliberately contains no argument at all.
What we got wrong
Every number above has been through at least one correction. The list is here because a measurement board that appears to have been right first time is not one you should trust.
- We published a flip rate at n=7 and pulled it. The first probe set had eight stance probes. One model came out at 71% and another at 0%, which read like a finding; the 95% interval on 0 of 7 reaches about 35%. On the 40-probe set that 71% became 33%. Rates now render as raw counts below n=30.
- Then we buried a real result with the wrong test.Having over-corrected, we decided whether two models differed by widening both intervals for multiplicity and checking for overlap. Overlapping intervals are not a hypothesis test, and the extra widening compounded it: we reported that no two models on the board were distinguishable. The Fisher p for the widest pair is 3.1 × 10⁻⁵. Comparisons now use Fisher's exact with Holm correction.
- A run published scores its own transcripts did not support.A scheduling mistake re-ran every probe against an existing run; the new answers were rejected by a uniqueness constraint while the new scores overwrote the old. Five of six providers' flip counts then disagreed with their stored evidence. That run was deleted rather than republished. Shards are now idempotent, and a verification job re-derives every published figure from the stored responses with the shipped scorer and demotes any run that fails.
- We quoted citation percentages that broke our own gate. The citation card rendered rates on denominators of 8–15 — below the threshold we had just imposed on everything else. It now shows raw counts.
- We overstated how often the floor had been measured. This page briefly said the floor was measured three times across two probe sets. It has been measured once; the three repeated measurements were of between-run drift, which is a different quantity.
Reproduce this
- The probe corpus — all 72 prompts, versioned and frozen. Set v2 is current; v1 is unchanged and still published.
- The latest run as JSON — per-model scores, floors, the separability matrix, and the exact pushback string.
- Full methodology — scoring rules, thresholds, the local pinned encoder, and what is deliberately not measured.
Xyntherium's product orchestrates several of the models measured here, including Anthropic's, and pays each provider for that use. The scoring path is deterministic and identical for all six, and no provider reviews results before publication. These numbers describe model output on a fixed probe set on the dates shown. They are not measurements of safety, correctness, alignment, or harm.