What our testing proved: one ledger, one dial from consensus to ground truth¶

The engine runs one arithmetic over one attestation ledger, and the question you answer as its author is not "which product did I buy" but how much external reference to feed it — a dial from 0 to 100 that you set by strategically deploying trainer evidence:

At 0, nothing external: the ledger is fully self-referential. Your population is its own reference point, the engine measures where your reviewers agree and disagree, and the scores record the structure of belief — the Consensus mode read.
At 100, every review is anchored to your answer key: the scores are calibrated instruments recording the accuracy of belief against reality — the Ground truth mode read.

Trainers are the dial's control, not a mode the engine switches into: you deploy them in any proportion, at any rate, and most real deployments live between the named ends. The overview states the finding this page evidences, the decision math page shows the arithmetic, and what follows is the evidence — collected at both ends of the dial, proving each end for its purpose — plus what it means for your integration.

How we tested it¶

The validation was done at the dial's ends — that is where a reading is fixed — with three kinds of evidence, all running the production decision code exactly as you run it:

Simulated reviewer populations with known ground truth. We built synthetic reviewers whose true abilities we knew, and had them review thousands of work items through the real engine. The engine never saw the answer key — so when its competency scores and its verdicts turned out right, we could measure how right against truth.
A real long-running deployment. A production workload of roughly a million review rounds over 17 days: 48 reviewers, 2 projects, no trainers, no benchmark answers. We analyzed the finished competency matrix in full.
Adversarial scenarios. What happens when a work item tricks several reviewers into the same mistake — the realistic failure of real review panels. We ran the same measurements under that stress.

The 0 end — Consensus mode: measuring what your reviewers believe¶

No trainers means the dial sits at zero and the ledger is self-referenced: the population itself is the reference. The questions it answers are do we agree? and who agrees with whom?; its outputs are a stable consensus gate and an alignment map, and the ledger records the structure of belief, never its correctness.

What the evidence shows¶

Run without trainers, the engine's competency matrix organizes and stabilizes. The million-round production run held steady trends from cold to finish, and reviewers clearly differentiated from each other.

But what it organized was agreement, not correctness — in two activity regimes. (The matrix is per agent per competency: a row of scores per skill, not one number per person. GET …/adjustments returns it in full — every (agent × competency) cell — so you can inspect any agent's complete vector of skill alignments across the project.)

Lightly-used reviewers — three-quarters of the population — settled near a personal baseline. Their review traffic covered only a small fraction of the skill set (k far below m, in the Minary formalism the decision math states), and the scores across all their competencies sat within about a hundredth of one per-person value — a general temperament score, and exactly the halo effect the framework predicts for sparse coverage: a reviewer's overall standing dominates their profile. A matrix that had measured skill would show someone strong in one area and weak in another; for these reviewers, the population's alignments — who answers with whom — are what the scores track.
The busiest reviewers — the remaining quarter — built distinct multi-skill profiles. Enough traffic across enough competencies let per-skill separation develop (spreads of five to eight hundredths) — the multi-skill behavioral structure that also powers the similarity index behind competency interpolation. But those per-skill scores don't cohere as skill: the same reviewer's scores for the same underlying skill in two different review contexts don't agree, and different reviewers' per-skill patterns don't line up with each other. That structure is local review history — which panels each score happened to run with — not skill. Which is exactly what that means for interpolation in a trainer-less project: the index transfers panel-alignment priors, not skill priors — still a better cold-start estimate than the flat 0.5 default (that is its job, and why it stays the default), but read a spanish → italian transfer there as "these review contexts share my panel history," not "these skills share my ability." The skill-transfer reading is the trainer-present regime, where the same index was validated against known ground truth.

In both regimes the scores map social alignment: general temperament for most, local panel alignment for the rest — never verified correctness.

And that is not a data problem: the production run delivered hundreds of times more evidence than a calibrated cell ever needs, and the picture stayed social, not factual. Our formal analysis of the underlying mechanism (the Minary framework this engine instantiates for review) predicts exactly this shape: the channel is provably closed — the item under review cancels out of the learning signal — so its converged state is each reviewer's relative standing within your population, dominated by their overall position rather than per-skill accuracy. The measurement page of the decision math states the theorem; the scores are doing precisely what the math says they must.

What this reading is for¶

This is the engine doing its designed job on the questions that have no answer key:

Mapping your population. The scores tell you who converges with whom, and how decisively your population settles questions — genuine, honest measurement of belief on the population's own terms.
Finding your alignments. The matrix's structure reveals your archetypes — the enthusiasts, the strict gatekeepers, the contrarians — without anyone being labeled.
Designing the experiment. The reading reflects the experiment you ran — which agents you seat and which competencies each family activates. Sparse coverage reads as temperament, dense per-competency traffic is what separates skill profiles; the scores mean what your selection made them mean (the dimension structure).
Assembling good panels. Disagreement between different alignments is what produces informative, well-calibrated confidence. The engine does not discount correlated votes inside C — seating is where diversity enters. Use the alignment map to seat diverse panels, not one trusted clique.
Tracking convergence. A stable matrix means a population whose judgment is consistent — a real, usable signal.

What not to expect¶

Don't read these scores as individual skill. A high score says "this reviewer answers with their population's winning side" — a social property, not a correctness claim. Our held-out tests confirmed that a belief-only matrix does not improve the accuracy of verdicts (it stayed at coin-flip levels).
Don't set confidence bars above your population's honest ceiling. A panel's confidence tops out at its sharpest member's lone read — a unanimous fresh panel reads 0.689, a panel of 0.6-standing reviewers 0.748 — so a bar above the ceiling only adds rounds and halts (see the structure underneath the table).
Don't wait for more data to change that. We ran the clock to a million rounds. It stays a social measurement. That is what the reading is.

The 100 end — Ground truth mode: calibrating to your answer key¶

Trainers on the deciding competencies push the dial toward full external reference: your answer key is the reference, the question the ledger answers is are we right?, and its output is a calibrated instrument — the ledger records the accuracy of belief against reality. (Full 100 — a trainer on every review — is available but not the point; see the mix below.)

Mark agents as trainers and their answers become your answer key: trainers' answers win their reviews outright, and every other reviewer's scores learn from agreement-with-truth at the fast training rate.

What the evidence shows¶

Scores learn the truth you feed them. Against known ground truth, calibrated scores tracked true reviewer ability with ~98% correlation, converging in roughly 1,000–2,000 scored reviews per competency — and you only need about 75–150 trainer evaluations per cell to reach strong alignment (r ≥ 0.95).
Decisions get measurably better. With calibrated scores and enough review traffic, verdict accuracy on fresh work rose from 56% (cold) to 81%, and the engine's claimed confidence matched its actual accuracy — it never over-claimed.
You don't need perfect benchmarks. Scores learned from realistic noisy benchmarks made better decisions than idealized perfect ones (the noise spread reviewers apart more usefully).
Calibration sustains itself. After trainers open a competency, plain peer review held the calibration across tens of thousands of subsequent reviews. Trainers are needed at the gate, not on every item.

How to calibrate effectively¶

Cover the competencies your decisions actually read. Calibration is per reviewer per competency — prioritize the ones gating your verdicts.
Budget about one to two thousand scored reviews per competency for full convergence, then let normal review traffic carry it.
Top up on drift. If your team turns over or your domain moves, re-introduce trainer evidence occasionally — the sustain result means small top-ups are enough.
Treat trainer answers as your answer key. Whatever a trainer answers becomes the truth those cells learn. Curate benchmarks accordingly.

The dial position is a rate, not a state¶

Nothing above asks you to live at 100. Trainer evidence is deployed strategically, for two purposes:

Bootstrapping. A cold population's consensus is weak — there is no informative disagreement yet for the self-referential reading to feed on. A short burst of trainer evidence establishes initial standing and opens the competency (the calibration results above), after which ordinary peer traffic sustains it.
Regularization. Populations drift — turnover, domain shift. A periodic top-up re-anchors the scores to your standards; the sustain result above is why small doses suffice.

The steady state the evidence supports is a mix: mostly peer review maintaining the population's organic structure, select trainer rounds anchoring it. You choose how much epistemic reference each competency needs, and feed that much. (One caution about the far end: at a sustained 100, every review echoes your key and the population's own agency is out of the measurement — the calibration channel works precisely because you don't need that much.)

Underneath the scores: the credit records, and what they proved¶

Whichever end of the dial you run, every resolved review already implies a second, finer-grained record: for each reviewer, whether their answer earned credit against that family's reference, per measurement axis — keyed to the axis and the exact instrument text (an edited instrument starts a fresh record; the old one is never contaminated). These credit records are what a family that declares no reviewer vocabulary is weighted by (the crossover), and we tested the layer itself the same way we tested the dial: a complete production-shaped review corpus — a thousand-plus review records — replayed prediction by prediction, every guess made only from the evidence that existed at the time.

What the evidence shows¶

We compared four ways of guessing a reviewer's next grade (mean squared prediction error — lower is better):

How the next grade was guessed	Overall	First-ever grades
No record at all (the 50/50 coin flip)	0.250	0.250
The labels as written on the form	0.0377	0.250
+ records keyed to the instrument's exact text	0.0377	0.250
+ borrowing from the reviewer's related record	0.0364	0.165

Keeping any record at all is the big win — about 7× better than the coin flip, before anything fancier matters.
Keying records to the instrument's exact text cost exactly nothing here (no instrument ever changed), and a benchmark where an instrument is quietly rewritten mid-stream to measure something 25 points different prices what it buys: pooling by the label name alone mixes the old and new records (about 5% worse predictions); the text-keyed records stay clean. Free insurance against instruments that change under you.
Borrowing from related records cut first-grade error by a third (0.250 → 0.165): a reviewer proven on one axis is not a coin flip on a related axis. The same borrowing shape, measured against known ground truth on the competency side (above), cut cold-start error by roughly 19%.

The labels that measured one thing¶

The same corpus declared three review axes — and they turned out to behave as one dimension: the same reviewers earned credit on all three, family after family. The platform's vocabulary audit surfaces exactly this finding, read-only, as an advisory; acting on it is an authoring decision. We acted, and measured both sides:

Merging the three axes into one lost nothing — the merged record predicted every future grade identically. The three labels were carrying one bit of information three times.
And it paid: each reviewer's own record reached trusted standing 2.5× sooner — three facts pool into the record per family instead of one.

The advisory proposes; the measurement disposes: on labels that move oppositely, merging would blur both records toward the middle, and the before/after comparison catches that. The authoring lesson: redundant labels are free to remove, and removing them pays in faster trust.

The failure mode to plan for: shared mistakes¶

When a work item misleads several reviewers the same way — a subtly broken draft, a trickily-worded claim — this is what our adversarial testing found:

Calibrated, well-fed matrices held up. Their decision-quality advantage survived every shared-mistake rate we tested.
Cold or thin matrices did not. Under shared mistakes, a cold matrix confidently converged on the wrong answer. Confidence comes from agreement, and without a calibrated anchor, agreement can be unanimously wrong.
Treat early high confidence on a thin matrix as "the panel agreed," never as "the answer was checked."

The lesson: calibrate before you trust density. This failure is the standing risk of the dial's 0 end — agreement is the only reference a self-referential ledger has, and a like-minded panel can share one mistake with nothing external to catch it. Trainer evidence is the anchor that catches shared mistakes; panel diversity is what keeps disagreement informative between calibrations.

How much epistemic reference do you need?¶

The dial position is a property of your domain, not a product feature. One question sets it: do you have ground truth available, and is your use case about correctness?

Yes — set the dial high: trainer evidence on the deciding competencies. Trainers should be deployed whenever a reference is needed, and the finding above is blunt about why: without a calibrated anchor, a cold or like-minded panel can converge on the wrong answer at full confidence.
No — set it to 0 and stay honest about what you are measuring. When the judgment is genuinely subjective — no key exists, or pinning one would measure your expectation instead of the work — the self-referential read is the correct one.
In between — the usual case: a fluid mix of mostly peer review with select trainer doses, tuned per competency as you learn how much anchoring your population needs.

you want...	set the dial	expect
to gate work on your reviewers' consensus and know who sides with whom	0 — pure peer review	a stable consensus gate and alignment map
correct decisions on real work	high — trainer benchmarks on the deciding competencies	scores converge to truth in ~1–2k scored reviews per competency; verdicts then improve with traffic
organic population structure, kept honest	mixed — trainers at the gate and on drift, peer review in between	a calibrated matrix sustained by everyday traffic, re-measuring your population continuously

One engine, one arithmetic, one dial. Where you set it — how much external reference you feed the ledger — decides what its numbers mean; read the scores accordingly.