Don't let the LLM
pick a number.

Seven principles.

Each one rules out a way the LLM-pick-a-number failure mode sneaks back in. Together they define a scoring pipeline you can defend against an adversary who's read the prompt.

1. 01

   Separate observation from scoring

   The LLM finds evidence. A formula, not the LLM, produces the score. The number is computed, not chosen.

2. 02

   Discrete signed impact items

   Every piece of evidence gets one of {+5, +3, +2, +1, −1, −2, −3, −5}. Forces commitment. Removes the 7-out-of-10 anchor.

3. 03

   Diminishing returns (sqrt)

   normalized = net_impact / sqrt(total_items). The 40th item adds less than the 4th. Evidence farming is punished.

4. 04

   Density-weighted confidence

   Confidence = how much evidence the scorer found, not how sure the scorer feels. Sparse runs are visibly low-confidence.

5. 05

   Anchored center

   Sparse-evidence runs regress toward 50. The multiplier never exceeds 1.0 — high evidence confirms, never amplifies beyond raw.

6. 06

   Bounded scale with self-check

   Final scores live in [0, 100]. Across criteria, the spread must be ≥ 20 — otherwise the evaluator was not discriminating.

7. 07

   Separation of LLM and deterministic computation

   Independent passes by different model families collect evidence. Math, not the LLM, combines them. Adversarial synthesis catches contradictions.

Discrete impact → diminishing returns → confidence-weighted.

Two application patterns. Use the pooled variant for simple checkers (one bucket of evidence, one final score). Use the per-criterion variant for matrix benchmarks (formula runs once per criterion; weighted average produces the overall).

# Per criterion
net_impact         = sum(item.impact for item in items)
total_items        = len(items)
normalized         = net_impact / sqrt(total_items)
raw                = clamp(50 + normalized * 8.0, 0, 100)
density            = total_items / 20
multiplier         = 0.75 + 0.25 * clamp(density, 0, 1)   # never > 1.0
final              = round(50 + (raw - 50) * multiplier)
confidence         = clamp(density, 0, 1)

# Across criteria (overall)
overall_score       = round(sum(c.final * c.weight))
overall_confidence  = min(c.confidence for c in criteria)
self_check_span     = max(c.final) - min(c.final)
                      # must be >= 20

Discrete impact set: {+5, +3, +2, +1, −1, −2, −3, −5}. Hard cap of 5 items per perspective per criterion per pass. The multiplier never exceeds 1.0 (confirms, never amplifies). Sparse evidence is visibly low-confidence, not silently confident.

Run the math by hand.

Four small examples + one real-world rescoring. The toy cases show how the pieces interact; the rescoring shows what changes when an existing benchmark drops the LLM-picks-a-number step.

1. A

   Strong, abundant evidence

   given · net_impact = +25, total_items = 25

   • · normalized = 25 / sqrt(25) = 5.0
   • · raw = 50 + (5.0 × 8.0) = 90
   • · density = 25/20 = 1.25 → multiplier = 1.0
   • · final = 90, confidence = 1.0
2. B

   Strong, sparse evidence

   given · net_impact = +25, total_items = 4

   • · normalized = 25 / sqrt(4) = 12.5
   • · raw = 50 + (12.5 × 8.0) = 150 → clamped 100
   • · density = 4/20 = 0.2 → multiplier = 0.80
   • · final = round(50 + 50 × 0.80) = 90, confidence = 0.2
3. C

   Average

   given · net_impact = 0, total_items = 20

   • · normalized = 0
   • · raw = 50
   • · density = 1.0 → multiplier = 1.0
   • · final = 50, confidence = 1.0
4. D

   Weak, well-evidenced

   given · net_impact = −15, total_items = 25

   • · normalized = −15 / sqrt(25) = −3.0
   • · raw = 50 + (−3.0 × 8.0) = 26
   • · density = 1.25 → multiplier = 1.0
   • · final = 26, confidence = 1.0 (high confidence in a low score)

v0.8 — calibration-conditional

Will it help on your model? Run the probe first.

The methodology is not a free lunch. We ran the v3 prompts across six model families (gpt-5.5, deepseek-flash, gemini-3-flash, gemma4, gpt-oss-20b, nemotron) on a held-out 77-submission hackathon dataset and observed a clean regime structure: principled rescoring wins on every metric for the most-inflated model (gemini-3-flash), can over-correct on already-calibrated models (gpt-5.5, deepseek-flash, gpt-oss-20b — the 3-of-6 over-correction pattern), and cannot rescue intrinsically weak signal (nemotron, the textbook PICKS_A_NUMBER case).

The fix is to run a 30-second probe first — a 20-item synthetic rating test, no ground truth required — and let the regime label tell you how much methodology to apply.

Order of checks: JITTERY first (high run-to-run variance makes the shape rules unreliable), then INFLATION_LIKELY / DEFLATION_LIKELY (extreme clusters), then PICKS_A_NUMBER (compressed middle), then CALIBRATED. Full classifier thresholds and the empirical six-model grid are in Sections 5.5, 5.7, and Appendix E of the paper.

In production

A whole product runs on this formula.

MyBench is a 45-minute interview that turns your real work into a private, saturate-resistant AI benchmark suite — then scores it across model × harness combinations. The scoring engine is the seven-principle methodology on this page, reused unchanged. Same discrete impact set. Same sqrt normalization. Same 5×5 perspective × criterion matrix. Different domain, same math.

One preflight, three ways to score.

One repo, four skills. calibration-probe is the 30-second preflight — run it first to find out which regime your model is in. Then pick the scoring skill that matches your input shape. All four are skills.sh-installable into Claude Code, Cursor, Goose, OpenCode, and any other skills-aware agent.

// preflight calibration-probe first — 30 seconds, tells you whether the methodology will help on your model.
// not sure which? what-works-feedback-judge is the simplest. hackathon-judge if you have a code submission. evidence-scoring if you're bringing your own domain.