AI Evals

The problem with vibes

Ask an LLM "rate this post 1–10" and you'll get a number. Run it again and you might get a different one. Ask a different model and you'll get a third. The number feels precise, but it's measuring the judge's general impression — not any specific property of the output.

This is the vibes problem: a single holistic score tells you almost nothing actionable. It went from 7 to 6 — why? Which part got worse? Was it the opening, the tone, the specificity? You can't debug what you didn't decompose.

A rubric fixes this by turning one vague question into several precise ones. Instead of "is this good?", you ask five binary questions, each targeting one concrete quality. Now when the score drops, you know exactly which dimension fell.

Decomposing quality into dimensions

The first skill: taking a vague quality goal and breaking it into orthogonal, observable, discriminating dimensions. Each word earns its keep.

• Orthogonal — you can improve one dimension without touching the others. If "tone" and "voice" always move together, they're measuring the same thing — merge them.
• Observable — a human can point to specific text as evidence. "Does the post feel authentic?" is not observable. "Does the post contain at least one named personal experience?" is.
• Discriminating — scores actually vary across outputs. A dimension that's always PASS teaches you nothing; it's dead weight in your rubric.

Worked example — decomposing "write a good LinkedIn post" into five dimensions:

Notice: each dimension targets a different part of the text (opening, throughout, body, ending) and can fail independently. A post with a killer hook but no specificity scores PASS/FAIL/FAIL/PASS/PASS — immediately telling you where to focus the rewrite.

Binary vs. Likert — and why binary usually wins

Two scoring approaches for each dimension:

In practice, binary almost always wins. Here's why:

• Calibration collapse — What's a 3 vs. a 4? Two judges (LLM or human) will answer differently. Binary forces you to define the threshold in the rubric text, not in the judge's head.
• Higher agreement — Binary judge-human agreement rates are typically 15–20 percentage points higher than Likert on the same rubric.
• Cleaner aggregation — "4 of 5 checks passed" is immediately meaningful. "3.7/5 average score" is relative to what baseline?
• Easier few-shot — You need one PASS and one FAIL example. For Likert you need examples at every level to calibrate the judge.

Few-shot examples inside judge prompts

A judge prompt is a prompt — it responds to the same prompt engineering principles as any other LLM call. Few-shot examples are the single highest-leverage addition you can make to a rubric.

The pattern: for each dimension, show the judge 2–3 graded examples with reasoning. Include at least one PASS, one FAIL, and ideally one borderline case — the borderline is what teaches the judge where the line actually sits.

Three principles for good few-shot examples:

• Include a borderline case — the easy PASS and easy FAIL are obvious. The borderline case is where the judge learns your actual threshold.
• Show at least one FAIL — all-PASS examples train a lenient judge. The judge needs permission to fail things.
• The reasoning matters more than the score — the reasoning section is the judge's inner-monologue template. It learns to reason the way your examples reason.

Chain-of-thought grading

The order matters: quote first, reason second, score last. This is chain-of-thought grading, and it exists to prevent the judge from deciding the score before looking at the evidence.

Without CoT, the judge often anchors on a first impression and rationalizes backward — "this seems pretty good → 4/5 → let me find reasons it's a 4." With CoT, you force a different sequence:

This sequence reduces two common judge biases:

• Position bias — without CoT, judges disproportionately weight the beginning of the output. Quoting forces them to process the full text.
• Verbosity bias — longer outputs tend to get higher scores because they "look" more thorough. CoT redirects the judge to evaluate substance, not length.

The structured output format isn't just for readability — it's a forcing function. By requiring the judge to fill EVIDENCE before REASONING before SCORE, you physically prevent the anchor-then-rationalize failure mode.

Putting it together

Here's the anatomy of a complete rubric for one dimension, showing all the pieces assembled:

This is a production-grade rubric dimension. It has: a precise definition, a PASS example with reasoning, a FAIL example with reasoning, a borderline case that teaches the threshold, and a structured output format that enforces CoT grading.

A full rubric repeats this pattern for each dimension. Five dimensions, each with this level of specification, is a complete judge prompt.

The bootstrap problem

You built a rubric in Module 1. You wired it into a judge prompt. The judge is scoring outputs and the scores look reasonable. Five dimensions, binary checks, chain-of-thought grading — everything by the book.

But "looks reasonable" is exactly the vibes problem we just spent a module escaping. The judge says PASS on Voice and FAIL on Hook — is it right? You don't know, because you haven't checked it against anything. You're trusting the judge on faith.

Meta-evaluation is the practice of evaluating your evaluator. The idea sounds recursive but the implementation is concrete: get humans to grade the same outputs, then measure how often the judge and humans agree — and more importantly, how they disagree.

Building a calibration set

The foundation of meta-evaluation: a set of outputs graded by humans using the same rubric the judge uses. This is your ground truth — the thing you compare the judge against.

The calibration set is not a one-time artifact. Rebuild or extend it whenever the content distribution shifts — new product features, different user demographics, or a changed generation prompt all change what the judge needs to handle.

Percent agreement — and why it lies

The naive metric: count how often the judge and human give the same score, divide by total items. Simple, intuitive, and dangerously misleading.

The problem is base rate inflation. If 90% of outputs genuinely pass a dimension, a judge that always says PASS achieves 90% agreement. It looks great. It's useless — it has zero discriminative power, agreeing only because it never risks a FAIL.

Percent agreement rewards judges that follow the base rate. In any skewed distribution — and most eval distributions are skewed — it's nearly meaningless as a quality signal.

Cohen's kappa — adjusting for chance

Cohen's kappa asks a sharper question: how much better is this judge than random guessing? It computes how often two raters would agree by chance (given their individual tendencies), then measures how much the actual agreement exceeds that baseline.

The interpretation scale:

The key insight: kappa penalizes the easy agreement that comes from skewed distributions. A judge that always says PASS on a 90%-PASS dataset gets 92% percent agreement but κ ≈ 0.15 — exposed as barely above chance.

Try it: the kappa calculator

Edit the confusion matrix below and watch percent agreement and kappa diverge. Try the presets to build intuition for when percent agreement lies.

Finding disagreement patterns

A kappa score tells you how much your judge disagrees with humans. But to fix the judge, you need to know how it disagrees. Four patterns to look for:

• Leniency bias — the judge passes things humans fail. Check: is the judge's PASS rate higher than the human PASS rate? Fix: add more FAIL examples to few-shots, especially borderline FAILs that the judge is currently passing.
• Severity bias — the judge fails things humans pass. Less common with LLM judges but appears when rubric language is overly strict. Fix: revisit the rubric definition — is it asking for something unreasonable?
• Dimension-specific blindness — the judge is well-calibrated on Hook and Structure but poorly calibrated on Voice. This means the Voice rubric needs work, not the whole judge. Fix dimension by dimension.
• Content-dependent failure — the judge handles straightforward posts well but struggles with sarcasm, technical jargon, or code-heavy content. Fix: add few-shot examples from the failing content type.

The diagnostic procedure: for every disagreement, read the judge's chain-of-thought reasoning. Is it quoting the right evidence? Is it reasoning correctly from evidence to score? The CoT format from Module 1 makes this forensic analysis possible — without it, you'd just see a bare score with no way to trace why it went wrong.

When to retrain or retire a judge

Calibration isn't a one-time exercise. Judges drift — the content distribution changes, the underlying model updates, or your quality standards evolve. Here's when to act:

Pairwise Comparison & Ranking Systems

Binary rubrics tell you whether an output is good enough. But what if you need to compare two outputs — which is better? Module 3 covers pairwise comparison, the Elo rating system adapted for LLM outputs, position bias in A/B judging, and when to use ranking instead of absolute grading.