Back to the blog

Tony CantaveHow I Turn Gut Feelings About AI Output Into a Scoring Rubric

By Tony Cantave ·

Part of my work as a freelance AI evaluator is scoring model output: images, video, audio and user interface designs, each judged against a rubric. The hardest lesson I learned early is that my first reaction to an output is almost never useful on its own. "This looks off" or "this is pretty good" feels like a judgment, but nobody can train a model on it, and I cannot even repeat it reliably the next day.

What works is turning that gut feeling into criteria someone else could apply and get the same answer. This post is the process I use when I write or interpret a rubric, plus a short script I use to check my own consistency.

Step 1: Name what the reaction is actually about

When an output bothers me, I stop and write down every specific thing I noticed before I score anything. For a generated interface screen that might be:

  • the button text is cut off
  • two labels say the same thing
  • the color contrast on the secondary text is weak
  • the layout does not match the prompt, which asked for a sidebar

Each line is a candidate criterion. Usually the list falls into a few groups: did it follow the instructions, is it technically correct, and is it usable or pleasant. Those groups become rubric dimensions.

Step 2: Separate dimensions that move independently

A common mistake is one score that mixes everything. An image can follow the prompt perfectly and still have obvious artifacts. Another can be beautiful and ignore half the request. If both get a 3 out of 5, the score hides what went wrong.

I keep dimensions separate when they can fail separately. For most visual tasks I end up with something like:

  • Instruction following: does it include what was asked, and nothing that contradicts it?
  • Technical quality: artifacts, distortions, broken text, audio glitches, layout bugs.
  • Usability or clarity: could a real person read it, use it, or understand it quickly?

Step 3: Anchor every score level

A scale of 1 to 5 means nothing until each number has a description. The anchors are where consistency comes from. Here is an example for instruction following:

  • 5: every requested element is present and correct
  • 4: all major elements present, one minor detail missed
  • 3: one major element missing or wrong
  • 2: several major elements missing
  • 1: the output does not address the request

The test I use for an anchor: if two careful people read it and look at the same output, would they pick the same number? If the honest answer is "maybe", the anchor needs a concrete example.

Step 4: Write the reason before the number

I write one or two sentences of justification first and pick the score second. When I do it the other way around, I catch myself writing reasons that defend a number I already chose. Writing the reason first also makes disagreements easy to resolve, because the reviewer can see exactly which observation drove the score.

A good justification names the evidence: "The prompt asked for three pricing cards; only two appear, so instruction following is a 3." A weak one restates the score: "Mostly follows the prompt."

Step 5: Check your own consistency with a small script

Rubrics drift. After a long session, my 4 can quietly become someone else's 3. One way I catch this is to rescore a few items I already scored earlier, without looking at my first scores, and compare. A tiny Python script makes the comparison quick:

import csv
from collections import defaultdict

def load_scores(path):
    scores = {}
    with open(path, newline="", encoding="utf8") as f:
        for row in csv.DictReader(f):
            key = (row["item_id"], row["dimension"])
            scores[key] = int(row["score"])
    return scores

first = load_scores("first_pass.csv")
second = load_scores("second_pass.csv")

gaps = defaultdict(list)
for key, score in second.items():
    if key in first:
        gap = len(range(min(score, first[key]), max(score, first[key])))
        gaps[key[1]].append(gap)

for dimension, values in sorted(gaps.items()):
    exact = sum(1 for v in values if v == 0)
    print(f"{dimension:22} items {len(values):3}  exact match {exact:3}  max gap {max(values)}")

The output shows, for each dimension, how often my second pass matched my first exactly and the biggest gap. If one dimension keeps showing gaps of two points, the problem is usually the anchors for that dimension, not my attention. That is a signal to rewrite the anchor descriptions with clearer examples.

What this taught me beyond AI work

Before software, I spent years in delivery and logistics operations and in manufacturing, where quality checks only worked when the standard was written down and specific. A vague instruction like "make sure the parts look good" never holds up across shifts. A rule that names exactly what to reject, and shows an example, does. Scoring AI output is the same problem in a new setting: the value is not in having taste, it is in writing your taste down so clearly that someone else can apply it.

If you evaluate model output, test prompts, or review any kind of generated content, try this on your next batch. List what you noticed, split it into independent dimensions, anchor every level, write the reason first, and rescore a sample later. Your scores will become something a team, or a model, can actually learn from.