How to evaluate an LLM feature without fooling yourself

The most dangerous moment in shipping an AI feature is when it works in the demo. You type a question, the answer looks great, the room is impressed, and a quiet voice in your head decides it is done. That voice is lying. To evaluate an LLM feature honestly you have to fight your own optimism, because a model that nails ten cherry-picked examples can still fail half the inputs real users throw at it.

I have shipped enough features built on these models to know that the gap between looks good and is good is wide, and the only thing that closes it is measurement you cannot fudge.

Vibes are not an eval

The default way people assess an LLM feature is to poke at it a few times and form an impression. This feels like testing and is not. You unconsciously feed it inputs you already know it handles, you read the output charitably, and you remember the hits more than the misses. Human judgment in the loop is fine as a final check. As the primary evidence it is worthless, because the person judging is the same person who wants it to work.

The fix is to write down what you are testing before you look at the output. Decide on the inputs, decide what a correct answer looks like, and only then run the thing. Separating the question from the answer is the whole trick to not fooling yourself.

Build a real evaluation set

Before I trust a feature, I assemble a fixed set of inputs that look like what production will actually see, including the ugly ones. Not the clean demo cases. The typos, the half-formed questions, the edge inputs, the things users do that no designer imagined. For each one I define what an acceptable output is, even if acceptable is a range rather than a single string.

This set is the asset. Models change, prompts change, providers change, and the eval set is the stable thing you measure all of them against. A few things make it useful:

• Pull real examples from logs once you have them, because invented test cases are too polite
• Include known-hard cases on purpose, the ones that exposed a bug before
• Cover the failure you most fear, like the feature confidently answering something it should refuse
• Keep it version controlled and growing, adding every new failure you find in the wild

Define what good actually means

You cannot score a feature until you can say what success is, and for language output that is harder than it sounds. Exact string matching rarely works because there are many right ways to phrase an answer. So I get specific about the dimension that matters for this feature. Is it factual correctness against a source. Is it format compliance, like always returning valid JSON. Is it refusing the things it should refuse. Different features need different yardsticks, and naming the yardstick is half the work.

For things that resist a clean automated check, I use a second model as a grader with a tight rubric, then spot check the grader against my own judgment. It is not perfect. It is far more consistent than me eyeballing a hundred outputs at midnight.

Watch the tails, not the average

An average score hides the failures that hurt. A feature that is right 95 percent of the time sounds excellent until you realize the 5 percent includes confidently telling a user something false about their account. The shape of the failures matters more than the headline number. I always read the actual wrong outputs, not just the percentage, because a few catastrophic errors should block a launch even when the average looks fine.

This is where I decide what kind of wrong is acceptable. A wrong answer that is obviously wrong is annoying. A wrong answer that is plausible and confident is a real problem, because users will believe it. Those two failures get scored very differently in my head.

Re-run on every change

The reason to do all this work upfront is that it pays off forever. Once the eval set exists, every prompt tweak, model swap, and parameter change gets run through it before it ships. You stop arguing about whether the new version is better and start knowing. That single discipline, measure against a fixed honest set every time, is what separates teams that improve their AI features from teams that change them randomly and hope.