Teams often run an eval, get a number, feel in control, and ship. But a score is not the same as useful evidence. That gap causes many LLM projects to fail: production breaks in ways the eval should have caught.
These are the five patterns we most often see behind valid-looking but unreliable eval results.
Pitfall 1: your eval set mirrors your training examples' distribution
An eval should predict behavior on unseen inputs. When its cases closely match the examples used to write the prompt, it measures memorization rather than generalization.
This usually occurs when teams create the eval set and prompt examples together. A developer writes the prompt, adds a few input-output pairs showing the desired behavior, then evaluates on those same pairs. Since the prompt was tuned to them, it naturally scores well.
Before writing the prompt, hold out part of the eval set. Write without viewing those cases, then test on them. A significant drop from the development cases to the held-out cases indicates that the prompt overfit its examples.
A stronger approach adds adversarial inputs, meaning cases expected to be difficult. Test unusually short or long inputs, another language, and incorrect formatting. A good prompt handles them with some grace. Clean, expected cases alone will not expose its response to noise.
Pitfall 2: your judge prompt favors length
LLM-as-judge methods help, but have a documented failure mode: models trained on human preference data often rate longer outputs higher regardless of quality. Human raters trained on summary quality can mistake thoroughness for quality.
A judge prompt asking "Is this response high quality?" without a precise definition likely measures length too. More verbose outputs may score higher after a prompt change even when their content has not improved.
Write judge prompts around specific properties, not general quality. "Does this response correctly identify the entity mentioned in the query?" is better than "Is this a high-quality response?" For less specific properties, show high- and low-scoring examples where the difference is not length.
Test the judge too. Run ten manually labeled outputs through it, then compare judge scores with your labels. A judge that conflicts with manual labels 40% of the time is not a reliable eval signal.
Pitfall 3: one average over mixed eval data hides distribution issues
An average of 0.84 across 100 cases sounds strong. Yet 20 easy cases may always score 0.97, while 80 harder cases score 0.80, hiding the real result. Similarly, 15 input types below 0.60 disappear inside the overall average.
This matters most when one eval set mixes input types. A document summarization prompt may handle short documents well but long ones poorly. An entity extraction prompt may handle English well but non-English poorly. One average can make gains on easy inputs look positive despite losses on hard ones.
Stratify the eval set by input type and report each stratum's score. When a prompt change raises the overall average, inspect that breakdown before calling it an improvement. A significant drop in one stratum deserves investigation before shipping.
Pitfall 4: the eval omits the real production prompt context
Most applications add runtime context to prompts: retrieved documents, user account data, conversation history, or session state. LLM behavior depends on that context as well as the base prompt template.
An eval using only the base template, or synthetic context unlike retrieved documents, tests a different prompt from production. Scores can be high while production quality is low because real inputs contain untested context.
Run evals with representative context. For retrieval-augmented generation, include document chunks resembling actual retriever results, including noisy or irrelevant chunks. For conversation history, use realistic histories rather than only single-turn inputs.
Setup takes longer, but this is the only way to make eval scores predict production behavior instead of a simplified problem.
Pitfall 5: your eval dataset is not versioned with your prompt
Eval scores matter only against a specific dataset. If you add, remove, or revise cases and compare scores, you are comparing different datasets. A higher score may show a better prompt, or simply that hard cases were removed.
This pitfall is subtle because it resembles normal development. You add cases for a discovered failure, remove ambiguous ones, and the overall score rises. Was that caused by the additions, removals, or a better prompt? Without separate dataset and prompt versions, you cannot know.
Commit dataset and prompt changes separately, with separate eval runs. After changing the dataset, run the current production prompt on it to set a new baseline before testing prompt variants. Then prompt A and prompt B are compared on the same dataset.
What makes evals trustworthy
Eval scores are not inherently untrustworthy. They become trustworthy when cases represent production inputs, scoring is calibrated against manual labels, results are stratified by input type, full production context is included, and dataset and prompt versions are tracked separately.
This checklist is real and takes more than an afternoon. The alternative is evals that look rigorous while creating false confidence. Teams making the most progress on LLM app quality invest early in trustworthy eval signals and use them for decisions. Good eval infrastructure pays off when it catches a regression before production.