Two weeks after adding LLM evals to CI, we still had not run a test. We had reviewed every eval framework, debated scoring methods, and wrestled with handling nondeterminism at scale. Our first automated eval check was completely over-engineered.
Here is what day one should have included, based on what found bugs rather than adding ceremony.
Why CI evals differ from unit tests
Unit tests are deterministic: the same input to a function returns the same output every time. LLMs differ. The same prompt can produce outputs that mean nearly the same thing but use different words. So asking "does the test pass?" is much harder.
Cost is another difference. A suite covering a thousand examples makes a thousand API calls. In CI, where every pull request may trigger a run, costs rise quickly. Cutting coverage to save money is a real risk, so plan for it from the start.
Latency is the third difference. A unit test suite finishing in 12 seconds is fine. An eval suite taking 8 minutes creates deployment friction. People may skip the gate or ship hotfixes to avoid waiting.
These differences are not reasons to skip CI evals. They mean the CI eval layer must account for these constraints from the start.
Three essentials for a minimal setup
After enough CI eval experience, we narrowed the requirements to three things.
First, use a fixed golden dataset with ground truth. It can be small: Twenty to fifty examples covering key cases is enough to begin. Version it beside the prompt, so every historical run has a known dataset.
Second, add at least one deterministic assertion per prompt. Before using LLM-as-judge, write checks that need no model call. Are the expected JSON keys present? Is the output within its character limit? Does a specific input produce the required opening phrase? These checks are free, instant, and fully reproducible.
Third, establish a regression baseline. Run the suite on the current production prompt and save its scores. When a PR changes that prompt, rerun the suite and compare. A 3% average score drop merits investigation. A 20% drop should block the merge.
An eval platform, custom scorer, and multi-dimensional rubric can wait. These three pieces in your existing CI environment will catch most real regressions.
Assertions: deterministic first, probabilistic next
Teams most often turn to LLM-based scoring before using all available deterministic assertions. LLM-as-judge helps, but adds variance, cost, and the possibility of inconsistent judgments between runs.
Before creating a judge prompt, ask what you can check without another model call. For structured extraction, verify fields exist and have the right type. For summarization, check the output length is acceptable. For classification, ensure the result is an allowed category.
Deterministic assertions catch major failures: empty strings, a broken JSON schema after prompt refactoring, or a changed format after a model upgrade. These important bugs need no judge.
LLM-as-judge helps with properties that deterministic rules cannot express well, such as tone, helpfulness, and factual accuracy against a knowledge base. Calibrate the judge on a small set of human-labeled examples first, to learn the score range for acceptable outputs.
CI cost and latency management
Use tiers to control CI eval cost. Your full dataset may contain 500 examples, while CI uses 30 covering high-risk edge cases and common usage. Run the full set on a schedule, nightly or weekly, instead of each commit.
For lower latency, parallelize eval calls. Do not run fifty model calls sequentially. Batch them under a concurrency limit that respects your API rate limits. A sequential suite taking 8 minutes often finishes in under 90 seconds with reasonable parallelism.
Cache deterministic results when possible. If the prompt is unchanged, rerunning an assertion adds no value. A content-addressed cache using the prompt hash and input hash enables free, fast reruns for unchanged prompts.
Gate or warn: a policy choice
An early question is whether a failed eval should stop CI or issue a warning. The answer depends on what it measures and how much confidence you have in its signal.
Block on deterministic assertions. If output parsing fails, the prompt is broken in a way production will expose immediately. It should not merge.
For probabilistic scores, warn while calibrating the threshold. A few weeks of data reveal which drops are noise, from model nondeterminism or rephrasing, and which signal real quality loss. Once baselined, block drops beyond the calibrated threshold.
A frequent error is choosing an aggressive blocking threshold on day one without data. Blocking valid changes makes the team distrust the eval gate and seek workarounds. Start permissive, collect data, then tighten it after learning the normal score distribution.
What CI evals cannot tell you
A well-designed CI eval suite finds prompt regressions, not production distribution shift. Your golden dataset covers anticipated inputs. Real users will provide unfamiliar ones, and their behavior will not affect CI scores until you find those cases and add them to the dataset.
CI evals also miss latency changes at your API provider. If average model response time rises by 40%, the suite will not flag it unless latency is measured and asserted explicitly.
CI evals are not a complete quality signal. They provide a regression floor, showing that you have not worsened results against a known baseline. Production observability complements this by logging real inputs and outputs, revealing unexpected behavior on actual traffic.
A tight CI eval suite plus production logging covers both ends: known datasets reveal regressions, while logs show behavior on inputs you did not anticipate.
Start in one afternoon
If a production prompt has no eval setup, you can create a minimal CI eval today. Select your ten most important cases. Add three assertions per case: a deterministic check, a length check, and a simple judge prompt asking "is this output coherent and on-topic?" Save scores from the current production prompt as a baseline. Have CI run the checks and compare results with it.
That is all: ten cases, three assertions each, a baseline, and a comparison step. It is not glamorous, but it catches the most important bugs, accidental regressions that could reach production unnoticed and emerge in user feedback two days later.
Add complexity when you learn what the simple setup misses. Build trust through catches, not architecture.