Evals are the product

Traditional software has a comforting property: the same input produces the same output, so a passing test today means the same thing tomorrow. LLM features have neither property. The same prompt can produce different text on consecutive calls, and a provider can change the model underneath you without notice.

This is why teams get stuck. The first version ships quickly, then nobody dares touch the prompt, because there is no way to know whether a change made things better or quietly worse.

The golden set

Start with 100–200 real inputs with known-good outputs. Not invented ones — real queries from real users, including the awkward ones. Every bug report becomes a new case, so the set grows in exactly the places the system is weak.

This is the highest-value artefact in the whole project. It is also the least glamorous, which is why it usually does not exist.

Grade what you can grade cheaply

Not everything needs a model to evaluate it. Work down this ladder and only escalate when you must:

1. 1Deterministic checks. Valid JSON, required fields present, values within range, citation spans that actually exist in the source. Free, instant, catches real bugs.
2. 2Retrieval metrics. Recall@k against known-correct chunks. No model needed, and it isolates the retrieval layer from the generation layer.
3. 3LLM-as-judge. For qualities you cannot express as a rule — tone, helpfulness, faithfulness to the source. Slower, costs money, imperfect.
4. 4Human review. On a sample, for the cases the judge is least confident about.

Guardrails are separate from quality

Evals tell you whether the system is good. Guardrails stop specific bad things regardless of how good it is, and they run in production on every request rather than in CI.

• Prompt injection. Retrieved documents are untrusted input. A support ticket containing "ignore previous instructions" is a real attack, not a hypothetical one.
• PII leakage. Scan outputs for data that should not cross a tenant boundary — cheaper to catch on the way out than to explain afterwards.
• Grounding checks. If a claimed citation does not exist in the retrieved context, do not return the answer.
• Refusal thresholds. Below a retrieval confidence floor, say you do not know. A confident fabrication costs more trust than an admitted gap.

Trace everything

For each request, store the input, the retrieved chunks and their scores, the prompt actually sent, the raw output, tokens and latency. When a user reports a bad answer three days later, this is the difference between diagnosing it and guessing.

It also gives you the pipeline back to evals: any traced request can be promoted into the golden set with its corrected output. Production failures become regression tests almost for free.

Run them where they matter

Deterministic checks and retrieval metrics are fast enough to run on every commit. Judge-based evals are slower and cost money — run those on prompt or model changes, which is exactly when they earn their keep.

The point is not a perfect score. It is being able to change a prompt on a Friday and know by Friday afternoon whether you made it better.

That confidence is what separates an LLM feature that keeps improving from one that shipped once and has been frozen ever since — because nobody can prove a change is safe.