How to Test AI Features: QA Strategies for LLM Products

Why classical QA breaks down on LLM-powered features

Unit tests and end-to-end suites are built on a simple contract: given input X, expect output Y. That contract holds for a login form, a checkout flow, or a REST endpoint returning JSON.

It collapses for a chatbot, a summarization feature, or a RAG-based search assistant. The correctness question shifts from “did it return the right value” to “did it return a semantically valid answer, grounded in the right context, without inventing facts.” That is a qualitative judgment, not a boolean comparison.

This is why teams that try to test AI features with the same regression suites they use for the rest of the product tend to see one of two outcomes: a flood of false-positive failures that erode trust in the test suite, or a suite so loosely defined it stops catching real regressions. Neither is a QA strategy. Both are symptoms of applying the wrong testing model to a nondeterministic system.

Start with a golden test set, not a spec

A functional spec tells a QA engineer what a feature is supposed to do. For an LLM feature, a golden test set does the equivalent job, but it needs to cover more ground than a typical spec would.

A working golden set for an AI feature usually includes:

• Representative queries that reflect real user intent and phrasing, not idealized test-writer language.
• Edge cases where the correct answer is “I don’t know” or a refusal, and where a confident wrong answer is worse than no answer at all.
• Adversarial prompts that test prompt injection resistance, ambiguous phrasing, and attempts to extract system instructions.
• Known-answer pairs with verified ground truth, scored against a rubric rather than a string match.

Each item in the set needs an expected behavior, not necessarily an expected string. For a support chatbot, that might mean “response cites the correct policy document” rather than “response equals this exact sentence.”

Building this set is the highest-leverage QA work on an AI feature, and it is the part most teams underinvest in. A thin golden set produces a test suite that looks comprehensive on a dashboard and misses the failure modes that matter to users.

Metrics that tell you the feature works

Pass/fail is not enough for AI outputs. Teams that get this right score along several dimensions instead of one.

Faithfulness measures whether the response is grounded in the retrieved or provided context, rather than in the model’s general training data. This is the metric that catches quiet hallucination, where the answer sounds plausible but was never actually supported by the source material.

Relevance measures whether the response actually addresses what was asked, independent of factual accuracy. A perfectly grounded answer to the wrong question still fails the user.

Groundedness through citation forces the model to point at specific source spans for its claims, which makes verification tractable. Our guide to building production RAG systems covers this in more depth, since retrieval quality and citation discipline are where most RAG-based features lose or gain user trust.

Confidence and refusal behavior matter as much as accuracy. A system that fails closed, refusing to answer when the retrieved context is thin, is safer in production than one that always produces a confident-sounding response.

None of these metrics replace each other. A feature can score high on relevance and low on faithfulness, which is exactly the pattern that looks fine in a demo and falls apart with real user traffic.

LLM-as-judge: where it helps, and where it doesn’t

Scoring faithfulness and relevance by hand does not scale past a handful of test cases. This is where LLM-as-judge comes in: using a second model to score the outputs of the first against a rubric.

It works well for consistency checks across large test batches, for catching obvious groundedness failures, and for flagging outputs that need human review. It works less well as the sole source of truth. Judge models carry their own biases, including a documented tendency to favor longer, more confident-sounding answers regardless of accuracy, and they can miss domain-specific errors that a subject-matter expert would catch immediately.

The practical pattern that holds up: use LLM-as-judge to triage the bulk of test runs, and route a sampled percentage, weighted toward low-confidence or borderline scores, to human review. Calibrate the judge periodically against that human-reviewed sample, and retire or retrain it if the two start to diverge.

Regression testing when the model changes underneath you

Traditional regression testing assumes the system under test only changes when your team changes it. AI features break that assumption. A model provider ships a new version, a prompt gets tweaked for a different feature, or a routing layer sends a request to a cheaper model to control cost, and the behavior of your feature shifts without a single commit to your own repository.

Shadow evaluation addresses this directly: run the new model version against real production traffic without serving the responses, then compare outputs against the current baseline on your full metric set before cutting over. This catches regressions that a static test set, run once, will never surface, because production traffic includes query patterns your golden set does not.

Version-pin your evaluation runs the same way you version-pin dependencies. When a provider updates a model silently, or when cost-optimization work like the routing strategies covered in our guide to reducing LLM API costs changes which model handles a given request, you want your regression suite to catch the behavior shift before your users do.

Cutting QA time by 60–80% without losing coverage

The goal for most scaling QA teams is not more headcount. It is more leverage per test cycle. A few changes account for most of the time savings we see teams achieve:

Risk-based prioritization ranks test cases by business impact and failure likelihood, then runs the highest-risk cases on every commit and the rest on a schedule. Running the full golden set on every pull request is rarely necessary and slows the whole pipeline down for marginal benefit.

AI-generated test case expansion uses the model itself to generate paraphrased variants of existing golden set items, multiplying coverage of phrasing and edge cases without multiplying manual writing time.

Automated LLM-as-judge triage, as covered above, removes the bottleneck of manual scoring for the bulk of test runs and reserves human time for the cases that actually need it.

Sampling instead of exhaustive runs on production shadow traffic catches drift without evaluating every single request, which keeps evaluation cost proportional to risk rather than to traffic volume.

FAQ

Can traditional test automation tools test LLM outputs? Tools like Selenium, Cypress, or Playwright still work for testing the surrounding application, the UI, the API calls, the data flow into and out of the model. They are not built to score whether a generated response is factually grounded or semantically correct, which requires a separate evaluation layer.

How big should a golden dataset be for testing AI features? Size matters less than coverage of failure modes. A few hundred well-chosen cases covering common queries, known edge cases, and adversarial prompts will catch more real issues than a few thousand near-duplicate examples.

What’s the difference between evaluation and testing for AI features? Testing typically checks specific, known cases against expected behavior before release. Evaluation is broader and often continuous, scoring production traffic on an ongoing basis to catch drift that a fixed test set would miss.

How often should regression tests run when the underlying model changes? Any time the model version, the prompt, or the routing logic changes, run the full evaluation suite before rollout. Shadow evaluation against live traffic should run continuously, independent of whether a change has shipped.

Does AI-powered test automation replace QA engineers? It replaces the mechanical parts of the job, generating and scoring high-volume test cases, so QA engineers can spend their time on judgment calls, edge case design, and calibrating the automation itself.

If your QA process was built before AI features entered the product, retrofitting it case by case usually costs more than rebuilding the evaluation layer with the right primitives from the start. A short audit of where your current process loses time, and where it’s already blind to AI-specific failure modes, is often the fastest way to see what a rebuild would actually save. That’s the starting point of our AI-Powered QA & Test Automation work.