AI chatbots, assistants and LLM-powered features are now part of everyday products, from customer support widgets to document summarisation and smart search. They also break in ways traditional software does not. The same question can produce a different answer tomorrow, a small prompt change can shift the tone of every response, and a model upgrade can quietly make answers worse. LLM application testing is the discipline of catching those problems before your customers do.
This playbook explains what to test in an LLM application, how to build a repeatable evaluation process, and how to fit it into your existing QA and release workflow.
Key takeaways
- LLM outputs are non-deterministic, so you test with evaluation sets and scoring, not exact string matches.
- Test five dimensions: accuracy, groundedness, safety, consistency and performance/cost.
- Re-run a prompt regression suite on every prompt, model or data change, just like a normal regression suite.
Why LLM Apps Need a Different Testing Approach
Traditional tests check that input A always produces output B. Large language models do not work that way. Their answers vary with wording, context, temperature settings and model versions. On top of that, most business AI features combine several moving parts: a system prompt, retrieved company documents (retrieval-augmented generation, or RAG), tools or APIs the AI can call, and guardrails. A failure in any of them can surface as a wrong or unsafe answer.
What to Test in an LLM Application
| Dimension | What it means | Example check |
|---|---|---|
| Accuracy | The answer is correct and complete | Compare against expected answers in an evaluation set |
| Groundedness | The answer is supported by your data, not invented | Flag responses that cite facts absent from retrieved sources (hallucinations) |
| Safety & policy | No harmful, biased or off-brand content; no data leaks | Adversarial and prompt-injection test cases |
| Consistency | Similar questions get similar, on-brand answers | Run paraphrased questions and compare tone and content |
| Performance & cost | Fast enough and affordable at real traffic | Load tests, latency percentiles, tokens per request |
Build an Evaluation Set First
An evaluation set is a curated list of realistic questions or tasks with the answer (or answer criteria) you expect. It is the foundation of AI quality assurance:
- Collect real inputs from support tickets, search logs and sales questions.
- Cover edge cases: ambiguous questions, out-of-scope requests, multiple languages and misspellings.
- Add adversarial cases: attempts to reveal the system prompt, bypass rules or extract personal data.
- Define pass criteria: exact facts that must appear, facts that must not, required tone, and when the assistant should say “I don’t know” or hand over to a person.
Prompt Regression Testing
Every change to a prompt, model version, retrieval settings or knowledge base can shift behaviour. Prompt regression testing means re-running the full evaluation set after each change and comparing scores with the last approved version. Answers can be scored with rules (required keywords, format checks), with a separate “judge” model, and with human review for a sample of responses. A release only goes ahead when scores stay at or above the agreed threshold. Designing prompts that pass these suites reliably is its own skill; see our guide to prompt engineering for business.
Monitor After Release
Testing does not stop at launch. Log conversations (with privacy controls), track user feedback, sample responses for human review each week and add every real failure to the evaluation set. Over time, your test suite becomes a precise description of what “good” looks like for your product.
Where This Fits in Your QA Process
LLM testing works best as part of a broader quality strategy: functional tests for the surrounding application, API and integration tests for the tools the AI calls, performance tests for the endpoints, and security testing for prompt injection and data exposure. A QA as a Service engagement can cover the application around your AI feature with functional, API, performance and AI-driven testing; any AI-specific evaluation work is scoped per project. If you are still building the AI feature, plan evaluation into the product from day one.
Launching a chatbot or AI feature? Talk to our QA team about testing the application around your AI feature.
Frequently Asked Questions
How do you test an LLM application?
You build an evaluation set of realistic inputs with expected answers or criteria, score the model’s responses for accuracy, groundedness, safety and consistency, and re-run that suite after every prompt, model or data change, alongside normal functional, performance and security tests.
What is prompt regression testing?
Prompt regression testing re-runs a fixed evaluation set whenever a prompt, model version or knowledge source changes, and compares the scores with the last approved version so quality does not silently drop.
How can you detect AI hallucinations in testing?
Compare each answer with the source documents the system retrieved, flag facts that are not supported by those sources, and include questions the assistant should refuse or answer with “I don’t know” in your evaluation set.
Can LLM testing be automated?
Largely, yes. Rule-based checks and model-based scoring can run automatically in CI/CD on every change, while QA engineers review a sample of responses and every failure to keep scoring accurate.
Why test AI chatbots for prompt injection?
Prompt injection tries to trick a chatbot into ignoring its instructions, revealing its system prompt or exposing data. Testing with adversarial prompts before launch helps you add guardrails and prevent security and compliance incidents.
