Service
AI & LLM Testing
Testing for AI chat products and LLM integrations, including systems that use RAG. We check answers against a source of truth, catch hallucinations, measure latency, and cover the chat UI.
Fixed projects typically start from about £3,450 (approx. $4,500 USD) for one system: about 50 to 100 labelled questions, scored by hand against your docs, plus latency and Playwright on the chat. RAG retrieval is an extra scoring layer when the product searches a knowledge base first. At Bupa we tested a patient-facing ChatGPT chat grounded against Bupa's own knowledge base.
What makes AI testing different
AI products fail probabilistically, a response can look correct, read naturally, and be completely wrong. Standard functional QA misses this entirely.
Accuracy against a source of truth
Replies are checked against your docs, policy or labelled answers. Paraphrase is fine. Invented facts are not. If the product uses RAG, we also score whether the right documents were retrieved.
Hallucination and refusals
We probe for fluent answers that are not supported by the source, plus out-of-scope questions that should refuse or hand off, and adversarial prompts.
Latency benchmarking
Time to first token and total time, against a threshold that fits the use case: live support, healthcare, or async.
Regression prompt library
A versioned prompt set you can re-run after every model, prompt or index change. One demo pass is not QA.
How we test an AI chat
RAG is not the method. A labelled query set against a source of truth is. Retrieval is an extra layer when the product searches a knowledge base first. The starting price is this process on one system, about 50 to 100 questions.
- 1
Agree the source of truth
Your docs, policy, or labelled answers. That is the gold, not the model.
- 2
Build a labelled query set
About 50 to 100 questions for the starting engagement: real cases, edge cases, out-of-scope (should refuse), and adversarial prompts. If the product uses RAG, we also record which documents should have been retrieved.
- 3
Score the answer, not the wording
Each reply is checked for support in the source, the facts you care about, and invention. RAG retrieval is scored separately when it exists.
- 4
Latency and the chat UI
Time to first token, total time, and Playwright (or Cypress / Selenium) on send, stream, errors, citations and handoff.
- 5
A regression library you keep
A versioned prompt set you re-run after the model, prompt, or index changes.
What we test
AI products fail differently from traditional software. An LLM can return an answer that looks correct, reads fluently, and is wrong. Playwright can prove a reply appeared; it cannot prove the reply is true. We agree a source of truth (your docs, policy, or labelled answers), build a labelled query set, and score each reply against that gold, not against an exact string. RAG is how many chat products are built (search a knowledge base, then generate). It is not the testing method. When the product uses RAG we also score retrieval on its own: wrong documents in, fluent nonsense out. Our engineers have tested AI chat in production, including a Bupa patient-facing ChatGPT system grounded against Bupa's own knowledge base.
- Answers checked against a source of truth
- Hallucination and out-of-scope checks
- Retrieval scoring when the product uses RAG
- Latency and time-to-first-token
- Playwright around the chat UI
- Regression prompt library after model updates
What you get
- Accuracy audit: replies checked against the agreed source of truth
- Hallucination and out-of-scope report, including cases that should have refused
- Retrieval scores when the product uses RAG (right documents, ranked separately from the written answer)
- Latency benchmarks (time to first token and total time)
- Versioned prompt library your team can re-run after model, prompt or index changes
- Notes on the chat UI: send, stream, errors, citations and handoff
Read how we tested a ChatGPT system at Bupa → Bupa case study
Frequently Asked Questions
- An LLM can return an answer that looks correct, reads fluently, and is wrong. Playwright can prove a reply appeared; it cannot assert an exact string, because the wording changes every run. We check each reply against a source of truth (your docs, policy, or labelled answers), plus hallucinations, out-of-scope refusals, adversarial prompts, and latency.
- RAG is how many chat products are built: the app searches a knowledge base, then the model writes an answer from those chunks. It is not the testing method. The method is a labelled query set scored against a source of truth. When the product uses RAG we also score retrieval on its own, because the wrong documents in produce fluent nonsense out.
- We test the product, not a single vendor. GPT, Claude, Gemini, Llama, Mistral, or a fine-tuned model. What we cover is the prompt, the answers against your source of truth, the chat UI, and the retrieval layer if the product uses one.
- An accuracy audit against the agreed source of truth, a hallucination and out-of-scope report, latency numbers, a versioned prompt library you can re-run after model or content changes, and notes on the chat UI. If the product uses RAG you also get retrieval scores, separate from the written answer.
- Fixed projects typically start from about £3,450 (approx. $4,500 USD) for one system. That covers a labelled set of about 50 to 100 questions, human scoring against a source of truth, hallucination and out-of-scope checks, latency, Playwright around the chat, and a prompt library you keep. RAG retrieval is scored separately when the product uses it. Voice, agents with many tools, a larger corpus, or wiring the library into CI are quoted higher.
- We do not rewrite your prompts, re-chunk the index, or ship model changes. Those are your engineering work, from our findings. Ongoing weekly re-runs, a dedicated red-team, and voice or speech pipelines are separate scopes.
What makes AI testing different from standard software testing?
Is RAG how you test, or something you test?
Can you test any AI model or just ChatGPT?
What do we receive at the end of an AI testing engagement?
What does AI and LLM testing cost?
What is not in the starting price?
Ready to protect the next release?
Tell us whether you need Playwright, Cypress or Selenium automation, iOS and Android testing, AI and LLM testing, WCAG accessibility, or a Next.js or React Native build. We'll come back with an approach, a timeline and a quote.