Testing a ChatGPT-Powered Patient Chat System for Accuracy, Speed & UX at Bupa

The Challenge

Bupa deployed a patient-facing chat system powered by ChatGPT, grounded against a curated knowledge base of Bupa health resources. The system needed to answer patient queries accurately using only the information in that knowledge base — not hallucinate beyond it. It also needed to respond fast enough for a real-time support context, and the UI needed to be genuinely intuitive for patients who might be anxious or unfamiliar with AI chat tools. Standard functional testing approaches don't cover these requirements — testing an AI system requires specifically designed accuracy verification methods.

Our Approach

Our engineer designed a structured accuracy testing protocol. A large corpus of patient queries was assembled — covering common health questions, edge cases, queries deliberately designed to probe the boundaries of the knowledge base, and adversarial prompts intended to elicit hallucinated responses. Each response was evaluated against the source knowledge base material to verify that answers were grounded in the provided resources and not fabricated. Response latency was benchmarked against acceptable thresholds. The UI was evaluated through structured usability review with attention to clarity, error states, and accessibility.

The Outcome

The accuracy audit identified specific categories of query where the system was prone to deviating from the knowledge base — enabling the team to refine the grounding configuration before patient-facing launch. Latency benchmarks established clear performance thresholds. UI findings resulted in targeted improvements to response formatting and error state messaging. The engagement produced a reusable prompt library for ongoing regression testing as the model and knowledge base evolve.

Tools & Techniques