AI testing for chatbots, RAG assistants and agents
We test AI products the way they fail in real use: wrong answers, leaked data, ignored instructions and unsafe actions. Each engagement combines code-based checks, a golden dataset you approve and repeated runs, and ends with a report where a QA engineer has reviewed every finding and shown how to reproduce it.
What do we offer under AI testing?
What kinds of AI products do you test?
We test products built on large language models: customer-facing chatbots, assistants that answer from your documents, and agents that call tools or take actions. We test the whole product, including the prompt, the retrieval, the tools and the guardrails, not only the model underneath.
What can go wrong with an AI product?
The common failures are confident wrong answers, answers that ignore your documents, private data appearing in replies, instructions that a user can override, and agents that take the wrong action. Many only show up across several turns or on unusual input, which is why a few manual spot checks aren’t enough.
- Wrong or invented answers, stated with confidence
- Answers that ignore or contradict your own documents
- Private or internal information appearing in replies
- Instructions that a user can override or extract
- Agents that pick the wrong tool or repeat an action
How does an AI testing engagement work?
We agree what good looks like with you, build a golden dataset from answers you approve, run code-based checks and repeated test runs, and review every failure by hand. You then receive ranked findings with steps to reproduce them, and the test code to run again after each release.
- Scope the product, the risks and what correct means
- Build the golden dataset with you
- Run code checks and repeated test runs
- Review every failure by hand
- Deliver findings, test code and a readout
Do you test our model or our product?
Your product. We treat the model as one part of a system and test what a customer actually experiences: the prompt, retrieval, tools, guardrails and interface together. We don’t train or fine-tune models, and we don’t promise a model is safe. We show where your product fails and how often.
What does AI testing cost?
Each project is quoted on its scope, so there’s no price list. The main cost drivers are how many AI features you have, how much material they answer from, how many risks matter most, how often you release, and how much reporting you need. Request a quote and we reply by email.
Frequently asked questions
Can you test a chatbot that is already live?
Yes. We test against a staging copy or a set of test accounts agreed with you, so real customers aren’t affected.
Do you need access to our code?
Not always. Many checks run against the chatbot interface or an API. Access to prompts, retrieval settings and logs helps us find causes, and we agree what we need during scoping.
Will you use AI to test our AI?
Partly, and carefully. Code checks come first, and an AI judge handles only fuzzy qualities such as tone, with a written rubric. A person reviews every failure. The methodology page explains the full method.
How long does an engagement take?
It depends on scope, and we give a timeline in the quote. A focused audit of one chatbot is much shorter than ongoing testing across several AI features.
Ready to find out what is breaking?
Tell us what you’re shipping. We reply by email, and quote each project on its scope.