How can you trust AI to test AI?
You can trust it only if the AI never has the last word. Our method uses code for everything that can be checked exactly, your approved answers as the source of truth, and an AI judge only for fuzzy qualities, with its limits measured. A QA engineer reviews every failure before you see it.
AI handles the volume. A QA engineer signs off on every finding.
What do you check with plain code first?
Anything with a right answer: facts against your documents, response format and schema, personal data in replies, banned phrases, response time and cost per answer. Code gives the same result every time, costs little and can’t be talked round, so it does as much of the work as it can.
- Facts checked against your documents
- Response format and schema
- Personal data in replies
- Banned phrases and off-limits topics
- Response time and cost per answer
What is the golden dataset?
It’s a set of questions and expected answers, built from your documents and approved by you. It’s the source of truth for what correct means. We never invent the answer key ourselves, and when the golden set and the AI disagree, the golden set wins.
When do you use an AI judge?
Only for qualities that code can’t check, such as tone, helpfulness or whether an answer is clear. The judge scores against a written rubric and must quote the evidence for each score. It never decides facts. It’s a different model from the one under test, so it can’t grade its own work.
How do you know the AI judge is accurate?
We compare the judge with human labels on a sample of 50 to 100 answers. If it agrees with the humans at roughly 90 percent or better, we use it. If not, we don’t use it for that quality and a person reviews those answers instead. We re-check when the judge or rubric changes.
How do you handle answers that vary between runs?
AI answers change from run to run, so one pass proves little. We run each test several times and report the pass rate and how much it varies. A test that passes every time and one that passes only sometimes are different findings, and both are shown.
Who signs off the findings?
A person. AI handles the volume, and a QA engineer reviews every failure, confirms it's real and signs it off in plain language. Nothing reaches your report on the strength of an AI score alone, and you can ask to see the evidence behind any finding.
Can I reproduce what you found?
Yes. We keep full logs of inputs, outputs, settings and versions, and each finding lists the exact steps and how often it occurred. You can run the same tests yourself, and the test code and golden dataset are yours to keep.
How does Playwright testing follow the same standard?
No hidden flaky retries, tests that run in your own CI, readable code you own, and traces and video saved on every failure. The standard is the same as for AI: a result should be explainable and repeatable, and a green build should mean something.
Where this method is applied
Frequently asked questions
Do you use AI to write the tests?
We may use AI to draft test ideas or code, and a person reviews everything before it’s used. We don’t accept output we can’t explain.
What if the AI judge and the human disagree?
The human decides, and the disagreement is recorded. A pattern of disagreements is a sign the rubric or the judge needs to change, so we fix that before relying on it.
Ready to find out what is breaking?
Tell us what you’re shipping. We reply by email, and quote each project on its scope.