What AI Assistants Say About Your Product, and How to Test It

Why do AI assistants get answers about your product wrong?

Assistants answer from what they were trained on, plus any pages they fetch when they search the web. Both can be old, thin or contradictory. If your product is described differently on your site, in reviews and in directories, an assistant may blend them, guess, or leave you out.

The usual causes are easy to list:

• Training data that lags behind your latest release
• Little clear, plain-language text about what you do and who it’s for
• Conflicting names, descriptions or features across different sources
• Third-party pages that are wrong, which an assistant then trusts
• Questions where the honest answer is "there isn’t enough information", and the assistant guesses instead

How do you run an AI answer audit?

Write the questions a buyer would really ask, run each one in the assistants that matter to you, and record the exact answer. Compare every answer with facts you have approved. Because answers vary, repeat each question several times and note how often it’s right, wrong or missing.

1. List the questions: what is it, who is it for, how does it compare with alternatives, what does it not do, how is it priced, which integrations exist, is it secure.
2. Write the approved answer for each one, from your own documentation.
3. Run each question in each assistant, starting a fresh conversation every time. Save the full text of the reply.
4. Score each answer as correct, incomplete, wrong or missing, and note any source the assistant cites.
5. Repeat after every major release and on a regular schedule, and keep the history so you can see the trend.

A simple log is enough. Keep one row per run, with these columns:

Question | Assistant | Date | Saved answer | Score | Cited source | Fix needed

What can you do when an assistant gets it wrong?

You can’t edit an assistant's answer directly, but you can fix the sources it draws on. Make your own pages clear and consistent, correct mistakes where they were published, and keep facts dated and up to date. Results aren’t guaranteed, so measure them with the same audit rather than assume they worked.

• Give the product a page of its own that says what it does, who it’s for and what it doesn’t do.
• Use the same product name, description and feature names everywhere you publish.
• Write headings as the questions buyers ask, and answer in the first sentence below each one.
• Add structured data such as Organization and FAQPage markup, and make sure it matches the text people can see.
• Date your documentation and retire outdated pages, so old facts don’t compete with new ones.
• Fix errors at the source by contacting the site that published the wrong fact.
• If you want search-based assistants to cite you, check that your robots.txt doesn’t block their crawlers. Crawler names change, so check each vendor's documentation.

An llms.txt file is a proposed convention for describing a site to language models. Not every assistant reads it, so treat it as a small extra, not a fix.

What if the assistant is one you built into your own product?

Then the answers are your responsibility, and you can test them far more directly. Use a golden set of approved questions and answers, check facts with code where you can, and repeat runs to measure consistency. The same discipline applies as for checking third-party assistants, with more control.

Our pages on the chatbot audit and RAG evaluation describe how we do this, and the methodology explains why a person signs off every finding.

How often should you repeat the audit?

Repeat it after any major release, any big change to your site or documentation, and on a regular schedule such as monthly. Assistants and their sources change without notice, so a result from last quarter says little about today. Keep every run so you can see whether the trend is improving.