Write down the questions your customers actually ask, run every one through the agent yourself, and check each answer against what your own site already says. Anything it gets wrong is a page to fix or a rule to write. You do that before a single customer sees it, not after.
Your own site is the answer key
A test needs a right answer to compare against. You already have one. It is sitting on your published pages: service area, hours, call out fee if you publish it, license number, warranty, the brands you work on.
Copy those facts onto one sheet of paper. That sheet is what you grade the agent against. If the agent says something your site does not say, one of the two is wrong, and you decide which before launch day.
The National Institute of Standards and Technology published the Artificial Intelligence Risk Management Framework 1.0 in January 2023. It runs 48 pages. Its MEASURE section says plainly: “AI systems should be tested before their deployment and regularly while in operation.”
Regularly while in operation is the half everybody skips.
Write the questions down before you start typing
The same framework has a line for this. MEASURE 2.1 says “Test sets, metrics, and details about the tools used during TEVV are documented.” Those four letters stand for test, evaluation, verification, and validation. Plain version: write your questions down and keep the sheet.
Start with what people really type at nine at night:
- Do you cover a town at the edge of your area, and one just outside it.
- What does it cost for you to come out.
- Are you licensed and insured.
- Can somebody come tonight.
- Do you work on an old brand you stopped servicing years ago.
- My unit is making a noise. What is wrong with it.
- What is the warranty on a job you do every week.
- Who owns the company.
Twenty to thirty questions is enough to find the cracks. MEASURE 2.3 asks that performance be shown “for conditions similar to deployment setting(s)”. Plain version: test it on your phone, on cell data, late at night, with typos and half sentences. Not on the office computer in perfect English.
There are 13 subcategories under MEASURE 2. A two truck shop will not run all thirteen. Two of them earn their keep here: 2.1, write down what you tested, and 2.5, which asks that the system be shown to be valid and reliable and that its limits be written down as well.
Confidently wrong is the failure to hunt for
NIST published a second document for this kind of AI. The Generative Artificial Intelligence Profile, NIST AI 600-1, came out in July 2024. It names confabulation as a risk and defines it as “the production of confidently stated but erroneous or false content.”
That is the whole problem in one line. A wrong answer does not sound wrong. It sounds like your best dispatcher.
So spend real time on the questions your site does not answer. If you do not publish a price, ask for a price and watch what happens. If you do not touch commercial work, ask for a commercial quote. The behavior you want is a flat “I do not know, here is the number to call.” What an agent should and should not say about money is its own question, covered in can an AI agent give a customer a price.
The same NIST document says to “avoid extrapolating GAI system performance or capabilities from narrow, non-systematic, and anecdotal assessments.” Plain version: asking it four questions, liking the answers, and switching it on is not a test.
Test where the lead goes, and test the off switch
Half the failures are not in the words at all.
- Send three test leads all the way through. Check they arrive. Check the junk folder too.
- Confirm the name, phone number, address, and what the person said all came across.
- Ask the emergency question you dread and watch whether it hands the person to a human. What it should say in that moment is set out in what your chat should do when someone smells gas.
- Find the off switch and use it. Time how long it takes.
The framework has a line for that last one too. MANAGE 2.4 asks for mechanisms and assigned responsibilities “to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use.” Plain version: somebody knows how to shut it off at two in the morning, and it is written down.
NIST AI 600-1 adds one more. Share the results of pre-deployment testing with “those with system release approval authority.” In your shop that is you. One name signs off, and that person has read the sheet. What an on-site agent is set up to answer, and what it hands straight to a person, is listed on the AI assistant page.
What it costs to skip this, and when not to buy at all
On February 11, 2025 the Federal Trade Commission finalized an order against DoNotPay, a company that promoted its online subscription service as the world’s first robot lawyer. The FTC announcement repeats what its complaint charged. The company “did not test whether its ‘AI lawyer’ operated to the level of a human lawyer when generating legal documents and giving advice,” and it did not hire or retain attorneys to test the quality and accuracy of the law-related features. The order requires $193,000 in monetary relief and a notice to people who subscribed to the service between 2021 and 2023. After five comments came in, the Commission voted 5 to 0 on January 16, 2025 to approve it.
This is not legal advice and it is not tax advice. What your agent is allowed to say about licensing, permits, code, or price is a question for your own attorney and your own state licensing board, and the answer moves from state to state. Ask before you turn it on, not after.
That is a software company, not a plumber. The point still carries. They sold answers nobody had ever checked, and your agent speaks for your business the way a person on your phones does. Whether what it says can hold you to anything is a separate question, taken up in are you bound by what your chatbot says.
Here is the case for not buying anything at all. If your hours are wrong, your service area is three towns out of date, and your prices have not been touched since 2023, an agent will read all of that and repeat it faster. Fix the pages first. No test saves you from a wrong answer key.
One limit on everything cited above. Both NIST documents are voluntary. The framework says it is intended to be “voluntary, rights-preserving, non-sector-specific, and use-case agnostic,” and the profile says the same of itself. Nobody will audit your shop against either one, and neither was written for contractors. Borrow them as a checklist, not a rule.
What to do next
- Open your own site. Write hours, towns, prices, license number, and warranty onto one sheet. That is your answer key.
- Write out twenty to thirty questions. Include five your site cannot answer.
- Run every one on your phone. Write the answer next to the question. Mark it right, wrong, or made up.
- Fix the wrong ones at the page, not at the agent. The agent sits downstream of the page.
- Send three test leads and confirm all three land somewhere you look every day.
- Find the off switch, use it once, and tell one other person where it is.
- Put the sheet in a drawer and run the whole list again in a month. Answers drift after a page changes.
None of that promises the agent books you more work. It means that when it does talk to somebody, it says what you would have said.
If you want to see what the assistants already tell people about your trade in your town, a free scan asks what a customer would ask and writes down who gets named.