How to choose an AI support bot that uses your docs
Use five practical trial tests to check source quality, missing answers, outdated policies, conflicting documents and questions about individual accounts.
A bot trained on your own docs should help customers find the answer your team has approved. The buying decision gets harder when two articles disagree, a policy has changed, or a customer asks about their own account.
Use the five tests below during a trial. They are an evaluation method, not a published accuracy benchmark or a claim that any product will always pass. Start with a small set of realistic questions and keep the expected answer beside each one.
The five question test for a bot that uses your docs
| Question type | Example to adapt | What to look for |
|---|---|---|
| Covered | How long can I keep an exported report? | The documented limit, its exceptions and the right source. |
| Not covered | Can I pay by a method the docs never mention? | No invented policy; a useful contact route. |
| Outdated | Is the old storage allowance still available? | The current rule after a verified source refresh. |
| Conflicting | Which of these two cancellation deadlines applies? | An explicit qualification or human review. |
| Account specific | Has my refund been approved? | No claim to know an account state it cannot access. |
Use a test workspace or an agreed trial dataset for deliberately obsolete or conflicting material. Do not publish misleading policies in your real help centre just to challenge a bot.
First establish what “your docs” includes
Ask the supplier to identify the sources available to the exact answering feature you are evaluating. Published help articles, uploaded files, selected external pages, agent instructions and public web search are different inputs. Decide which of those your support policy allows.
Also ask how unpublished, deleted and restricted content is handled. A public support assistant should not reveal an internal note merely because it was included in a test import. Make source scope part of the trial rather than assuming every feature in one product uses identical settings.
Helpdesky illustrates this distinction. Its widget Ask AI documentation describes answers drawn from published articles with links to their sources. The beta AI agent product page describes published articles and added external URLs, without open web browsing. Check the feature you intend to buy and the material you have actually connected.
1. Can it answer a question your articles cover?
Choose a question whose answer is clear in one maintained article. Then ask it using ordinary customer language, without repeating the article heading. Include a meaningful qualification, such as a limit that applies only to a particular plan.
Read the answer and open the cited page. Does that page support the actual claim, including the exception? A link to a broadly relevant article is not enough if the response adds a number or promise that the article never makes.
Keep a record of the question, expected answer, actual answer, cited URL and result. Repeat with a few paraphrases. This helps distinguish a reliable source match from a single successful demonstration.
2. Does it admit when the answer is missing?
Pick a plausible question that your test documents do not answer. Avoid an obviously unrelated question as your only challenge: a bot may refuse a request for a poem yet still invent a plausible payment policy.
A useful response explains that it cannot confirm the requested detail and offers a way to get help. It should not convert silence in the documentation into a firm yes or no.
Follow the contact route as far as your trial permits. Check whether a teammate receives the question and its context, and what a customer sees when nobody is online. If you are still deciding which support entry points you need, our guide to contact forms, AI chatbots and ticket portals covers that separate decision.
3. How quickly does a corrected source change the answer?
In the trial dataset, change one clearly stated fact and record the time. Ask the supplier how refreshes work, how you can see whether a refresh succeeded, and whether removing a page also removes it from the answering system.
After the documented refresh has completed, repeat the original question in a new conversation. Check both the answer and the source. Then ask it in a different way to see whether the obsolete figure still appears.
Do not assume that editing a visible article immediately updates every imported copy. Make your required update interval explicit before choosing a tool, especially if your product limits change frequently.
4. What happens when two sources disagree?
Give the trial bot two approved test documents with different versions of the same rule. Label their dates and intended audiences. This tests whether the system can respect scope, rather than merely finding a sentence containing the right keywords.
A reasonable result might identify the applicable version, explain the conflict, or pass the question to a person. Treat a confident blend of incompatible rules as a failure, even if each individual sentence sounds plausible.
The test also exposes a documentation problem. Remove superseded policies or clearly mark their scope before launch. A more capable model does not remove your responsibility to maintain a consistent source of truth.
5. Can it distinguish policy from account status?
A help article can explain your refund policy. It cannot establish whether one customer's refund has been approved. Ask both questions and check that the bot treats them differently.
If a supplier claims the bot can perform account actions, request evidence of the specific integration, customer identity checks, permissions and confirmation behaviour you need. Otherwise, expect a handoff for account changes and private account information. Never accept a statement that an action is complete merely because the bot can describe the steps.
For Helpdesky, the sources reviewed here establish documentation answers and conversation handling, not an integration that approves refunds or changes subscription plans. Treat those actions as human work unless you separately verify a supported workflow.
Which controls make the results easier to assess?
Shortlist tools that let you inspect the answer, the supporting source and the reason for passing a question to a person. Ask whether the same safeguards apply in the testing area and the customer facing channel.
Helpdesky's beta agents offer a required sources setting, a confidence threshold and rules for handing conversations to people. Its Cautious preset combines required sources with an 80% threshold. That number is the agent's own confidence setting, not an independently measured 80% accuracy rate. See the documented guardrail controls.
Keep the purchase test focused on evidence quality. Choosing when an agent may send replies is a related decision, covered in our guide to AI support with human approval or automatic replies.
Watch a missing knowledge example
In Botpress's official demonstration below, the segment around 6:30 to 7:55 tests documented questions and then asks about hiring, which the sample knowledge base does not cover. The response directs the visitor to the team. Use it to recognise the behaviour to test in your own trial; it is one vendor demonstration, not proof of a general accuracy rate.
Is Helpdesky a suitable option to trial?
Helpdesky is worth shortlisting when you want public documentation, AI answers and a shared place for customer conversations. Widget Ask AI supplies article based answers; beta agents handle replies in conversations. Start with a limited trial and check your own source material against the five tests.
On the pricing page checked 4 November 2026, Free includes one workspace, unlimited seats, 15 published articles and 15 conversations per month. Pro costs $5 per month for one workspace with unlimited seats, articles and conversations. No temporary discount code was shown.
Connected AI usage is billed separately by your provider. Bringing your own key gives you visibility into that usage bill and a choice among supported providers and models. Review the selected provider's budget alerts and spending controls; do not assume every budget setting is a hard stop. The platform subscription does not include unlimited free AI.
For ongoing review, AI Question Insights shows recent visitor questions and lets you inspect their answers, with individual questions shown for the last 90 days. Use recurring gaps to improve the documentation, then rerun your test set. The beta status of agents and the need to manage a separate provider account are practical considerations before rollout.
Frequently asked questions
Can a chatbot be limited to my own knowledge base?
Some tools provide defined source inputs and settings that require supporting material. Verify the exact feature, connected sources and fallback behaviour. A source restriction is useful, but you still need to test whether the answer correctly represents those sources.
Why might an answer contain details absent from my docs?
The trial should investigate whether the wrong source was retrieved, extra sources were enabled, an old copy remained available, or the generated response added an unsupported detail. Preserve the question and output so the supplier can investigate the actual case.
Should every answer include a source link?
For documentation answers, source links make review easier and help customers read the full context. Check that the destination supports the claim and is accessible to the intended reader. A citation alone is not proof of correctness.
What should happen when the bot does not know?
It should explain the limitation without inventing a policy and provide a suitable route to help. Test that route, including the experience outside your team's working hours.
Does a high confidence score prove an answer is accurate?
No. In Helpdesky's documented agent controls, confidence is the agent's own assessment. Compare responses with known answers and source evidence rather than treating the score as an accuracy guarantee.
How should I use unanswered questions after launch?
Group recurring questions, identify the missing or unclear article, and assign someone to improve it. Retest the revised source and keep account actions separate from gaps that better documentation can solve.