AI agent test question generator
Create test questions from your knowledge base to check what an AI agent should answer, clarify or decline.
Enter a few details
Add your context, generate a draft and make it your own.
Limited free use. Powered by Gemini 2.5 Flash-Lite.
Your result
Complete the form to get started.
What does AI agent test question generator do?
Build a test set for an AI agent from the knowledge it uses. Questions are grouped as answerable, edge case, not answerable from the knowledge, and out of scope, each with the expected behaviour and the supporting passage where one exists.
How to use this tool
- 01
Paste the knowledge your AI agent uses and describe what it should and should not handle.
- 02
Choose how many questions you want, then choose Generate.
- 03
Ask your AI agent each question and compare its reply with the expected behaviour.
Example inputs
Use this example as a starting point, then replace it with your own details.
- Knowledge the AI agent uses
- We are open Monday to Saturday, 9am to 6pm. Haircuts cost £30. Bookings can be moved up to 24 hours before the appointment using the confirmation link.
- What the AI agent should and should not handle
- Answer opening hours, prices and booking changes. Do not give medical or legal advice.
- Number of questions
- 10
Limits and what to check
The test set is only as good as the knowledge you paste, up to 12,000 characters. Add questions from real customer conversations, and rerun the set after changing your AI agent’s knowledge or instructions.
Limited free usage is available with human verification. AI results are drafts for you to review, edit and copy. Avoid submitting confidential or sensitive information.
Frequently asked questions
Using the tool
Q1What kinds of questions are included?
Answerable questions with the supporting passage, edge cases that test wording and boundaries, unanswerable questions the agent should admit it cannot answer, and out-of-scope requests it should decline.
Q2Does the tool test my AI agent for me?
No. It writes the test set and expected behaviour. You run the questions against your AI agent and record the results.
Q3What should I paste as the knowledge base?
The text your AI agent answers from, such as FAQs, policies and service details. The AI model is instructed to draft questions and expected behaviour from that text only.
Q4How do I use the test set?
Ask each question to your AI agent, compare its reply with the expected behaviour and fix the knowledge or instructions where it falls short.
AI Smarty for your website
Let customers ask.
Let your AI agent answer.
Create an AI agent using your website and documents. Give visitors a way to find answers and leave an enquiry, even when your team is unavailable.
Starter and Growth include a 7-day trial.
Do you offer weekend appointments?
An illustration of an agent on your website.
Creating an agent is separate from using this tool. Your tool inputs are not added to an agent automatically.