# AI agent test question generator

Create test questions from your knowledge base to check what an AI agent should answer, clarify or decline.

Canonical page: https://aismarty.com/tools/agent-test-question-generator

## Purpose

Build a test set for an AI agent from the knowledge it uses. Questions are grouped as answerable, edge case, not answerable from the knowledge, and out of scope, each with the expected behaviour and the supporting passage where one exists.

## How to use this tool

1. Paste the knowledge your AI agent uses and describe what it should and should not handle.
2. Choose how many questions you want, then choose Generate.
3. Ask your AI agent each question and compare its reply with the expected behaviour.

## Example inputs

- Knowledge the AI agent uses: We are open Monday to Saturday, 9am to 6pm. Haircuts cost £30. Bookings can be moved up to 24 hours before the appointment using the confirmation link.
- What the AI agent should and should not handle: Answer opening hours, prices and booking changes. Do not give medical or legal advice.
- Number of questions: 10

## Limits and what to check

The test set is only as good as the knowledge you paste, up to 12,000 characters. Add questions from real customer conversations, and rerun the set after changing your AI agent’s knowledge or instructions.

## Frequently asked questions

### What kinds of questions are included?

Answerable questions with the supporting passage, edge cases that test wording and boundaries, unanswerable questions the agent should admit it cannot answer, and out-of-scope requests it should decline.

### Does the tool test my AI agent for me?

No. It writes the test set and expected behaviour. You run the questions against your AI agent and record the results.

### What should I paste as the knowledge base?

The text your AI agent answers from, such as FAQs, policies and service details. The AI model is instructed to draft questions and expected behaviour from that text only.

### How do I use the test set?

Ask each question to your AI agent, compare its reply with the expected behaviour and fix the knowledge or instructions where it falls short.

## Related tools

- [AI answer grounding checker](https://aismarty.com/tools/ai-answer-grounding-checker)
- [AI meeting notes summarizer](https://aismarty.com/tools/ai-meeting-notes-extractor)
- [AI customer persona builder](https://aismarty.com/tools/ai-customer-persona-builder)
- [All tools](https://aismarty.com/tools)