Introducing User Simulation in ADK Evaluation

Published on Nov 7, 2025

Dima Melnyk, Product Manager, Cloud AI
Sita Lakshmi Sangameswaran, Senior Developer Relations Engineer, Cloud AI

Agents are naturally conversational, often requiring follow-up questions, clarification, or additional details from users. Creating manual tests for such interactions is tedious and fragile—any minor change in the agent’s behavior can break scripted tests, making maintenance frustrating.

To address this, we’re launching a new feature in the Agent Development Kit (ADK): User Simulation. This tool enables more dynamic, realistic evaluations of your agent’s ability to understand and achieve user intent, moving beyond rigid, scripted testing.

What Is the User Simulator?
Powered by large language models (LLMs), the User Simulator generates user prompts based on high-level goals. It is integrated directly into the ADK, allowing local execution for quick, iterative testing. Instead of a separate service, it dynamically creates conversations that reflect real user behavior, making testing more flexible and natural.

How It Works
Defining a Conversation Scenario
Instead of pre-written dialogue, you define a JSON object called a ConversationScenario, which includes:
– starting_prompt: The initial user message.
– conversation_plan: A natural language description of the user’s goal.

For example, for an agent that rolls dice and checks prime numbers:
{
“scenarios”: [
{
“starting_prompt”: “What can you do for me?”,
“conversation_plan”: “Ask the agent to roll a 20-sided die. Afterward, ask if the number is prime.”
}
]
}

When you run the evaluation, the simulator generates a conversation that continues until the conversation_plan’s goal is achieved, regardless of specific prompts or responses. An example:

USER: What can you do for me?
AGENT: I can roll dice and check if numbers are prime. How can I help?
USER: Roll a 20-sided die.
AGENT: Result is 17.
USER: Is 17 prime?
AGENT: Yes, it is.

This approach ensures tests focus on outcomes, not exact wording, making them resilient to stylistic or internal changes.

Configuring the Simulator
You can customize the simulation with an EvalConfig file, adjusting parameters such as:
– Model: Choose the underlying user model (e.g., gemini-2.5-flash).
– Model settings: Fine-tune model behavior.
– Turn budget: Limit maximum interactions to avoid infinite loops.
– Custom behaviors: Adjust how the user acts within defined goals.

In conclusion, the new User Simulation feature in ADK enables more adaptable, realistic testing of conversational agents, improving development efficiency and robustness.

FAQs

Q: What is the main benefit of User Simulation in ADK?
A: It allows for more natural, goal-oriented testing by generating dynamic conversations, reducing manual scripting effort and improving test resilience.

Q: Can I customize how the simulator behaves?
A: Yes, through the EvalConfig, you can control model selection, interaction limits, and specific user behaviors.

Q: Does this replace manual testing entirely?
A: No, it complements manual testing by providing scalable, automatic evaluation, but manual testing may still be useful for detailed case analysis.

Q: Is the User Simulator suitable for all types of agents?
A: It is most effective for agents involved in multi-turn, goal-driven interactions, especially where flexible conversation flows are needed.