Agent simulations
Agent simulations for training AI agents
An agent simulation runs an AI agent against a world built from its own tools. A separate model plays the user, and every finished run is graded.
What a run contains
A simulation needs four parts. Each one exists so the result can be scored, not just read.
State the tools can read and change: a booking, a balance, an order. Tool calls return real results instead of stubs.
A goal and a person pursuing it. A separate model plays them, so the agent is not talking to itself.
Timeouts, empty results and wrong data, scheduled on purpose. Happy paths never show how an agent recovers.
A verdict on the finished run. Without one the run is a transcript, not training data.
Not agent-based modeling
Search results for this term cover two fields. Agent-based modeling simulates crowds, traffic and markets to study emergent behavior. This page means the other one. It runs a single AI agent through realistic tasks, to test it or to train it.
It is also not a fixed eval. An eval scores a set of cases someone wrote in advance. A simulation writes the cases first and then scores them, so coverage is not capped by the list.
How Zero Proof Labs does it
The engine reads an agent's tools and system prompt and builds the world from them. It writes the users, plays the agent against scheduled tool faults, and grades every trajectory. You can watch a row get made step by step.
Every row is JSONL with four objects: Task, Rollout, Judgment and Marker. Four modes cover eval sets, supervised fine-tuning, reinforcement learning and adaptive runs. Three lines of Python start a run in the Simulations SDK, and synthetic training data for agents covers what a usable row is made of.
The same behavioral markers score production traces sent over OpenTelemetry, through trace ingest. One metric runs before training and after it. Every term on this page is defined on the glossary.
Claims and sources
A two-minute simulation run produces 317 to 808 graded rows, depending on mode. Simulations SDK.
A 15 dollar simulated set with no human-written row took Llama-3.1-8B from 5.0% to 30.0% on tau2-bench telecom. The same count of expert-written rows reached 17.5%. The fine-tuning post.
zeroproof-ecommerce-1b, trained inside a simulated marketplace, scores 75.3% macro intent accuracy at 18 cents per million output tokens. Intent model.
Common questions
A run of an AI agent against a world built from its own tools, with a separate model playing the user. Every finished run is graded.
An eval scores cases someone wrote in advance. A simulation writes the cases first, then scores them, so coverage is not capped by the list.
Yes. The same graded rollouts export as SFT rows, or as contrastive pairs for RL methods such as GRPO.
A two-minute run produces 317 to 808 graded rows, depending on mode.
Questions, or a higher limit: jacob@zeroproofai.com