Simulated Data Pipeline for Fine-Tuning

Ours came from a simulator that was given nothing but the agent's public definition, its tool schemas and its policy document. Expert demonstrations are the accepted way to post-train a tool-using agent on the reasoning that only a person who knows the domain can write the situations that matter. On telecom, a spec and a simulator did that job six times better than no data and better than the experts. At forty tasks the margin over expert rows is suggestive rather than settled; the next step is the full 114-task split with repeated trials. The sixfold gain over base is already beyond doubt (p = 0.006).

Result

Single trial on the official test split, GPT-4.1 customer simulator at temperature zero, protocol pre-registered and matched to AfterQuery's published run line by line. Counts are 12, 7 and 2 of 40; the simulated-data model is the only one of the three that separates from base at this sample size. Their figure is a per-domain peak checkpoint; ours is the final checkpoint, the more conservative choice. One caveat belongs next to the number: the model learned when to hand a customer to a person better than it learned when to keep going, and restraint is a graded dimension in the next set. Retail and airline, trained the same way, did not match the expert-data numbers; that analysis is a separate note.

Two ways in

The simulator has two entry points and one engine.

Describe the behavior. One sentence, such as "I want a financial agent that confirms before it moves money," becomes a drafted policy, a situation grid over that policy, rollouts against a simulated world, and graded rows ready for training. This is the cold-start path for an agent that does not exist yet.

Point at the agent's traces. Production telemetry, including plain OpenTelemetry spans, is read into a behavior state: which regions of the agent's world are failing, which are new since the last model version, which have stopped failing. That state sets the generation budget, so the next training set spends its rows where the deployed agent is actually weak.

Telecom took the first path from a public spec rather than a sentence, which is the same machinery with the policy supplied instead of drafted. The point of the result is that a spec was enough.

Method

The situation space

The simulator reads the policy as data. It parses the document into rule clauses, 16 for telecom, and treats each as a value on an axis. The 13 tools form a second axis. Four fixed axes complete the space: tool condition (success, timeout, malformed, stale, permission denied), world state (exists, missing, already acted on, duplicate, partially completed), customer stance (ten values from ordinary to adversarial), and history (fresh, prior failure, prior partial action, contradicts earlier, repeat visit).

Six axes, 360,000 cells. A pairwise covering array reduces that to 240 regions in which every pair of axis values appears together at least once, the strength the combinatorial testing literature settled on because most field failures come from two interacting parameters. We rendered 150 regions and let an embedding diversity objective choose among candidates, so coverage is a measured property of the set rather than a hope.

Persona sampling

A region says the customer is hurried and their line was already suspended. A second layer, drawn per row, says how that person types: texture, tone, length, pressure. Of the 310 telecom rows that shipped, 126 are ordinary prose, 26 all lowercase, 26 without punctuation, 25 clipped, 18 abbreviated, 11 with typos, 7 run-on. The writer model never sees those labels; it sees an aside in prose, because a model told to be terse writes an essay about terseness.

Rollouts

One hosted Qwen3-4B-Instruct model played both sides: the agent, with the real tools and policy, and the customer, with the situation card and nothing about the policy. A mock world answered every tool call, deterministic per seed. Two rollouts per situation, a six-turn target, median eight messages. Every conversation opens with the agent greeting a customer who then withholds details, matching how the deployment harness opens.

Grading

Generation is cheap; the set is made at the keep-or-drop step. Deterministic checks remove structural failures: an action claimed without a tool call, an identifier the customer never gave, success declared after a failed call. Then the customer's own grader reads every surviving conversation against a written rubric and keeps or drops it with a reason. For telecom, a pool of 369 became the 310 rows that trained. The grader is the customer's authority over the data, and the SDK's job is to make every row it produces worth grading.

Telecom passes

Sierra's answer key marks 12 of the 40 telecom tasks as ones where the correct outcome is handing the customer to a person. The model passed 9 of them. The winning transcript is representative: the phone lookup fails, the name lookup returns empty, the name lookup returns empty again, and the model transfers instead of guessing at an identity. Fifteen of the 310 training rows contain that pattern; nobody wrote them, the simulator drew them from the policy's own rules about when to escalate. Passes split evenly across the benchmark's hard and easy persona tags, so the gain is not confined to cooperative customers.

Recipe

The recipe is AfterQuery's, unchanged, because the fine-tune is not where the difference lives.

Next

• A world that can say no: lookups that fail for unknown entities and state that persists across a conversation.
• Repeated trials on all three domains, so the next comparison carries intervals that separate.
• The traces path on a live agent, end to end, with the same protocol.

The expensive layer in an expert-data pipeline is a person writing the situation. On telecom, a public spec and a simulator replaced it.