Post-training data
RL datasets for agents
An RL dataset for agents is graded rollouts grouped by prompt, where the same prompt both passes and fails. The contrast inside a group is what the update reads.
What a row has to carry
Four properties separate an RL set from a pile of transcripts.
A row is a whole run: the user turns, the tool calls, the outcome. Not a question beside a gold answer.
The same opener is played several times. One rollout per prompt leaves nothing to compare it against.
Each rollout carries a reward from a judge or a verifier. An ungraded rollout cannot rank anything.
A group where every run passes, or every run fails, moves no weights. That is the row count people overpay for.
The number that decides it
Group relative methods such as GRPO score each rollout against its own group. When the rewards in a group are identical, the advantage is zero and so is the update. A set can be large, clean and completely flat.
The readable version of that is gradient support: the mean of p(1-p)/0.25 over prompts with two or more graded rows. Above 0.3 it trains. Under 0.05 it is flat. The profile reports it before you spend GPU time.
pass@k minus pass@1 reads the same fact another way. It is the headroom RL has to reach, and it comes out of the same graded groups. Three numbers after grading covers all of them. Datasets and policy gradient are defined on the glossary.
How Zero Proof Labs does it
The engine reads an agent's tools and system prompt and builds a world from them. It writes the users, plays the agent against scheduled tool faults, and grades every trajectory. You can watch one conversation become four objects, or generate rows from a spec with the Simulations SDK.
RL mode plays each opener several times, so every prompt arrives as a group. optimize(mode="rl") then keeps prompts the agent passes 20% to 80% of the time, drops dead groups, duplicates and truncated rollouts, and runs a reward-hack scan on what is left.
Rows come out as JSONL with four objects: Task, Rollout, Judgment and Marker. The same rows serve SFT, DPO pairs and GRPO, and zps.train(dataset, method="grpo") starts the run. The published sets are on Hugging Face; tool-call-efficiency ships SFT, preference, GRPO and held-out eval splits across six agents.
Claims and sources
Gradient support that trains. Under 0.05 the set is flat. Read the profile.
The difficulty band RL mode prunes to. Always-pass and always-fail prompts carry no gradient. Prune, then publish.
What a two-minute simulation run produces, graded, depending on mode. Measured.
pass@1 over 120 GRPO steps on a 707-prompt set with a 159-prompt holdout. RL, end to end.
Common questions
Graded agent rollouts grouped by prompt, where the same prompt both passes and fails. The contrast inside a group is what the update reads.
SFT needs one good trajectory per prompt. RL needs several graded ones per prompt, failures included, so the rewards inside a group differ.
Enough that a prompt shows both a pass and a fail. The pruner keeps prompts passed 20% to 80% of the time and drops the rest.
Yes. A graded group reads as group-relative advantages for GRPO. It also reads as length-matched preference pairs, one pass against one fail, for DPO.
Questions, or a higher limit: jacob@zeroproofai.com