Tool-call efficiency
Tool-call efficiency for AI agents
Tool-call efficiency is whether an agent makes every tool call the task needs and no others. A call that repeats a successful one, looks up what is already known, or fires after the task is done is slower, more expensive, and harder to trust.
What it measures
Two things decide it, and they are kept apart on purpose. A deterministic marker reads the call sequence; a judge reads the transcript. A rollout is efficient only when both agree.
Three ways a call is wasted
The same tool with the same arguments as an earlier call that already succeeded. Retrying a call that failed is a retry, not a repeat. This one is computed, not judged.
A lookup of something the user or an earlier result already gave, a call unrelated to the ask, a call after the task was done, or an action the user never asked for.
The opposite failure: skipping a call the policy required and guessing instead. Fewer calls is never the target on its own, so this is scored too.
The foundational dataset
Every row was simulated on hosted Qwen3-4B playing both the agent and the user, then graded by a separate judge (Claude Sonnet 5, a different model family). 6,681 rollouts over 1,133 tasks, six agents, each task rolled out several times so it carries its own reference: the fewest calls any complete, waste-free rollout needed.
The leanest efficient rollout per task, OpenAI chat format with a per-message loss mask.
Same-task pairs: lean and complete versus a rollout with a repeated, unnecessary, or skipped call.
Tasks where the base model was mixed, with a reference call count and a reusable reward in score.py.
A held-out task set, split by task so it shares no scenario with training data, with the base model numbers per task.
Get it on Hugging Face: zero-proof-ai/tool-call-efficiency, in the Post-Training Foundational Datasets collection.
How Zero Proof Labs does it
The engine reads an agent's tools and system prompt, builds a world from them, writes the users, and plays the agent against scheduled tool faults. You can watch a row get made step by step. The same run produces the SFT, preference and GRPO splits from one set of graded rollouts, through the Simulations SDK.
Tool-call efficiency is one behavioral marker among many. The same marker scores production traces sent over OpenTelemetry through trace ingest, so the number you train against is the number you watch in production. Every term here is defined on the glossary.
Claims and sources
Base Qwen3-4B-Instruct is efficient on 52% of 6,681 rollouts across six agents, and repeats a successful call on 6%. Dataset card.
Every call is labeled by Claude Sonnet 5, a different model family from the Qwen policy, so the judge is not grading its own writing. Every rollout ships with its per-call labels and reason.
Train and eval share no task. A word 8-gram scan of prompts against the eval set flags only stock phrasing from the situation writer, not shared tasks.
Common questions
Whether an agent makes every tool call the task needs and no others. A call is wasted when it repeats a call that already succeeded, looks up something already known, or fires after the task is done.
A deterministic marker counts repeated calls, and a separate judge labels every call needed, repeat or unnecessary and decides whether the task was completed. Efficient means no waste, no skipped required call, and an honest outcome.
Base Qwen3-4B-Instruct is efficient on 52% of 6,681 rollouts across six agents, and repeats a call that already succeeded on 6%.
Six agents, 6,681 graded rollouts. SFT, preference, GRPO and held-out eval splits, plus every rollout with its per-call labels so the rubric can be re-checked.
Questions, or a higher limit: jacob@zeroproofai.com