Evals for the agent you already have

pip install whileai
whileai signup --email you@example.com     # new account
whileai login                              # existing account

zp, ZeroProof and While are the same product: the package is whileai, the key starts with zp_, and pip install zeroproof still works as a shim over it. Full steps on get started.

One callable is the whole integration. It takes a message and returns the trajectory it produced:

agent(message: str) -> {"steps": [{"tool", "arguments", "result"}], "final_text": str}

Describe the tools in OpenAI function-calling shape and pass the system prompt the agent runs with in production. A callable agent runs its own real tools, so nothing is mocked; it is played single-turn, one message in and one trajectory out.

Put the real ids your world has, the order numbers and account names, in the tool descriptions or the seeds. Leave them out and the situation writer invents ids that do not exist, every run comes back "not found", and the number you get measures nothing but the missing ids.

A judge is a function over one row. It returns a reward, a reason and named markers:

judge(row: dict) -> {"reward": 0 | 1, "reason": str, "markers": {name: 0 | 1}}

Read the trajectory, not the prose: which tools were called, with what arguments, in what order. An agent that says it issued the refund and never called issue_refund should fail, and only the tool calls can tell you that.

Name markers so 1.0 is always the good outcome, or the averages read backwards. Because the policy lives in the judge, there is one source of truth: change the rule in one place and every run is rescored against it.

import whileai.simulations as wai

TOOLS = [{"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by id. Orders on file: A1001, A1002, A1003, A1004.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}}, "required": ["order_id"]}}},
         {"type": "function", "function": {"name": "issue_refund", "description": "Refund an order.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "amount": {"type": "number"}}, "required": ["order_id", "amount"]}}}]
POLICY = "Refund delivered orders within 30 days. Over $200 needs a manager. Always look the order up first."
SEEDS = ["Refund order A1001, the shoes did not fit.", "Please refund A1004, the headphones were a gift.", "I want my money back on A1002."]

def agent(message: str) -> dict:
    calls = []                                  # your bot runs here, with its real tools
    reply = my_bot.answer(message, record=calls)
    return {"steps": calls, "final_text": reply}

def judge(row: dict) -> dict:
    refunds = [s for s in row.get("steps") or [] if s.get("tool") == "issue_refund"]
    allowed = refundable(row)                   # the policy, as a program, reading the order on file
    ok = bool(refunds) == allowed
    return {"reward": 1.0 if ok else 0.0, "reason": "refunded" if refunds else "no refund",
            "markers": {"refund_only_when_allowed": 1.0 if ok else 0.0}}

data = wai.simulate(agent, tools=TOOLS, system_prompt=POLICY, seeds=SEEDS,
                    simulator=False,            # template writer: offline, no key. Drop it for the hosted writer.
                    mode="rl", repeats=4, repeat_policy="fixed", reproducible=True)
scored = wai.evaluate(data, judge)               # stamped as eval, never mistaken for training data
print(wai.pass_at(scored.rows))                  # pass@1 with a 95% interval, pass^k, pass@k
for note in scored.warnings:                     # hollow-run checks (whileai >= 0.57); fix before reading the number
    print("!", note)

Run it offline first with simulator=False. That is the template writer: no key, no model calls, seconds. Once the shape is right, drop the argument and the hosted writer produces the situations, which needs the key.

mode="rl" with repeats=4 and repeat_policy="fixed" plays every ask all four times, so no ask is quietly sampled less than the others. pass_at then gives pass@1 with a 95% interval for how the agent does in production, and pass^k for whether it does it every time.

If zero rollouts called a tool, or a marker fired on zero rows, the run is hollow and the number is meaningless. The SDK prints that warning. Fix the seeds or the tool descriptions before reading the score.

Once the number is real, make it a build step: exit non-zero when pass@1 falls under a floor you pick.

result = wai.pass_at(scored.rows)
if result.pass_at_1 < 0.80:
    raise SystemExit(f"pass@1 {result.pass_at_1:.2f} is under the 0.80 floor")

Keep a second, fast lane for the judge itself: a handful of hand-labeled transcripts run through the judge offline. No model calls, a second of wall clock, and it catches the case where an edit to the judge silently changed what passes. Run that lane on every commit and the full eval on the slower schedule.

A score is only as good as the judge behind it. Label some rows by hand and measure the agreement:

wai.attach_labels(rows, labels, kind="human")
print(wai.judge_trust(rows))

A FAIL on eight labels means label more, not that the judge is wrong. The Wilson lower bound needs about thirty labels before it can clear the bar, so a small sample fails on width alone.