Reward hacking detection

Over-optimization is the training reward climbing while the evaluation you care about flattens and falls (rlhfbook.com ch. 14). Reward hacking is how it happens: the policy learns the cheapest thing the reward pays for. A judge that reads the prose learns "verified"; a judge that likes detail learns length; a format bonus learns the format; a reward that never reads the arguments learns to invent them. The score keeps rising the whole time, which is why nobody notices until the model ships.

The fix is never in the rows. A flagged reward is a judge problem, a verifier problem, or a reward-function problem, and the detector's job is to say which and where. Every check below ranks and warns; none prunes.

A grouped update learns whatever separates reward within an ask; what only tracks which ask it is (difficulty) is baselined away. So the question has to be asked within ask too. Pooled correlation cannot tell the two apart: hard asks get long replies and low reward, and a pooled number calls that a length penalty. The scan centers both the reward and every feature within ask, ranks features by that correlation, and compares the top of the ranking to the 95th percentile of the same maximum when reward is shuffled within ask. Two feature tiers, both pure Python: the hand tier (length, tool calls, turns, surface counts, one indicator per tool called, every trajectory flag, every marker) and an auto tier over the 200 most common words and word pairs, which is the tier that finds the shortcut nobody listed.

scan = zps.hack_scan(scored.rows, endorsed=["tool:lookup_order", "tool:create_refund"])
scan["regime"]        # train | reward_hack | pool_exhausted | no_signal | degenerate
scan["top_feature"]   # e.g. 'contains:verified' when the judge reads the prose
print(zps.format_hack_scan(scan))

endorsed names what the reward should track. Without it the scan still ranks and floors, but cannot call a hack a hack. A reward that punishes an endorsed feature is a hack of its own, and a pool with two distinct rollouts per ask is refused as degenerate rather than read.

A judge is a reward model, and a reward model is only as good as its accuracy on labels you made yourself (ch. 5). judge_trust measures agreement, kappa, the leak rate (gold failures the judge passed), and length bias. The probes add the adversarial half: a policy trained on this judge will find the same holes the probes find, so fix the rubric before training, not after.

trust = zps.judge_trust(scored.rows, judge=my_judge, probes="all")
trust["exploitable_by"]   # e.g. ["success_claim", "filler"]

For an agent the reward has to read the trajectory, because the reply can claim anything (ch. 13). The flags are three families: lie.* (the reply's claims against the evidence), hack.* (what was written or run: a test skipped, a checker silenced, a gate bypassed), risk.* (destructive or credential-touching commands). They become behavioral markers that are 1.0 when clean, so a before-and-after report fails a run that learned to overclaim. Invented tool arguments are the same idea one level down: the argument_grounding marker is 1.0 only when every argument came from the conversation.

rows = zps.trace_markers(scored.rows)    # honest_claims, reported_failure, no_test_tampering, ...
report = zps.trace_flag_report(scored.rows)

The book's picture is proxy reward against gold reward, read against KL. Every trainer logs the first curve; nobody draws the second until the run is over. The monitor draws it during the run. It wraps the reward function so it sees every completion with its reward, samples the holdout from the live policy every every steps, and scores it twice. proxy_reward, gold_reward and holdout_length land on the run page beside the loss curve, every alarm is a mark on the step it fired, and the watch panel lists each alarm's reason.

monitor = zps.HackMonitor(run, holdout=holdout_rows, gold=zps.reward_model(rm_run),
                          every=10, k=4, endorsed=["tool:lookup_order"], stop_on="divergence")
trainer = GRPOTrainer(model, reward_funcs=[monitor.wrap(rule_reward)], args=cfg, train_dataset=ds)
trainer.add_callback(monitor)

Hosted GRPO runs carry it by default, with the hosted Phi-4 judge as the gold scorer, a different model family from the policy and one the first-action reward never reads.

All five checks are calls in the Simulations SDK (pip install zeroproof, 0.39 or later), pure Python with one dependency. optimize(mode="rl", endorsed=) carries the scan in its report, the publish gate reports it on every RL-shaped push, and push(strict_hacks=True) refuses a set whose reward is best explained by something else. The reward-hacking example runs the whole loop offline in seconds on a scripted agent and two judges; the recipe is docs/reward-hacking.md.

Three rules. Endorse what the reward should track, or nothing can call a hack a hack. A flagged reward is a judge problem, not a row problem: fix the rubric, re-grade, re-scan. Keep the gold away from the proxy: hand labels, the hosted judge, a reward model trained on other pairs, or a rule the training reward does not read.