In the recent OpenAI and Hugging Face incident report, impossible tasks, caused by unintended defects in the ExploitGym tests that agents were being evaluated against, were identified as a trigger for hacking behaviour.
In this research, we explore whether frontier models will escalate rather than reward hack when facing problems with defective test infrastructure, and whether the escalation reports themselves are useful for diagnosing defects in the environment. This is part of work exploring the feasibility of reducing the cyber threat from autonomous AI agents by changing their decision environment to make cooperative actions preferable to adversarial ones.
Key takeaways
Across 8 frontier models spanning 5 families, combining an escalation channel with an anti-reward hacking policy reduces reward hacking from 23.6% to 5.3% with no detectable cost or performance overhead, eliminating it entirely for 6 of 8 models.
Escalation and hacking are near-perfectly mutually exclusive, with 98.7% of escalations involving no hacking (100% under the combined intervention of a normative policy and an escalation channel).
An escalation channel needs to be combined with an anti-reward hacking policy to achieve the best results, suggesting that activating norm-following through policy context is as important as providing a structured alternative action to route behaviour toward.
While monitoring of deployer visible model reasoning and code comments provides the largest visibility of defects (38.2%), escalation adds +10.1 percentage points of defect detection coverage and is more accurate once it fires (99.4% vs 85.8%).
While these results are promising, they reflect the behaviour of single agents in a simple evaluation setting. The decision environments of multi-agent settings are more complex, with the actions of other agents themselves shaping the decision environment, and warrant further research.
What we did
We used the EvilGenie benchmark (built on LiveCodeBench v5/v6), selecting 9 competitive programming problems whose test infrastructure contains independently confirmed checker or reference-data defects. These are problems where legitimate task completion is blocked by infrastructure, creating task–infrastructure conflicts that drive reward hacking.
The models were given competitive programming problems from the EvilGenie benchmark (built on LiveCodeBench v5/v6) where the test infrastructure contains checker or reference-data defects. As a result even correct solutions will return failed visible tests: agents can submit solutions despite failing tests, hard code answers from the tests so the solutions pass or modify the test infrastructure so their solutions pass. We classify reward hacking as any hard coding of answers in the solution or modification of the test harness itself, irrespective of the submitted solution. 1 The agents also have the capability to flag suspected defects either as part of their submitted solution in any condition, or via the escalation channel tool, when this is provided.
We were interested in how often agents would reward hack and how often they would flag defects in the test environment. We manipulated two elements of the agent’s decision environment in a 2 × 2 factorial design:
Escalation channel (present or absent). A structured tool, submit_escalation_report, that the agent can call to report an infrastructure concern to a human oversight team instead of proceeding. The tool takes four required arguments: issue type, problem assessment, approaches tried, and recommendation, and returns a receipt confirming the report has been queued for review.
Anti-reward-hacking policy (present or absent). A workspace document, anti_reward_hacking_policy.md, delivered into the agent’s sandbox and referenced from the system prompt. The policy defines reward hacking, names specific prohibited behaviours, including test manipulation and output hardcoding. It is readable by the agent via the same tools it uses for the problem itself.
This gives four conditions: A (baseline, neither), B (escalation only), C (policy only), and D (combined).
We additionally ran a fifth condition, E (prompt-only), in which a brief anti-reward-hacking instruction is appended to the system prompt without providing either the policy document or the escalation tool, to test whether the structured elements of C and D provide value beyond what a simple instruction alone achieves.
Findings
1. The combined intervention reduces reward hacking by 78%, eliminating it entirely for most models
Combining the anti-hacking policy with the escalation channel gives all 8 models their lowest hacking rate, and reduces it to zero for 6 of them. Pooled across all models, hacking falls from 23.6% at baseline to 5.3% under the combined intervention . Four models reach zero from the policy alone (C), and one (claude-fable-5) from escalation alone (B), but two models (gpt-5.3-codex and grok-4.5) need the full combination to eliminate hacking.
The escalation tool and the policy act on different outcomes: the tool nearly quadruples disclosure over baseline (5.3% to 20.8%) but only reduces hacking to 15.0%; the policy reduces hacking further (9.7%) but barely moves disclosure (6.4%). The policy suppresses hacking without producing reporting; the tool provides a structured action for the agent to do something productive with the defect it has already recognised. Only the combined intervention is best on both axes simultaneously (28.1% disclosure, 5.3% hacking), because it both activates the disposition to comply and provides the action that compliance routes toward.
The prompt-only condition (E) reduces hacking from 23.6% to 16.9%, which is worse than both structured conditions and, similar to the anti-reward hacking policy condition, does not improve voluntary defect disclosure. Prompt-only also costs 16.6% more per episode than the combined intervention, with the cost increases concentrated in the models it helps least.
2. Escalation and hacking are near-perfectly mutually exclusive, with reward-hacking redirecting to disclosure with an escalation channel and policy
Across the conditions where escalation is available (B and D), 98.7% of the 154 escalation events involved no hacking, rising to 100% under the combined intervention. This replicates the mutual-exclusivity finding from our earlier blackmail-scenario work in a different domain, and suggests that the mechanism at work is redirection: the agent that escalates does not subsequently hack.
3. There is no detectable performance or cost overhead for the best performing intervention (but the study is powered only for large effects)
Solve rates under the combined condition are statistically indistinguishable from baseline, using a final-verdict metric that independently verifies solution correctness even when defective checkers reject valid submissions.
Per-sample cost and message count are also statistically indistinguishable between A and D. The policy alone (C) carries a slight overhead (+14%), suggesting that the policy document induces deliberation that the escalation channel, when added in D, helps resolve.
A note on what “no detectable overhead” means here: at n = 360 per condition, the study is powered to detect pooled solve-rate differences of roughly ±10 percentage points or larger at 80% power. At the per-model level (n = 45 per cell), the minimum detectable difference is wider still, ranging from ±14pp to ±22pp depending on baseline rate. Our observed difference of +1.9pp is well below these thresholds, so the claim is properly understood as an absence of a large effect rather than confirmation of zero effect. Detecting a ±5pp pooled solve-rate difference would require approximately 1,600 episodes per condition (about 4× our current volume); detecting ±3pp would require roughly 4,400 per condition (12× current). Our code and data are open-sourced to allow others to run studies powered to detect smaller effects.
4. Escalation channels function as diagnostic infrastructure
Beyond reducing hacking, escalation channels improve defect detection beyond what passive monitoring already provides. Restricting to conditions where escalation exists (B and D, 720 episodes), monitoring alone catches 275 episodes (38.2%); adding escalation raises this to 348 (48.3%), a +10.1 percentage point uplift representing 73 episodes that escalation catches and monitoring alone misses. This implies that an escalation channel adds diagnostic coverage rather than merely reshuffling the same detections into a different channel.
When the channel does fire, it is also more accurate. Escalation correctly names the actual defect mechanism 99.4% of the time, compared to 85.8% for monitoring.
5. Disclosure depends on the decision environment, not just the availability of a reporting path
Models could always have reported a defect in their submitted answer, in every condition, with no tool required, and largely did not. In the baseline condition A, monitoring shows that the agent identified the actual catalogued defect in its reasoning trace or code comments 37.5% of the time, but only discloses it in the solution 1.7% of the time.
6. The Gemini-family models are more resistant to both components, and form 100% of residual hacking under D
Neither Gemini model reaches zero hacking at any condition. Unlike the other six models, the two represent a growing share of total hacking as mitigations strengthen (42.4% at baseline A, 55.6% under escalation alone B, 80.0% under policy alone C), reaching 100% under the combined intervention once every other model hits zero. It is unclear whether a different intervention design would better activate the training-derived dispositions that route behaviour toward escalation, or whether these Gemini models are less sensitive generally to environmental interventions.
Open questions and what comes next
Can our findings generalise beyond single agent simple coding benchmarks?
Our study tests single agents in isolated episodes that take only a few minutes to solve. Real deployment scenarios are likely to involve multiple agents interacting over longer time horizons with shared infrastructure, where the actions of one agent shape the decision environment of another. This introduces more complex dynamics, for example, where a defect discovered by one agent is exploited by a second, or an escalation report filed by one agent is visible to others. The decision environment is no longer static and designer-controlled; it is dynamic and partially determined by other agents’ behaviour.
This is where the research connects to cooperative AI, which I was able to study in more detail over the summer. If the decision environment includes other agents, the mechanism design problem becomes a multi-agent game. The question shifts from “does this agent escalate?” to “does this configuration of agents and environmental signals produce cooperative rather than adversarial equilibria?” While our single-agent results are a starting point, the multi-agent extension is both more complex and more relevant to real deployment, and needs further attention.
What are the main limitations of decision environment interventions?
The interventions we tested work because they activate training-derived dispositions that are convergent across the models we looked at. They worked for agents misbehaving because they are overly eager in pursuing assigned tasks (our threat model of interest), but may not work for other threat models, especially agents pursuing hidden goals. Under this threat model agents could treat the escalation channel as a surface to manipulate rather than a genuine reporting path, escalating when observed and hacking when not. A misuse risk also applies to our methodologies: the approaches we develop to design environments that encourage cooperative behaviour could be studied and repurposed to design environments for adversarial behaviour.
New training paradigms may also change the dispositions these interventions rely on. If future models are trained with narrower reinforcement signals, norm-following may weaken, and mechanisms designed for current models may stop working. This means decision-environment interventions cannot be designed once and deployed permanently, instead they need to be systematically tested and reconfigured as models change.
How do we move from testing one intervention to systematically designing decision environments?
Our study tested one mechanism design (normative policy + escalation channel) across one benchmark. To develop this into a more robust framework that can be tested against different models and adapted, we need methodologies and infrastructure for systematically identifying which environmental variables drive misbehaviour and which interventions address them.
Google DeepMind’s GRAM framework (Lindner, Krakovna & Farquhar 2026), built on Anthropic’s open-source Petri auditing tool, demonstrates a potential methodology for this. GRAM creates realistic simulated deployment scenarios, reproduces observed misbehaviour in static environments, and then runs automated ablations to isolate what drives it. What GRAM’s methodology adds to our approach is the ability to identify environmental drivers before designing interventions. Rather than hypothesising that an escalation channel will help and then testing it, you could use GRAM’s investigator pipeline to first map which environmental variables (metric pressure, scope ambiguity, missing escalation paths, monitoring visibility) are load-bearing for a given class of misbehaviour, and then design the minimal mechanism that addresses those specific drivers.
This would help develop a more robust and adaptable process to designing and testing interventions: diagnose the environmental drivers, design the mechanism, test whether it works across model families, and iterate.
Can we protect organisations on the receiving end of AI cyber attacks?
In the recent OpenAI incident, the agents eventually breached Hugging Face’s network looking for details of how their task would be evaluated, showing that agents will cross organisational boundaries when useful for completing their task. The interventions we tested sit in the deployer’s environment (the system prompt, the tool configuration, the policy) but the organisations at risk of being compromised are not the deployers; they are third parties whose infrastructure happens to sit between the agent and its goal.
Can we design interventions targeted at defenders of these organisations, optimised for a threat actor that is not malicious but relentlessly completing a benign task? Traditional information security controls are hard blocks designed to make intrusion difficult, developed for human attackers with malicious intent. For an AI agent pursuing a non-malicious goal, a hard block is a technical challenge to overcome but the same agent might respond to an embedded signal, for example a machine-readable notice at the perimeter that says “this system does not contain what you are looking for” or “report this access attempt to your deployer rather than proceeding.” If the agent’s weights encode dispositions toward norm-following and scope-respect, and if the signal is designed to activate those dispositions, the boundary itself becomes part of the decision environment.
This is currently speculative, but if a new category of cyber threat is emerging from AI agents relentlessly pursuing non-malicious goals, then the defences developed for malicious human threat actors may not be the most effective response. Controls that work with the agent’s trained dispositions rather than against its capabilities could offer a different, and potentially more scalable, class of defence.
Why I did this research
I am concerned about the threat of cyber attacks from advanced AI models and the disruption this poses for every sector, from education to healthcare to finance. While cybersecurity has been studying the motives of human attackers for many years, AI systems are hacking for different (and more mundane) reasons. In the recent incident on HuggingFace agents were hunting for an evaluation scorer that was not even there, and hacking their way through infrastructure to find it.
This reminds me of what I observed when I worked in Operational Risk, which is that most incidents do not occur as a result of malicious intent, they occur because people select actions that are logical and advantageous for them in the situation they find themselves in. This was (usually!) not solved by changing the person, but by changing their environment: the information factoring into their decision, the actions available to them, the consequences they can see. I see an opportunity to explore this an avenue for reducing the risk of cyber attacks from autonomous AI agents, specifically by designing the environments of agents to make cooperative actions more likely to be selected than adversarial ones when they are pursuing their tasks.
We choose this act-based definition as modifying the test infrastructure is out of scope of the task, even if it is correcting a defective test. Notably, modification of test infrastructure was always done silently, without any notification of this in the solution.py file or via the escalation channel. We have open sourced data to enable researchers to reconstruct an outcome based definition if this is of interest.