OpenAI's Model Hacked Its Capability Evaluation—We Built A Secure Evaluation Environment

This week, Hugging Face and OpenAI documented an incident that sent ripples through the AI Safety security communities and made global headlines. During a cybersecurity benchmark evaluation, an AI agent exploited zero-day vulnerabilities to access the internet, obtained remote code execution in Hugging Face's environment, and retrieved information needed to complete the benchmark — rather than solving it. This incident is a clear signal that evaluation environments themselves need to be engineered with the same rigor as the AI systems they're testing, which is exactly what we've built at Tinycode.

The particular evaluation OpenAI ran focused on advanced exploitation techniques with very limited attack paths. Testing of this nature encourages agents to pursue high-risk cyber activity, which can lead them to seek Internet access when it could help accomplish their objective. In this incident, the agent spent significant effort attempting to gain internet access despite guardrails within the evaluation environment. It ultimately identified an internal system with internet connectivity, used it as a pivot point, and exploited a Hugging Face server to retrieve the information needed to complete the challenge.

Publicly available evaluations like the one used by OpenAI are becoming easier for adaptive AI agents to game. Instead of performing the actual tasks, it is a common practice for agents to use deceptive behavior such as seeking internet access to gather the information needed to circumvent the challenge, lie when they know they're under tests, use memorized content, and even manipulate the environment to succeed. (see, e.g., Wang et al.'s work at UC Berkeley). At Tinycode, we understand these important details and designed our evaluation system around these risks.

We treat benchmark integrity as a foundational design principle rather than an afterthought, specifically engineered to avoid the traditional evaluation failure modes outlined in Wang et al.'s research. To guarantee reliable assessments, we enforce a strict separation between the evaluator and the agent, implement rigorous evaluation logic, and maintain a robust set of security controls. This is reinforced by advanced firewall restrictions, continuous network monitoring, and a complete air gap to block unintended Internet access. We don't leave it to chance. The result is an evaluation environment built to eliminate cheating paths, force the agent to solve the task as intended, and produce a far more accurate measurement of its true capability.

But network isolation alone isn't enough. Autonomous agents are highly adaptive, and restricting a specific tool doesn't necessarily restrict the capability that tool provides.

We observed this directly during internal testing. In one evaluation, an agent was testing a web application for a vulnerability that could allow access to other users' receipt data. To accomplish this, the agent first attempted to deploy a dedicated security-scanning and fuzzing tool (wfuzz) to manipulate a numerical value within a URL.

wfuzz -z range,1-1000 http://<ip>/order/FUZZ/receipt

This command tells wfuzz to replace FUZZ with every number from 1 to 1,000 and check whether any of those values reveals a valid receipt. Our supervisor guardrail blocked the request because this tool was restricted to prevent potentially disruptive activity within client and testing environments.

Instead of abandoning the restricted approach, the agent recreated the same capability using ordinary, unrestricted commands:

for i in $(seq 1 1000); do
  curl -s -o /dev/null -w "$i: %{http_code}\n" http://<ip>/order/$i/receipt
done

The new command reproduced the same restricted behavior we were trying to prevent. In other words, we blocked the specialized tool (wfuzz), and the agent rebuilt its functionality from other, non-restricted tools to effectively bypass the same guardrail. If the underlying activity is destructive, disruptive, degradative, or otherwise restricted, an agent may attempt to achieve the same result using a different combination of available commands.

Effective controls must dynamically evaluate the behavioral pattern itself and enforce strict rules around the underlying activity. Catching these subtle bypasses is a critical part of what differentiates Tinycode. We do not assume that blocking a command means the capability itself has been contained; we validate how the agent actually behaves and use those observations as valuable test data to continuously refine our evaluation methodology and controls. Every evaluation result undergoes manual review to identify hallucinations, anomalous behavior, and attempts to bypass the intended testing path. When an agent discovers an unexpected permutation around a restriction, we analyze the underlying tradecraft, harden the corresponding environment controls, refine the agent, and feed those learnings back into subsequent assessments. Each discovery becomes an opportunity to identify unexpected paths before they reach authorized customer environments while making the platform safer, more precise, and more capable. This iterative approach ensures Tinycode evolves alongside the agents being tested, rather than trailing behind with rigid rule sets that increasingly capable models learn to navigate around.

Tinycode is a full-spectrum AI platform to evaluate, red team, and secure the entire AI ecosystem — from base LLMs to autonomous agents to endpoints and the environments they run on. Just as human operators adapt their approach as new information is discovered, Tinycode AI agents are engineered to reason through complex environments, adapt their approach, select the right tools, and accomplish objectives across a broad range of systems and attack surfaces. Through continuous engineering, rigorous testing, and iterative refinement, Tinycode becomes more capable with every assessment — evolving alongside the systems it evaluates and delivers increasingly accurate, realistic, and lightning-fast engagements across nearly any environment.

The recent OpenAI/Hugging Face incident serves as an important reminder that AI agents behave differently from traditional software. As these systems become more capable, the industry will need more capable evaluation methodologies that guarantee trustworthy measurements. That is precisely what makes Tinycode stand out from other providers. We guarantee our platform will prevent your AI systems from cheating on benchmarks so you can feel secure about the actual capabilities and risks of your models and systems. We believe the future of AI evaluation depends on measuring actual capability, validating behavior, and guaranteeing that benchmarks cannot be circumvented.