Table of Contents
This summer, several incidents involving AI agents undergoing security evaluations sent a particularly significant signal to cybersecurity teams.
Within a matter of weeks, models developed or evaluated by OpenAI, Anthropic and Meta interacted with real-world systems despite being intended to operate within controlled testing environments.
In some cases, they scanned thousands of targets. In others, they exploited vulnerabilities, obtained credentials or even published malicious software packages to exfiltrate data.
These incidents do not mean that AI has suddenly invented an entirely new generation of cyberattacks.
They reveal something potentially more significant: AI is fundamentally changing the scale and speed at which established offensive techniques can be deployed.
Traditional Attacks, Executed at Machine Scale
The first observation is that many of the techniques involved were anything but revolutionary.
Weak passwords, unauthenticated endpoints, SQL injections, exposed secrets, Kubernetes misconfigurations…
In other words, vulnerabilities and security weaknesses that security teams have been dealing with for years.
What is changing is an agent’s ability to discover and exploit them autonomously, at scale and without the usual human limitations.
In one evaluation documented by Anthropic, a model scanned approximately 9,000 targets.
Another went as far as creating email addresses, registering PyPI accounts, publishing a malicious Python package and using it to obtain credentials belonging to a real company.
An AI-powered attacker does not need sleep, does not lose focus and can test thousands of combinations.
Even a seemingly ordinary vulnerability can become critical when it can be discovered and exploited at this scale.
When a Testing Environment Becomes a Gateway to the Internet
Another important takeaway is that the security of an AI agent depends just as much on its operating environment as on the model itself.
In several incidents, agents operated under the assumption that they were interacting with simulated environments.
Yet misconfigurations gave them actual access to the internet.
The issue was not necessarily that the AI had managed to escape its sandbox.
The technical boundary designed to contain it had simply not been properly enforced.
For CISOs, the implications are significant: the security of an AI agent should never depend on the assumption that it will follow its instructions.
Restrictions must be enforced at the infrastructure level through network segmentation, egress controls, least-privilege access, environment isolation and comprehensive activity logging.
The Hugging Face Incident: An Attack Chain, Not Just a Single Vulnerability
The incident involving Hugging Face is a particularly clear illustration of this shift.
A previously unknown vulnerability provided an initial foothold. However, the subsequent stages of the attack relied primarily on a series of misconfigurations: privileged Kubernetes pods, secrets stored in environment variables, access to cloud metadata services and excessive permissions.
Individually, each of these weaknesses may seem relatively commonplace.
Combined and exploited automatically, they form a powerful attack chain.
This is precisely what offensive security has long sought to demonstrate: an organization’s actual risk exposure cannot be measured by assessing vulnerabilities in isolation. It must also account for the attack paths an adversary can construct.
With AI agents, the ability to explore these paths is reaching an entirely new scale.
Attackers and Defenders: A New Asymmetry
Paradoxically, one of the most revealing episodes involved the defenders themselves.
While investigating the Hugging Face incident, security teams were unable to use certain commercial AI models to analyze the discovered payloads. Their built-in safety mechanisms interpreted the analysis as an attempt to generate or execute malicious content.
The teams ultimately turned to a locally hosted open-source model to continue their investigation.
This situation highlights an emerging asymmetry: attackers can bypass or remove their models’ guardrails, while defenders may remain constrained by those built into the commercial solutions they use.
For organizations operating in critical sectors, developing an independent, locally controlled AI analysis capability could therefore become an increasingly important cybersecurity priority.
How Can Organizations Secure AI Agents?
Several key security principles are already emerging from these incidents.
1. Treat every AI agent as a privileged user.
An agent capable of interacting with systems, APIs or sensitive data should have its own identity, operate under the principle of least privilege and be subject to comprehensive activity logging.
2. Enforce strict network egress controls.
Agents should only be allowed to communicate with explicitly approved destinations. All outbound traffic should be controlled and logged.
3. Maintain human oversight of critical actions.
Actions such as publishing content, modifying data, calling external services or performing destructive operations should retain a human approval checkpoint.
4. Secure data as rigorously as models.
Externally sourced files, datasets and models can all introduce new attack surfaces. Artifacts should be verified, signed and executed within controlled environments.
5. Test AI agents the way an attacker would.
Theoretical security controls are not enough. Organizations must expose their agents to realistic offensive security scenarios to understand what they are actually capable of when given access to tools, data and systems.
Offensive Security Enters the Agentic Era
These incidents do not demonstrate that AI agents have become autonomous hackers capable of compromising any infrastructure.
They reveal something more tangible.
AI agents are now capable of chaining actions, exploring environments and exploiting often commonplace security weaknesses at scale.
For defenders, this is gradually changing the equation.
Security must become continuous, because attacks can be continuous.
It must be augmented by AI, because the sheer volume of scenarios to explore is increasingly exceeding what human teams can handle alone.
And it must remain fundamentally human, because the expertise of security researchers is still essential to understanding context, identifying new attack paths and distinguishing theoretical vulnerabilities from genuinely exploitable risks.
It is precisely at this intersection of human expertise, AI agents and offensive security platforms that the next generation of cybersecurity is taking shape.
The goal is not to replace security researchers with AI agents.
It is to equip defenders with the same scale, speed and adaptability that tomorrow’s attackers will have at their disposal.




