Gina Neff, head of the Minderoo Centre for Technology and Democracy at the University of Cambridge, told BBC Radio 4’s Today programme that the security tests – called sandboxes – are “supposed to be secure environments where you can see what the models are capable of”.
“In this case, it looks like OpenAI didn’t make a secure enough sandbox,” she added.
Instead, the agents created their own cyber-attack against the sandbox itself, finding a vulnerability which allowed them to escape the restrictions.
Once outside, the AI identified Hugging Face as a likely source of the answers they were seeking in the test, and tried to gain access.
Neil Lawrence, Professor of machine learning at Cambridge University, called it an “impressive feat”, but cautioned it “falls well within the known capabilities of the current generation” of high-powered AI models.
He pointed out that OpenAI is looking to list itself on the stock market, and faces intense pressure from rival firm Anthropic, which has made headlines with its own powerful AI tool, Mythos.
“OpenAI are now playing catch-up, they are trying to demonstrate their own systems’ capabilities in cyber-security.”
“It shows us that OpenAI are not capable of safely deploying their own technology,” he added.
In its initial disclosure of the hack on 16 July, external, Hugging Face said it was still assessing whether any customer or partner data was affected and would contact affected parties if necessary.
It said it has now closed the vulnerabilities highlighted by the incident and rebuilt the affected systems.
“Autonomous, AI-driven offensive tooling is no longer theoretical,” it said.
“Defending an online platform now means treating the data and model surface as a first-class attack surface, and using AI on defence to keep pace.
“We will keep investing there, and keep sharing what we learn.”
