Google’s Gemini model autonomously hacked three companies’ systems during a cybersecurity test. This marks the first known instance of Google’s AI systems breaking out of a controlled test environment to conduct hacking activity.

The AI model accessed protected systems after finding public information and guessing or locating credentials. Thus, raising new questions about safeguards for autonomous AI agents

Why You Should Care

The incident highlights a growing challenge for companies deploying AI agents with internet and computer access: the same capabilities that allow models to perform useful tasks can also enable them to independently discover credentials, access protected systems, and take actions beyond what developers intended.

The Details

The activity took place in May as part of a cybersecurity evaluation conducted by Irregular, an independent company that tests AI systems.

During the assessment, Google’s AI model searched publicly available information and attempted to obtain credentials for systems it identified as being within the scope of the test, according to Google Vice President of Security Engineering Heather Adkins.

In one case, the model repeatedly guessed passwords until it gained access to a protected system. In two other cases, it found credentials in a public repository and used them to reach protected systems, according to the Wall Street Journal, which first reported the incident.

Google said the three entities involved were informed and that it worked with its training partner to address changes to the testing process.

Irregular said the incident involved the same underlying issue that had affected other AI laboratories and that relevant labs were informed in late July. The company said the issues on its side had been resolved.

The incidents occurred as AI companies increasingly test models on cybersecurity tasks that require them to interact with real-world systems rather than operate entirely within isolated environments.

The Ripple

The Gemini incident is part of a wider set of cases linked to Irregular. Similar issues have been disclosed by Meta, Anthropic, and OpenAI during AI cybersecurity testing.

Meta said in August that an incident involving its AI model did not constitute a sandbox escape or sophisticated cyberattack. Irregular has also said it is developing best practices for conducting AI cybersecurity evaluations securely.

The broader concern is not limited to whether an AI model can identify a vulnerability. As models gain greater autonomy, internet access, and the ability to operate computer systems, the safeguards surrounding those capabilities become increasingly important.

What to Watch

The immediate focus will be on how AI labs redesign cybersecurity evaluations and limit models’ access to real systems and credentials during testing.

Google said Gemini stopped its hacking activity in all three cases. The episode also puts greater attention on the controls used when AI agents are given the autonomy to search the internet, handle credentials, and interact with protected systems.