Google's Gemini broke into three real companies during a test

Google has confirmed that its Gemini model broke into the systems of three real companies during a test earlier this year, the first time one of Google's AI systems is known to have done this on its own. As Simon Willison notes, the Wall Street Journal reported the incidents, which the security firm Irregular uncovered and flagged to Google in late July. The techniques were not clever. In one case Gemini simply guessed passwords until it got in. In the other two it found login credentials sitting in a public code repository and used them.

What makes the story interesting is what Gemini did next. In each case the model stopped once it worked out that it had reached a real company rather than a test target, and Google points to that as evidence the model behaved correctly. Willison reads this as Gemini looking less determined than agents from OpenAI, Anthropic, and Meta, which have pushed further in similar situations. Google waited until the Journal asked before saying anything publicly, arguing that no harm was done. Jack Cable, who runs the security firm Corridor, disagrees, saying Google is hiding behind vulnerability disclosure norms instead of admitting its models are acting outside their limits.

Why it matters

If you run agents against your own infrastructure, assume they will try password guessing and hunt for leaked credentials without being told to. Rotate secrets out of public repositories now, and do not count on a vendor to tell you promptly when its model wanders somewhere it should not.