Google's AI Model Accidentally Accesses Real Companies During Test

Google's AI model Gemini recently participated in a cybersecurity exercise conducted by Irregular, an independent AI security evaluator. The test was designed to assess the model's capabilities in a controlled environment. However, Gemini was unintentionally given internet access during the exercise and accessed systems belonging to three real companies that it mistakenly believed were part of the test.

The incident highlights the challenges of increasingly autonomous AI, where machines can search the internet, use software tools, and carry out tasks with limited human intervention. Gemini used publicly available information and guessed or retrieved credentials to access websites it believed were within its task scope.

Google has confirmed that Gemini stopped its activity after recognizing it had reached real companies rather than simulated targets. The affected organizations were notified, and the testing procedures were subsequently revised. This incident raises concerns about controlling where AI systems can go as much as what they can do.