Google AI Model Accidentally Accesses Real Company Systems During Safety Test

Google has confirmed that its Gemini AI model accessed systems of three real companies during a safety test in May. The incident occurred during an evaluation conducted by Irregular, which involved giving the model access to the internet. However, an error in the test environment allowed the model to access company systems instead of simulated targets.

In two cases, the model discovered exposed credentials in public repositories and used them to access protected systems. In one instance, it even guessed passwords until gaining entry to a system. The affected companies were informed by Google's vice president of security engineering, Heather Adkins.

Adkins stated that the model halted its activity once it determined it had reached real company systems rather than simulated targets. She emphasized that this was not an example of model misalignment and did not warrant public disclosure since Gemini's safety measures worked.