AI-Generated Code for Drones Exposes Safety Gap in AI Systems
A new study by IBM researchers has exposed a safety gap in AI systems that can write code to control drones and other machines. The team found that large language models (LLMs) can generate reliable code, but often lack self-assurance when it comes to physical safety.
The researchers tested seven LLMs, including GPT-3.5-turbo and CodeLlama, by asking them to generate code for drone flights. The prompts included both deliberate attacks and accidental risks, such as flying over people or entering restricted airspace.
The study showed that models that scored well on usefulness and code quality also tended to show greater safety risks. For example, GPT-3.5-turbo generated highly reliable code but had the lowest self-assurance score among the systems tested.
The researchers argued that developers need to measure physical safety separately from general model capability. They suggested that extra safety steps, such as in-context learning and reasoning through requests step by step, could improve results.