Google Research has introduced Agile Deliberation, a framework designed to help humans and AI systems collaborate on defining subjective concepts. Traditional machine learning models rely on humans providing clear, static labels, but real-world applications often involve ambiguous categories like 'healthy food' or 'inspiring artwork,' where boundaries shift with context and personal perspectives.
The framework splits the process into two stages: concept scoping and concept iteration. In the first stage, users break down broad topics into structured sub-concepts and exclusions. For example, 'healthy food' might be decomposed into 'salads,' 'grain bowls,' and 'lean proteins,' with exclusions like 'processed foods.' The second stage involves active testing, where the system presents borderline images to challenge the user's definitions, helping refine the AI model's understanding.
To evaluate Agile Deliberation, Google conducted 18 live user sessions, each lasting 1.5 hours, across two subjective domains: 'healthy food' and 'pay-to-play' content moderation. The framework was compared against zero-shot classifiers, automated AI collaborators, and manual deliberation. Results showed that Agile Deliberation significantly improved labeling consistency and model performance, while reducing user fatigue and anxiety by replacing manual prompt engineering with direct feedback on ambiguous images.
The underlying code and algorithms for Agile Deliberation have been open-sourced on GitHub to support further research. This work highlights the importance of treating AI training as an iterative, collaborative process rather than a static labeling task.