Amazon Researchers Crack Long-Context Problem with GRPO Fine-Tuning

A team of researchers at Amazon has made a breakthrough in addressing the long-context problem in AI models. The issue arises when trying to feed massive documents into an AI model, causing it to forget information from earlier pages by the time it reaches later ones.

The Amazon AGI team thinks they've found a solution by applying a reinforcement learning technique called Group Relative Policy Optimization (GRPO) during training. This approach allows models to be more resilient to cache compression, which is necessary for handling long contexts but can throw away crucial information in the process.

The researchers developed a method called RetrievalAttention, designed to help models retain critical data even when the cache is trimmed. When combined with GRPO fine-tuning, the team achieved remarkable results: their model could handle context windows trained at 32,000 tokens but generalize to contexts as large as 128,000 tokens or even 1 million tokens.

The team tested their approach using two well-known multi-hop question answering datasets and saw significant improvements in performance. In-domain results showed a 20-point improvement on HotpotQA benchmarks, while out-of-domain results favored the GRPO fine-tuning approach by up to 9 points.