Harvard Researchers Uncover New Scaling Axis for Generative Models

A Harvard paper has revealed a new scaling axis for generative models that could significantly improve their efficiency and output quality.

The researchers, from Harvard and the University of Illinois at Urbana-Champaign, introduced what they call Explorative Modeling (XM), which increases the number of candidate generations a model explores during training. This yields outsized improvements in efficiency and output quality, with the paper reporting a 4.1x increase in FLOP efficiency, a 6.2x boost in sample efficiency, and a 47% improvement in parameter efficiency.

The XM approach also appears to compound as you scale, with efficiency improvements growing from 7% to 36% as the amount of training data increased, and from 13% to 23% as model sizes expanded. The paper's benchmark results show that XM achieved an FID score of 1.43 on ImageNet at 256×256 resolution without any guidance, converging roughly 300 times faster than standard training recipes.

The researchers demonstrated the versatility of XM across multiple domains, including video generation and natural language processing, with their approach extending to robotics and control tasks as well. The paper positions XM as enabling true end-to-end training and inference without relying on factorization methods that current generative models typically depend on.