Free Pause Tokens Boost Language Model Performance Without Breaking the Bank

Researchers from Microsoft and Cornell University have made an exciting breakthrough in AI research. They've developed a new training technique called Free Pause Tokens, which improves the performance of large language models while keeping inference speed essentially unchanged.

The team's paper, titled 'Free Pause Tokens,' was submitted to arXiv on September 3. It describes a parallel prediction stream that shares weights with a model's main state stream, adding extra computation near the end of training without the usual penalties associated with architectural changes.

Unlike previous research that inserted explicit 'pause tokens' into the input sequence, Free Pause Tokens sidesteps these problems entirely. Instead of adding tokens to the input sequence, it creates a lightweight parallel prediction stream during training. This stream shares weights with the main model through shared gated feedforward networks (FFNs), keeping additional parameters and compute minimal.

During inference, the prediction stream can be essentially ignored, meaning the model runs at roughly the same speed as a standard transformer. The performance gain on a 1 billion parameter model comes out to approximately 2 to 3 centinats improvement in next-token prediction.