Huawei Trains Record-Breaking Language Model Without Nvidia Hardware

Huawei has successfully trained a 505-billion-parameter language model called openPangu-2.0-Pro without using Nvidia hardware. The model was trained on Huawei's Ascend 910B NPUs, which were manufactured primarily by Semiconductor Manufacturing International Corporation (SMIC) using the N+2 process.

The training run is significant because it demonstrates that the Ascend software and architecture stack can sustain frontier-scale training. However, it does not yet prove that a fully domestic Chinese hardware stack can achieve the same feat.

A closer look at the supply chain reveals that the Ascend 910B chips used in the training process contained dies fabricated at TSMC's 7nm node, rather than SMIC's N+2 process. This means that Huawei relied on a stockpile of foreign-fabricated chips to keep production volumes viable until 2024 and 2025.

The model uses a Mixture of Experts (MoE) architecture with 505 billion total parameters, making it the first at this scale to not rely on Nvidia hardware. The attention mechanism is a layered hybrid, combining Multi-head Latent Attention (MLA), Decoupled Sparse Attention (DSA), and Sliding Window Attention (SWA) layers.

The Muon optimizer was used in the training process, which offers greater computational efficiency compared to Adam-family alternatives. This advantage is crucial on constrained Ascend clusters where per-chip throughput is lower than on Nvidia's equivalent hardware.