Baidu has released ERNIE 5.5 Flash Base, a new open-source foundation model built on a sparse Mixture-of-Experts architecture with 280 billion total parameters. According to Baidu, only 14 billion parameters are activated per token, and the model was pre-trained from scratch on 33.5 trillion tokens with a training context of up to 128,000 tokens.
The release is notable because Baidu is shipping the raw pre-training checkpoint without mid-training, post-training, or fine-tuning. Developers can use the weights as a starting point for domain adaptation, instruction tuning, and reinforcement-learning research, and a post-trained version is expected to follow later.
ERNIE 5.5 Flash Base specs and architecture
Baidu designed ERNIE 5.5 Flash Base around efficiency. Most of its 43 layers use Heavily Compressed Attention, which compresses every 128 tokens into a single cached entry, while one layer in seven keeps token-level access to the full context using Multi-head Latent Attention. This hybrid approach cuts KV-cache cost while still letting the model see the whole sequence.
The model also introduces a Multi-Router MoE system that replaces a single router with several independently initialized sigmoid routers, improving expert selection especially in middle and deeper layers. All layers except the first use an MoE feed-forward block, and the standard residual connection is replaced with mHC parallel residual streams.
- 280 billion total parameters with 14 billion activated per token
- Pre-trained on 33.5 trillion tokens with a 128K training context
- Supports up to 1 million tokens for fine-tuning and inference
