Baidu releases a 280 billion parameter base model for more training

Baidu released ERNIE-5.5-Flash-Base, a language model with 280 billion parameters. Parameters are the numbers that a model learns. Only 14 billion work on each token, a small unit of text. Baidu used 33.5 trillion tokens to train it from the start. The weights are the files from the end of the first training step. Developers can train them more for their tasks.

How it works

The model is a mixture of experts. It has many expert parts, and each token uses only some of them. One layer in seven keeps a full view of the text. The other layers compress each 128 tokens into one entry. This uses less memory.

The numbers

The training context is a maximum of 128K tokens. The model accepts 1M tokens for fine-tuning and for use. The new gate removes 75 percent of the gate weights, and Baidu tells that its tests show an improvement. The text gives no benchmark scores.

What is not known

The text gives no licence, no cost to operate the model, or results against other models. The weights are from before more training. This is a start point, not a complete model.

Insight: Open models compete to be the start point for other teams

Ant Group released Ling-3.1-flash-1019 on 27 October with 25 billion active parameters and an MIT licence. Its score of 44 came with 59 percent more tokens written, and Baidu gives base weights with 14 billion active. This reads as a move to be the start model that other teams select, where the cost for each task can count more than a score.