Google MaxText adds training on inputs of 14 million tokens

Google Cloud added long context training to MaxText, its open source training software, in version 0.2.4. Google tells that the input can have 14 million tokens, which are small text units, on its TPU chips. The method is context parallelism. It divides one long input across many chips, so no chip holds all of it.

How it works

MaxText has four methods to divide the input:

• All-gather: the easiest method. Each chip collects data from all other chips.
• All-to-all: chips move data between them. The attention heads of the model limit the chip number.
• Ring: data moves around a ring of chips, with no such limit.
• Hybrid: joins the last two methods for more chips.

The numbers

Google tested Llama 3.1 8B. On TPU 5p, it used 70% of the chip capacity with inputs of 4 million tokens or less. On TPU 7, it kept 50% below 6 million tokens.

What is not known

Google gave results for one small model, and the tests are its own. It gave no cost, no training time and no comparison with other chips or software.

Insight: Google builds long context training around its own chips

MaxText is open source, but all four methods use Splash Attention, a kernel from the TPU library of Google. On 9 November, Google also announced TPU 8i Max for agents with very large context. This reads as a plan to give away software, so that long context work operates on Google chips. Builders must test if their code moves to other chips before they commit.