Fused-Softmax gives attention 2.1 to 3.2 times the speed on AMD GPUs
The method works without training, and the error that it adds is less than the error that low precision numbers cause.
Researchers introduced Fused-Softmax Attention, a method for attention, the step where a model weighs which parts of its input are important. New GPUs have more speed for low precision matrix work than for other math. Softmax, the step that turns scores into weights, became the slow part. On an AMD Instinct MI355X GPU, the new kernels have 2.1, 2.8 and 3.2 times the speed of BF16 attention.
How it works
The method changes softmax to let the fast matrix units do most of its work. It has no training step. The kernels, which are small units of GPU code, use the low precision formats MXFP8, MXFP6 and MXFP4.
The numbers
- 2.1, 2.8 and 3.2 times the speed of BF16, for the 3 formats.
- Wan2.2 video model: maximum 2.05 times the speed, same visual quality.
- AMD MLPerf Inference v6.1 entry had 1.7 times the speed of its last result.
The 2 results used the same hardware. The kernels are in the open source AITER library of AMD.
What is not known
The work is for the AMD CDNA 4 architecture. The source gives no results on other GPUs and no results for language models.
Insight: AMD is turning new GPU speed into shipped software
Low precision matrix speed on GPUs increases more than the speed of other math. AMD put the solution in an open source library and used it in a benchmark entry. This reads as a move by AMD to supply the software that lets its GPU use the added speed, with the result open to test. Builders on AMD GPUs can check if the same speed holds for their models.
Sources
Posted
