A coding model with 138 billion parameters operates on one Windows PC

The coding model MAI-Code-1.1-Flash has 137.9 billion parameters. It works on Windows PCs with NVIDIA RTX Spark chips. The team cut it from 276 GB to 53 GB. A token is a small piece of text. On the PC the model writes a median of 51.12 tokens each second with an input of 120K tokens. This gives coding with no charge for each token in the cloud.

How it works

The model has many expert parts. These parts hold 96.6 percent of the parameters. The team gave each type of part the bits that it must have. A small draft model gives blocks of text, and the large model checks them in one step.

The numbers

The 53 GB version uses 3.31 bits for each weight. A second version is 46.1 GiB. The team gives no benchmark scores. It tells that 2 versions with the same score on one test can be different on a different test.

What is not known

There is no score for quality after compression. The speed is for one set up only. A long input uses more cache memory, and the cache uses the same space as the model weights.

Insight: Memory set up decides local AI results

The team tells that the cache gets more data with the input and uses the same memory as the weights. A different test on 4 RTX 3090 GPUs found that cache precision changed results more than model dimension. Speed numbers for local AI read as correct only with the memory set up. That set up is the number to find. Look for it from other vendors.