Stealthium shows how to find a hidden edit to AI model weights

The company Stealthium says it found a ROME attack in its lab. ROME means Rank-One Model Editing. A bad Python library changes some weights, the numbers that select what a model says, in GPU memory. The answers on one topic change, but no file on disk changes. Stealthium says its sensor connects each GPU task to the Python code that made it. It gave no detection rate.

How it works

Meng and others introduced ROME in 2022. The bad library changes one weight matrix in a middle layer when the model starts. The change is usual CUDA work from the same Python task. There is no file access and no network use. Tools that look at files or sockets see a usual task.

The proof

In the lab, a small PyTorch model gave two different answers to the same question with the same command. The file on disk did not change. Stealthium says the code paths behind the GPU tasks are different for ROME. It gave no test numbers.

What is not known

The work to find weight changes is not complete. The demo used a small model. The source does not tell how correctly the sensor works on large systems or how many incorrect alarms it gives.

Insight: Checking the model file is not enough

Earlier research hid encrypted networks in weights that audits could not find. This attack changes weights while the model operates, and the file stays the same. The two show one move: the fight goes from the model file to the machine in use. This reads as a new market for sensor vendors. If you operate models, find out what your tools see on the GPU.