ModelShield finds 41 malicious models on Hugging Face

Researchers made ModelShield, a tool that operates an AI model and records what the model does on the computer. It does not read the files of the model. In a check of 145,000 models from public platforms, it found 41 malicious models on Hugging Face. One had a reverse shell (code that gives an attacker remote control). One sent host data to an external server.

How it works

The tool monitors calls from the ML framework (the software that loads a model) and calls to the operating system. It links each step to the model task that caused it. It then compares the steps with 195 attack patterns.

The numbers

It found all 29 published evasion variants. Of the 41 malicious models, ModelScan found 29% and the best other tool, ModelAudit, found 93%. The median check took 12.3 seconds.

What is not known

It misses payloads that do nothing bad when they operate. The results come from the tests of the authors. No other group tested the tool.

Insight: Attackers move to where file scanners do not look

One earlier paper hides an edit to model weights in GPU memory, and no file on disk changes. One more made a backdoor in an open model for less than $50. This reads as a race: attackers move away from files, so defenders must monitor what a model does. Maybe model hubs will operate such checks before a download. A file check alone does not show that a model is safe.