An IBM plugin checks AI models as they work with 2% to 6% more time

IBM released vLLM-Hook MIA, a plugin for the vLLM engine. vLLM is open-source software that operates AI models for companies. In a report, IBM researchers give a speed result. The plugin is at most 20 times as fast as the first version, vLLM-Hook. Another method adds code to the model. The plugin is at most 14 times as fast as that method.

How it works

A fast engine records a group of steps one time and plays them again as one execution graph. Most tools that show the inner parts of a model only work when this mode is off, and the engine is then slow. MIA reads and changes the model before the execution graph is recorded. Then the mode stays on. Uses are to stop a dangerous message, mark an incorrect fact, or change the tone of a response.

The numbers

IBM made tests with 2 small models, a Llama 8B and a Qwen 1.5B. The added time was 2% in some tests. With a Llama 70B model, it was 6%. IBM researchers made these tests. The source names no other group.

What is not known

The source gives speed numbers only. It gives no data on how many dangerous messages it stops. It works only with vLLM. IBM and other groups also opened the Steerability Challenge.

Insight: Speed decides which AI safety check stays on

The plugin adds checks to vLLM, the primary open-source engine for the work of companies. Solcis says it checks each agent task in less than 200 milliseconds, with no test results. This reads as a race to hold the check point, and the fast check stays on. Tell each vendor where its checks operate and how much time they add.