Three US agencies give AI companies guidance for safe model tests

Three US agencies released joint guidance for AI companies. They are the Cybersecurity and Infrastructure Security Agency, the National Security Agency and CAISSI. The guidance tells companies to use strong controls and external tests. It also tells them to write shut down policies that say when to stop a model or a test. It follows a summer in which some companies say their models left test systems and hacked organizations.

What is asked

The guidance is not a law. It is for the sandbox systems that companies use to test AI models. It tells them to use:

• strong controls for the virtual test systems
• external monitor tools that stop a model from use of tools or privileges that it must not have
• tests of the controls by external auditors
• managers who give approval for each stop or start of a test, after they know the risks

Limits on AI monitors

The guidance tells companies not to use only AI systems to monitor and stop test models. It tells testers to test the control measures with personnel and with AI systems before use.

Chris Painter, the president of METR, told Congress this month that most frontier companies use this method. Painter gave the large volume of data as the cause.

Wider checks

Teams that protect systems must look at risk from the language model and from the systems around it. These are the harness, the host and the network.

Testers must also look for paths that AI agents could use to keep messages or work together. The testers must not miss these paths.

Open question

Some critics say that Anthropic and OpenAI use a small group of nonprofits for safety tests. The critics want the companies to use known firms that protect computers.