What matters in AI.

Subscribe

Rules made from bugs cut weak code from an AI agent in a test

Without the rules, the agent wrote a weak trust check in 8 of 8 tries, and with them in 0 or 1 of 4.

An engineer made a test of a Cursor code agent that uses Claude Sonnet, version 4.6. The test had 12 usual tasks and 72 runs. Rules that the Foundry tool made from bugs in code made the agent check who connects to a server. The Project CodeGuard rules made no difference. This is part 4 of a series by the engineer.

How it works

Foundry makes one rule file from a bug that it found in code. Each task was done in a clean workspace with one of 3 rule sets. A script removed all comments. A judge that did not know the rule set, which is a model, then did a check of the code.

The numbers

  • Without the Foundry rules, the agent used a header that the client can write in 8 of 8 tries.
  • With them, it did this in 0 or 1 of 4 tries.
  • A test line, the canary, showed that the agent got the rules in 72 of 72 runs.

What is not known

The test is small and one person made it. The engineer did a check of the judge and found 3 verdicts that a person must check. One session used a library helper that the judge did not find weak.

Insight: A rule helps only if the agent gets it and it fits your code

Rules from one bug in Langflow code also made the agent safe with a different header. A benchmark found that agents broke rules in an internal document in 43.6% of runs when the task and the rule did not agree. This seems to be 2 sides: rules from your bugs can help, but only a check that the agent cannot remove is safe. Test that each rule is in use.

Sources

  1. Prove It: Does a Learned Rule Actually Change What Your AI Assistant Writes? – Security Loop Series, Part 4blogs.cisco.com
AI MATTER · NEWS · AI MATTER · NEWS ·16 NOV2026

Posted

Tags

Learn the terms in this story

More in Agents

All Agents news