A new test finds which agent skills to remove for a model
A rule that removes skills by gain removes more than half of the uncovered skills in a test with four models.
Researchers make a test for AI agent skills, which are instructions that agents keep and get from open hubs of a maximum of about 100,000 entries. As models change, a hub must remove the skills that make no difference to a model. The authors keep the skill away from the model, give it a task made from the skill, and score the result. They release 1,251 skills with tasks and scores.
How it works
The usual rules compare tests with and without a skill, and remove the skill on the gain. The gain is near zero in two conditions. The model has the skill, or the model and the skill do not give the capability.
The new method accepts a high score without the skill only when one other model gets a gain from it.
The numbers
- Almost half of the high scores come from tasks that give no model a gain.
- On SkillsBench, 28 of the 31 removals have probes that no test gets fully correct.
- Removal on the score without the skill does not decrease the reward that the authors measure.
- Agreement across tasks is a kappa score of 0.42 for the new method and 0.23 for the gain rule.
What is not known
The results are different for each model, and a hub must read them for each model.
Insight: A set of skills needs tests for each model
A report shows that skill changes after the agents start to work remove 52 to 93 percent of the gain. This test shows that a weak removal rule can also remove skills in error. A set of skills reads as software. It must have versions and a test for each model, and the group that runs the tests selects the skills that stay. If you use an open hub, test each skill for your model.
Sources
Posted
