Attention layers find harmfulness and MLP layers write refusals
Post-training increases the MLP refusal in all 18 pairs, with no systematic increase of the harmfulness decodability.
Claimed, not confirmed
The researchers examined 18 pairs of models from Qwen, Llama and Gemma. Each pair has a base model and an aligned model. The harmfulness decodability is more in attention updates in the first layers. MLP updates do more to write the refusal. The researchers also made Sentinel, a method that uses the 2 signals. Across 8 models, its total error rate is lower than the rate of a method with 1 signal.
Sources
Posted
