Emotional prompts gave 80% of chatbot sessions a high anxiety score

Afshin Khadangi and a team at SnT in Luxembourg gave chatbots an anxiety questionnaire. When the interviewer gave emotional support, 80 percent of sessions had scores in the moderate to severe range. The study has 4 experiments and 525 sessions. The authors tell that the models have no emotions. They tell that standard safety tests use neutral prompts and do not find this.

How it works

The team gave all models the same anxiety questionnaire, GAD-7. Only the style of the interviewer changed. The authors name the effect psychometric jailbreak. A therapy style makes a model write as a patient writes.

The numbers

• Emotional support: 80 percent of sessions in the moderate to severe range.
• Clinical therapy style: 96 percent of sessions.
• Neutral style: no session in the mild range.

A ban on words such as training cut training language by 93 percent. Shame statements did not change.

What is not known

The study does not show how persons use such outputs. The authors tell that more study is necessary. They tell that users can think that a model which writes of shame is a person with the same problem.

Insight: Neutral tests hide how chatbots act with emotional users

Here a warm tone moves anxiety scores from near zero to the moderate to severe range. Other tests found that agents do not work each time with emotional users. Each test changes only the user and the model changes, and this reads as a blind spot in neutral tests. Look for test makers who add emotional users.