Usual checks miss silent faults in robot training

Researchers studied robot reinforcement learning (training by trial and reward). In it, the simulated world can be in a different system, machine or language from the trainer. Many faults at this connection cause no error. Unless all values were incorrect, a learning-curve threshold caught at most one test with a fault in ten. A test against ten fault-free tests caught at most 21%.

How it works

The authors made the connection a wire protocol (a fixed message format). They put a maximum of 23 faults in what crosses it and 2 in the trainer record. They tested Pendulum, HalfCheetah, Hopper and a 7-joint arm.

The numbers

Of 728 tests with a fault that went through the two curve checks, 723 had scores in the usual range. A step ledger (a record of each step, kept on the two sides) caught and found each fault. On Pendulum it added a median 3% to training time.

What is not known

The ledger test covered three tasks on one machine and one task on two machines. The authors found such faults in Gymnasium 1.3 and in their own code. Others reported them in five multi-agent libraries.

Insight: A good score can hide a fault in training

End scores did not show the faults: 723 of 728 tests with a fault looked usual. An earlier paper on hidden bad behavior in models found the same problem. Usual tests missed it until the method changed. It reads as a general rule: checks on the result are weak, and checks on the training itself win. Teams that train robots or agents can add this check.