Agents that send tables to the model are right in 45.7% of turns

Researchers measured a deployed award-travel agent in 24 dialogues, with 339 turns for each method. The method that keeps the records out of the model had 92.3% of turns fully correct, and the method that puts all records in the model had 45.7%. The data was the same, and only the method to give the records to the model was different.

How it works

The first method, SYNTHESIS, puts the full record sets in the context of the model, and the model answers follow-up requests alone. The other method, BYPASS, keeps the records in session state and turns each follow-up into arguments for a set filter. It sends the rows on a typed side channel.

The numbers

With more than 74 rows, the first method is correct in 8% of turns or less. With more than 100 rows, each correct answer costs about 200 times the flat $0.012 of the other method. The limit is more for better models.

What is not known

The test covers one travel agent and four models. The authors say that teams chose a side channel to show data, not to get correct data. They show that it also gives correct data.

Insight: The data method can be more important than the model

DataHub says agents with no business context are right about 50% of the time, and that its tool gives almost 100%. It names no external test. In this paper, a change of method lifted a travel agent from 45.7% to 92.3% with no new model. Maybe the data method matters more than the model, thus a better model or a context tool may not be the cheapest repair.