A method with a prior is above other methods when sources do not agree

Researchers study a model that answers from the text it finds and from its knowledge. The 2 sources do not agree when the text is not full or is incorrect. Rules that do not change try to select one source: follow the text, give the text more weight, or trust the memory. The authors say that each rule is incorrect in some use. They add a prior, an estimate of how many texts are correct.

How it works

The model writes its knowledge as notes. A small policy adds a confidence value to each of the notes. A reader gives more than one answer. A selector then gives one answer or gives no answer, with the prior.

The numbers

• With the correct prior, the system is above the top baseline at each mixture, by a maximum of 26 points.
• Its scores on GRACE are 45.3 and 35.0, against 29.4 and 27.5 for the top baseline.
• It is also above other methods on ClashEval, a test set that the authors set before the runs.

What is not known

The abstract does not say how a team finds the share of correct texts in a live system. Where the model does not have the knowledge, the method is not above the top baseline. The results are from the authors only.

Insight: Disagreement does not say which source is correct

A related study uses disagreement in a group of preference models as a sign of error. This paper shows that disagreement between 2 sources shows that there is an error, but not which source is correct. It reads as value that goes to the team that has data on how frequently its documents are correct. Monitor if tools that answer from documents add a control for that rate.