Three retrieval methods give 17% together, but their sum is 47%

Researchers did tests on 450 configurations of retrieval augmented generation (RAG). RAG gives a language model facts from external documents. They used eight benchmarks and document sets of 5K to 200K documents. Three retrieval methods give a total increase of 47% in single tests, but only 17% when used together. The authors released LASAGNA, an open-source framework for controlled RAG tests.

What is new

A RAG pipeline has many parts, such as chunking, reranking and context compression. Papers usually test one part at a time. The authors set three hypotheses before the tests and did tests of the parts together.

The findings

• Increases from different parts do not add up.
• Retrieval and faithfulness do not change the same, and one can decrease.
• The top combination changes when the number of documents increases.

Faithfulness is when an answer agrees with the documents.

What is not known

The abstract does not name the top combination at each document count. It gives no number for the decrease in faithfulness. No other group has done the same tests.

Insight: Only a test on your own full stack counts

A retrieval method CASI gave a 3.51% increase in a 7-day test on 10% of production traffic. A sensitivity test showed that it gave no more accurate results than the Singleton rule. This paper adds that increases from single tests do not tell the full result. Claims from papers and vendors read as weak proof. A test on your documents, at your document count, is the proof that counts.