A check against the first text finds 77% of AI finance errors
A check that uses the same data as the writer found only 35% of the added errors.
Researchers made FinVeriQA. It is an open-source prototype that gives an answer to each finance question. It uses reports of companies in the United States and live market data. A check with the same data as the writer finds only 35% of incorrect numbers. A check against the first text finds 77%. The authors tell that a check must not use the data that the writer saw.
How it works
A small model sends each question to the correct tool. It does not answer questions for advice. Python software does the maths. Mistral-7B writes the answer and shows the sources. A code check tests that each ID that it shows is in the data and that numbers agree.
The numbers
- Mistral-7B did 29% of 100 calculations correctly. Python did 100%.
- A live price cut a price error from 210% to 0.00%.
- The small model got 0.85 for routing. Mistral-7B got 0.52.
- Hybrid retrieval found the correct text for 76% of 50 questions.
What is not known
In one live test, the check accepted 2 answers with AMD sample data in the names of NVIDIA and Alphabet. The error tests used one benchmark, FinRank. The authors give no result for users.
Insight: A check on the same data repeats the writer error
In an earlier study, researchers found that finance assistants give a number from the data that answers a different question. Checks against the data accept it. This study shows the same weak point: a check on the same data finds 35%. A check on the same data as the writer is weak, and maybe buyers will need a check on different data. Leaders must tell vendors to show the data that the check reads.
Sources
Posted