Curated ontology flags are near random flags in precision

Researchers checked if an ontology is a good checker of AI agent answers. An ontology is a schema, a map of the data types in a knowledge graph. A knowledge graph is a set of connected facts. They compared curated ontologies, ontologies that a model writes, and schemas that software finds in the data. On the answers of agents, the flags of the curated ontology were more accurate than random flags, but the difference was small.

How it works

Agents find answers on a knowledge graph. A checker made from the ontology gives flags on answers that look incorrect. The agent can then try to repair them. The tests used open agents and agents of companies, and one graph of almost 1,000 million facts.

What they found

• A checker can not find some errors.
• A checker can give flags on correct answers.
• On UniProt, a database of proteins, the curated ontology gave flags on more correct answers than the 2 other types.
• A curated ontology can give facts that are not the same as its data.

What is not known

The abstract gives no precision numbers. It does not tell how much a mix of sources, or feedback from the checker to the agent, helps.

Insight: A schema is not always a good checker

Google Cloud released a Knowledge Catalog that puts company rules in one map. In the internal tests of Google, agents that use it are almost 4X more accurate. Google uses rules as input for agents, and this paper tests rules as a checker. Possibly rules are strong as input to agents but weak as a checker, and the flag rate on correct answers shows this.