Firecrawl and Apify get more right answers than Claude web search

A benchmark put 200 data tasks through 4 data tools for AI agents and through the web search of Claude. A data tool gets data from sites for an agent. The 4 tools were Firecrawl with and without its Alexandria catalog, Apify and Zyla. The cost of the 2 Firecrawl setups was $0.73 to $1.36 for each 100 questions. The cost of Claude web search was $2.93 to $3.37.

How it works

Each tool connected to the same agent, Claude Haiku 5.5, through MCP. MCP is a standard that lets agents use tools. The agent had no web tool. Alexandria adds connectors for sites to Firecrawl web search.

The numbers

• Alexandria got 85.3% on protected sites, against 80.0% for Firecrawl without it.
• Apify got 90.0% right on open data, but its cost was $8.75 for each 100 questions.
• Zyla got 43.7% right on open data.

What is not known

On open data, the Alexandria increase was in the margin of error. On trap questions that have no answer, one run gave an incorrect value. Alexandria gave a follower count for a nonexistent account.

Insight: With the same standard for all, catalogs and fees decide

Each tool in the test connected through MCP, a standard that the Agentic AI Foundation holds. Apify fees were 95% of its cost on open data. Alexandria used its catalog in 46% of the runs on protected sites. When the connection is the same for all, the contest reads as one of catalogs and fees. Leaders must compare the cost for each right answer, not the tool name.