A developer releases pagesift, a Rust web crawler for AI agents

YOLOW92 released pagesift, a web crawler written in the Rust language. It turns web sites, PDF files and Office files into clean Markdown text for AI agents. Agents use this text in RAG, where a model looks up documents before it writes an answer. The developer tested version 0.80.0 on 27 October 2026. In a local test, the tool read 200 million pages in 211 seconds.

How it works

One tool turns a site, a document or an archive into text and structured data. It copies the web interfaces of Firecrawl and Jina Reader, other tools. It reads the memory limit of its host and sets the number of connections to fit it.

The numbers

The local test gave 947,611 pages each second, not a speed on the internet. In a crawl of 32 university sites, useful pages rose from 29.6% to 66.1%, graded by a script of the developer.

What is not known

All tests come from the developer. No external test is given. For one change, the test sites could not be used, and those numbers are synthetic.

Insight: Rust tools for agents compete on small memory use

Cloudflare reports that its Rust checker tsrs uses 70% less memory than TypeScript 7.0, for code agents that share servers. Pagesift also adapts to a memory limit and copies the Firecrawl and Jina Reader interfaces. This reads as a move to be the primary part of an agent pipeline, where a known interface and small memory use can win. If you operate many agents, test memory for each task, not only speed.