Deepcrawl

Deepcrawl is an agent-oriented web scraping platform that extracts cleaned markdown content, hierarchical link trees, and metadata from websites. Designed for developers building AI agent workflows, it reduces token consumption and context switching by preparing data specifically for LLM consumption. The project describes itself as an open-source alternative to Firecrawl, offering better performance on general tasks and full transparency through its open codebase.

The toolkit provides typed contracts for type safety, a worldwide CDN for resilience, and developer-first tooling including a playground and npm package. It can be self-hosted on Cloudflare or Vercel. The GitHub repository indicates the project was built with Claude for code assistance.