spider-rs/spider

About

The fastest web crawler and scraper for Rust.

Spider is a concurrency-first crawling engine built in Rust. It streams pages the moment they arrive, renders JavaScript only when a page demands it, and scales from a single script to a distributed fleet without changing your code. The same engine powers Spider Cloud, so you can prototype locally and move to managed infrastructure with one config change.

The hardest part of crawling at scale isn't the code. It's the proxies, headless browsers, and constant anti-bot churn. Spider Cloud runs all of that for you behind the same API.

Smart mode routes through proxies first and escalates to the unblocker only on pages that fight back, so you pay…

Excerpted from github.com/spider-rs/spider