Scraping software

Start with the job, then inspect a tool. A listed project is not a tested recommendation.

22 of 22 projects

Botasaurus
Tested 2026-09-29

Python toolkit with request and browser modes. The current 30-site test uses its browser workflow.

Recent field test
browser-use

Let an AI agent control a browser to complete a task.

Listed, not tested
Camoufox
Tested 2026-09-29

Use a modified Firefox browser built for automation and anti-detection.

Recent field test
Chrome CDP

Control Chrome directly through its debugging interface.

Listed, not tested
CloakBrowser

A browser built to reduce automation detection. We have not tested it publicly.

Listed, not tested
Crawlee
Tested 2026-09-29

Crawl pages with queues, retries and browser options. The current test measures one-page CheerioCrawler HTTP downloads, not crawling scale.

Recent field test
curl_cffi
Tested 2026-09-29

Make HTTP requests with browser connection profiles. The current 30-site pass uses direct Chrome150 mode without JavaScript.

Recent field test
Helium

Write short Python scripts to click and read browser pages.

Listed, not tested
HTTPX
Tested 2026-05-24

Fetch pages with synchronous or asynchronous Python code.

Historical test
Lightpanda
Tested 2026-09-29

A lightweight browser engine for rendering pages. Stealth is not its focus.

Recent field test
Patchright
Tested 2026-09-29

Control Chromium with Playwright-style code and stealth patches.

Recent field test
Playwright
Tested 2026-09-29

Control a browser for JavaScript pages and clicks. The current 30-site test uses plain headless Chrome with stock driver defaults.

Recent field test
Puppeteer
Tested 2026-09-29

Control Chrome from JavaScript. The current 30-site pass uses plain headless Chrome, stock launch defaults and one navigation per URL.

Recent field test
Pydoll

Control Chrome from Python through its debugging connection.

Listed, not tested
Python requests
Tested 2026-05-24

Fetch HTML in Python with a simple request. It does not run page JavaScript.

Historical test
Scrapling
Tested 2026-09-29

Fetch and parse pages in Python. Its browser modes need separate tests.

Recent field test
Scrapy
Tested 2026-09-29

Crawl sites in Python with queues, parsing and pipelines. The current test measures single-page HTTP downloads, not crawling scale.

Recent field test
SeleniumBase
Tested 2026-09-29

Python browser automation with helpers and UC mode. The current 30-site pass uses standard headless Chrome with UC off.

Recent field test
Skyvern

Run multi-step browser tasks with an AI system.

Listed, not tested
spider-rs

Crawl many URLs with a Rust-based tool. Crawl coverage has not been tested here.

Listed, not tested
Stagehand

Give a browser higher-level tasks with help from an AI model.

Listed, not tested
wreq
Tested 2026-09-29

Fetch pages with browser-like network fingerprints, without launching a browser.

Recent field test