Find a scraping tool that fits your job.
See what each tool is for. Open a real test when you want proof.
What do you need it to do?
Choose a job or search. No rankings.
22 of 22 tools shown
Get HTML or API data
The data is already in the response. You do not need page JavaScript or clicks.
- curl_cffiPython HTTP
Try browser-like HTTP connections without rendering the page.
- HTTPXPython HTTP
Get HTML or API data with sync or async Python requests. No browser is opened.
Older test only - Python requestsPython HTTP
Get HTML from straightforward pages. It does not run page JavaScript.
Older test only - ScraplingPython scraping
Fetch pages and pull fields from HTML. Its browser modes are separate from our HTTP test.
- wreq PythonPython HTTP
Try browser-like network requests when you need HTML without opening Chrome.
Open and use a browser
The page needs JavaScript, scrolling, clicks, or a browser session.
- BotasaurusPython scraping
Build scraping workflows with browser and request helpers. Our test used its browser path.
- CamoufoxModified Firefox
Render and interact with pages using a Firefox browser built for anti-detection.
- Chrome CDPBrowser protocol
Control Chrome directly through its debugging protocol instead of a higher-level library.
Not tested here - CloakBrowserModified browser
Try a browser built for anti-detection work. We have not tested it here.
Not tested here - HeliumPython browser
Write short Python scripts to click through and read browser pages.
Not tested here - LightpandaLightweight browser
Render JavaScript with a lightweight browser engine when stealth is not the goal.
- PatchrightBrowser automation
Use Playwright-style scripts with stealth-oriented browser changes. Access is not guaranteed.
- PlaywrightBrowser automation
Render JavaScript pages, click, scroll, and read what the browser sees.
- PuppeteerNode.js browser
Control Chrome from JavaScript when a page needs rendering or clicks.
- PydollPython browser
Control Chrome through its debugging connection, without ChromeDriver.
Not tested here - SeleniumBasePython browser
Write browser scripts with Python helpers and examples. Our test did not use UC mode.
Work through many URLs
You need URL queues, link discovery, or a larger repeatable job.
- CrawleeCrawler framework
Manage URL queues and choose HTTP or browser crawlers. Our test used HTTP only.
- ScrapyPython crawler
Run larger crawls with parsing and data pipelines. Our test fetched one page at a time.
- spider-rsRust crawler
Discover and collect pages across a site with a Rust-based crawler.
Not tested here
Let AI handle the steps
The task changes from page to page and needs decisions or actions.
- browser-useAI browser
Give an AI agent a browser task, such as finding data or filling a form.
Not tested here - SkyvernAI workflow
Let an AI system work through multi-step browser tasks.
Not tested here - StagehandAI browser code
Mix AI-guided page actions with browser code for changing tasks.
Not tested here
Want to check a real run?
We tried 12 specific setups on the same 30 public sites, once each, without a proxy. See what came back, what failed, and the code used. One attempt is not a success rate.
Need a hosted service instead? Browse 16 providers. They are not part of this open-source test.