Find a scraping tool that fits your job.

See what each tool is for. Open a real test when you want proof.

22 tools12 tested on 30 sites

What do you need it to do?

Choose a job or search. No rankings.

Get HTML or API data

The data is already in the response. You do not need page JavaScript or clicks.

  • curl_cffiPython HTTP

    Try browser-like HTTP connections without rendering the page.

  • HTTPXPython HTTP

    Get HTML or API data with sync or async Python requests. No browser is opened.

    Older test only
  • Python requestsPython HTTP

    Get HTML from straightforward pages. It does not run page JavaScript.

    Older test only
  • ScraplingPython scraping

    Fetch pages and pull fields from HTML. Its browser modes are separate from our HTTP test.

  • wreq PythonPython HTTP

    Try browser-like network requests when you need HTML without opening Chrome.

Open and use a browser

The page needs JavaScript, scrolling, clicks, or a browser session.

  • BotasaurusPython scraping

    Build scraping workflows with browser and request helpers. Our test used its browser path.

  • CamoufoxModified Firefox

    Render and interact with pages using a Firefox browser built for anti-detection.

  • Chrome CDPBrowser protocol

    Control Chrome directly through its debugging protocol instead of a higher-level library.

    Not tested here
  • CloakBrowserModified browser

    Try a browser built for anti-detection work. We have not tested it here.

    Not tested here
  • HeliumPython browser

    Write short Python scripts to click through and read browser pages.

    Not tested here
  • LightpandaLightweight browser

    Render JavaScript with a lightweight browser engine when stealth is not the goal.

  • PatchrightBrowser automation

    Use Playwright-style scripts with stealth-oriented browser changes. Access is not guaranteed.

  • PlaywrightBrowser automation

    Render JavaScript pages, click, scroll, and read what the browser sees.

  • PuppeteerNode.js browser

    Control Chrome from JavaScript when a page needs rendering or clicks.

  • PydollPython browser

    Control Chrome through its debugging connection, without ChromeDriver.

    Not tested here
  • SeleniumBasePython browser

    Write browser scripts with Python helpers and examples. Our test did not use UC mode.

Work through many URLs

You need URL queues, link discovery, or a larger repeatable job.

  • CrawleeCrawler framework

    Manage URL queues and choose HTTP or browser crawlers. Our test used HTTP only.

  • ScrapyPython crawler

    Run larger crawls with parsing and data pipelines. Our test fetched one page at a time.

  • spider-rsRust crawler

    Discover and collect pages across a site with a Rust-based crawler.

    Not tested here

Let AI handle the steps

The task changes from page to page and needs decisions or actions.

  • browser-useAI browser

    Give an AI agent a browser task, such as finding data or filling a form.

    Not tested here
  • SkyvernAI workflow

    Let an AI system work through multi-step browser tasks.

    Not tested here
  • StagehandAI browser code

    Mix AI-guided page actions with browser code for changing tasks.

    Not tested here

Want to check a real run?

We tried 12 specific setups on the same 30 public sites, once each, without a proxy. See what came back, what failed, and the code used. One attempt is not a success rate.

Need a hosted service instead? Browse 16 providers. They are not part of this open-source test.