Botasaurus on the same 30 public sites

Content checks, dates and tested modes

Alphabetical order. “Met checks” requires HTTP 200, complete capture, minimum text, every required word group and any listing patterns, with no checked soft-block phrase or navigation/capture/process error. The source-text parser excludes head, script, style, template and noscript; it does not resolve CSS visibility or verify complete field extraction.

Botasaurus had 0 HTTP 200 soft blocks and 0 HTTP 200 captures with no checked text. Other HTTP 200 pages can still fail required-content checks. Every failure stays visible below.

A failed check can also reflect the frozen rule: Amazon showed laptop listings with RON prices, while its price pattern accepts only $, £ or €. Example.com displayed documentation-example text, but its letter spans and SVG did not match the source-text phrases. Both remain failed checks. Best Buy showed a country chooser; Target’s listing area was unpopulated at capture time. These four nonempty HTTP 200 captures are not labelled as detected soft blocks.

All 30 targets, six dated attempts each

Manifest order. Open a target for every tool’s exact checks, timings, errors and private capture hashes.

The exact Botasaurus browser method

Botasaurus 4.0.97, botasaurus-driver 4.0.101, Google Chrome for Testing 153.0.8010.12, Python 3.12.3. The documented @browser / driver.get workflow uses normal page JavaScript and resources, with a fresh isolated Python process and temporary Chrome profile per URL. Profile, tiny_profile and reuse_driver are disabled. Ordinary session cookies can be set, then discarded on teardown. Downloads are denied through CDP.

Proxy environment variables are removed case-insensitively; Chrome receives --no-proxy-server. No login, Google-referrer navigation, proxy, paid service, human cursor or CAPTCHA/Cloudflare bypass. cache=False, output=None and max_retry=0 disable framework response caching, output and target retries. The installed driver can retry browser startup up to three total launches on startup errors; this is separate from target navigation. Exact per-attempt flags and observed user agent, webdriver, viewport, language and timezone are in metadata.

The driver’s default headless policy removes “Headless” from the user agent. user_agent, window_size and lang are unset; no custom fingerprint, locale/timezone override or pre-document JS is added. Driver browser flags remain in place. Sandboxing is enabled. Using the same Chrome binary as Patchright does not make the launch/control policies identical or identify the cause of different outcomes.

driver.get has a 20-second deadline and waits for document.readyState to become interactive or complete, then a fixed two-second settle. This is not an exact DOMContentLoaded-event wait. Capture time includes navigation, settling, final-URL read and DOM serialization; startup, screenshot and teardown are outside that number. DOM capture is capped at 5 MiB, with a five-second stage deadline. No per-network-response byte cap is applied. Main-frame CDP document events supply status and ordinary HTTP redirect history; other final-URL changes are labelled separately. Chromium’s native redirect cap is 20.

A 40-second whole-worker watchdog, 3 GiB minimum available host RAM and 4 GiB sampled worker-tree PSS cap bound each attempt. The 1260-second slice check runs before each target. Targets are sequential with a two-second gap. PSS counts private memory plus each process’s share of shared pages. Samples every 100 ms include Python, driver and browser children from spawn through startup, capture, screenshot and teardown; the batch parent is excluded. Brief peaks or changing processes may be missed. These are sampled totals, not exact peaks or pure browser memory.

Run UTC: 2026-09-29T21:05:10.963199+00:00 → 2026-09-29T21:08:28.347202+00:00. Host preflight: 13638 MiB available of 32050 MiB. Full HTML, screenshots and driver logs stay private. Cleanup signals only recorded child PID/start-time identities. Each worker’s short private TMPDIR also keeps the driver’s profile garbage collection inside owned directories; Docker’s global zombie-cleanup mode is refused.

Before this batch, a long private TMPDIR made Chrome abort with “Socket path too long” before any navigation. A short owned directory fixed startup while retaining sandboxing. A local blank check and one successful frozen Web Scraping Dev canary followed. That extra canary is excluded from the 30-site count; this safe retry sent 31 public target navigations in total, plus ordinary redirects and page resources. Inspect the diagnosis and separate canary.

Code, dependency pins and inspection files

Use the pinned dependencies and exact existing Chrome binary. The runner refuses a changed binary hash, unsafe preflight or existing output directory. A rerun sends 30 new navigations; it does not install Chrome. This pass tests one browser workflow, not Botasaurus’s request mode, crawling or extraction helpers.

/tmp/scrapingevals-botasaurus-20260929/bin/python public/evidence/botasaurus-public-30-20260929/run.py --binary /tmp/scrapingevals-patchright-amazon-20260929/browsers/chromium-1243/chrome-linux64/chrome --manifest runner/corpora/public-30-v1.json --output runner/data/botasaurus-public-30-20260930-retry

Code SHA-256

fb5be15d4e6a21f960a4e479cc7ffe89f4adaa888ffc9b020d60656c9637412c

Chrome SHA-256

8c599d43aec53f2460a31ae2f4af6bd863f8258b34ff519564bc5d4726bfaa1e

Frozen manifest SHA-256

6ce519c4f7df3c32f5dfda379a8d3b280b7723bdeb48ddf64a1ae10ce6a7914c

The official Botasaurus repository and focused fetch-mode DeepWiki question guided the workflow. Installed APIs matched the official 4.0.97 release source and driver 4.0.101 release source byte for byte. Current repository summaries are not release-specific behavior; inspected driver source establishes the headless user-agent override. Read the general method.