Why Netweir is one library
A scraper that works today is usually Scrapy, an impersonation library, Playwright and a parser, held together by settings and hope. Here's what that costs you, and what Netweir does instead.
Ask anyone who scrapes for a living what their stack looks like, and you’ll get a list. Scrapy runs the crawl. curl_cffi, plugged in through scrapy-impersonate, makes the TLS handshake look like a browser’s. Playwright, through scrapy-playwright, renders the pages that are empty until their JavaScript runs. And parsel, Beautiful Soup or selectolax reads the HTML, depending on who wrote the spider and when.
Every one of those is good at its job. What hurts is everything around them.
There are two HTTP stacks in the project now, and a setting that decides which one a request goes through. There’s a browser that has to be told which cookies the HTTP client earned, and the other way round. There’s a parser that’s fine on a 50 KB page and takes most of a second on a 1 MB one. And there’s the Tuesday morning when the shop moved its prices into a new div, the crawl still exited 0, and the price column in a thousand rows came out empty.
I built Netweir because I got tired of maintaining the space between four libraries. It’s one package, Rust underneath and Python on top, and this post is about what that changes on an ordinary day.
Where the day goes
If you’ve run crawls for a while, you’ll recognise most of these.
The block that looks like a page. A bot-protection challenge comes back as a 403, or sometimes as a 200 with a page of script. Your callback parses it happily, finds no products, and moves on. Nobody finds out until someone asks why yesterday’s crawl has a fifth of the items.
The handshake that gives you away. Most sites that block scrapers don’t read your code. They read the first few hundred bytes of the connection: which TLS ciphers and extensions arrive, what the HTTP/2 settings frame says, which headers come first. Python’s HTTP libraries give themselves away before they’ve asked for anything. So you add an impersonation library, and then you spend an afternoon making sure the headers your middlewares add haven’t undone what it did.
The redesign. A selector that worked for eight months stops matching. There’s no error, because an empty match isn’t an error. There’s just a column of null.
The wait. Parsing is meant to be the cheap part, and on a small page it is. On a 1 MB product listing, parsel takes about a third of a second to pull out every price, title and link. On a 10 MB page it takes 54 seconds. Multiply by a crawl.
Looking like Chrome, byte for byte
A request from Netweir is what Chrome 155 sends: the same ClientHello, the same HTTP/2 settings and priorities, the same headers in the same order, and over HTTP/1.1 the same capitalisation. Cookies go where Chrome puts them, and redirects are followed hop by hop the way Chrome follows them.
import netweir
page = netweir.get("https://books.toscrape.com/")
page.css(".price_color::text").getall() # ["£51.77", "£53.74", ...]You don’t have to take that on trust, and I’d rather you didn’t. Chrome was recorded doing four navigations against a local server: a page that sets a cookie, a revisit, a redirect to another site and one within the site. A test has Netweir do the same four and fails if any request differs from Chrome’s. The public checker at tls.peet.ws reports the same JA4 fingerprint for both, t13d1517h2_8daaf6152771_cb7bf5808d99.
Firefox 158 is there too, as profile="firefox", held to a recording of the real browser by the same test. So is Safari 27, as profile="safari", down to its habit of sending two cookies with the same path newest first. There’s one known difference, and it’s written down: Chrome sends a priority header only over HTTP/2, and Netweir can’t know a server lacks HTTP/2 until it has answered once. So the first request to an https server that speaks only HTTP/1.1 carries that header. Every request after it doesn’t.
There’s one HTTP stack, so there’s nothing to keep in step. BoringSSL is inside the wheel. pip install netweir is the whole of the setup.
A block is a block, and the crawl climbs out of it
Every response is checked before your code sees it, against the block pages of Cloudflare, Akamai, DataDome, HUMAN, Kasada, Imperva and AWS WAF. A challenge that arrives as a 403 isn’t mistaken for a page, and page.blocked names the vendor.
In a crawl, a blocked request climbs a ladder you don’t have to build. It’s tried again with a new session, meaning an empty cookie jar and the next of your proxies if you gave several, and the site is slowed down. A 429’s Retry-After is honoured. A site that keeps blocking is paused for a while instead of hammered. And with one setting, a request that’s still blocked after all that gets a go in a real Chrome:
class Shop(netweir.Spider):
settings = netweir.Settings(browser="on_block")Chrome waits for the challenge to pass, then hands the cookies it earned back to the HTTP client, so the rest of the site goes at HTTP speed. If even that fails, your spider’s on_block(request, page) hears about it. Nothing about it is silent.
The Chrome is driven so that it doesn’t announce itself. navigator.webdriver is false, the user agent doesn’t say HeadlessChrome, and the DevTools features that page scripts can notice are never switched on. A test checks all three. It isn’t everything a site can check, and the browser guide lists what still shows.
A redesign is a warning, not an empty column
Name a selector, and Netweir remembers what its element looked like:
price = page.css(".price_color::text", track="price")When a new build renames price_color, wraps it in another div or swaps h1 for h2, the selector stops matching, and Netweir looks for the element most like the one it remembers: its tag, its text, its attributes, what’s around it, the label just before it. If the best candidate is close enough, you get it, along with price.relocated to say it had to look and price.score to say how sure it is. If nothing is close enough, you get nothing, rather than a guess. A Shipping cell is never offered up as the Total because the Total row went missing.
For the redesigns that get past that, checks turn a crawl that finished into a crawl that has to have worked:
class Books(netweir.Spider):
checks = [netweir.Checks(min_items=900, coverage={"title": 1.0, "price": 0.95})]When one fails, netweir crawl exits 1, so cron or CI notices on the morning it happens. netweir check runs Scrapy-style contracts from your docstrings against live pages, if you’d like to know before the scheduled crawl does.
And the crash at hour five? Give a crawl somewhere to keep its state, with -s checkpoint=crawls/books, and it picks up where it stopped after a crash, a reboot or a kill -9. Pages already done aren’t fetched again, and no item lands in the output twice. The tests kill a crawl at random moments until it finishes and check exactly that.
Fast where the time goes
Netweir parses with its own HTML parser, written in Rust from the HTML Standard. (Until 0.5.0 it used lexbor, the C parser selectolax uses; here’s why it changed.) Then the whole query runs in Rust, and Python gets a finished list of strings, not a Python object for every element along the way. The GIL is released while it works, so threads parse in parallel.
Pulling every price, title and link out of a generated shop page:
| 1 MB page | 10 MB page | |
|---|---|---|
| Netweir | 3.1 ms | 32 ms |
| selectolax | 5.5 ms | 66 ms |
| BeautifulSoup (lxml) | 173 ms | 2.0 s |
| parsel | 295 ms | 54 s |
Against selectolax, Netweir takes a little over half the time. Against what most Scrapy spiders use, it isn’t close. XPath tells the same story: six queries on the 1 MB page take Netweir 7.7 ms and lxml 825 ms.
A crawl is more than parsing, so there’s a benchmark for the whole thing. One local server plays 100 sites of 100 pages, each answer 50 ms late, and each crawler fetches all 10,000 pages with the same limits, 100 requests in flight and 8 per site:
| Pages a second | CPU per page | Peak memory | |
|---|---|---|---|
| Netweir | 1,770 | 0.22 ms | 68 MB |
| Scrapy 2.19 | 890 | 1.1 ms | 125 MB |
| httpx + selectolax | 220 | 3.2 ms | 190 MB |
With 100 requests in flight and 50 ms an answer, 2,000 pages a second is the most any crawler could manage there. Netweir is waiting on the server, not on itself.
Those numbers are from an Apple M-series laptop, and every one of them is a script in the repository you can run on your own machine. The parsing benchmarks also run in CI, and the build fails if any library beats Netweir. The benchmarks page has the commands.
What one library changes
Fewer moving parts, mostly, and that’s a boring benefit in the best way. There’s one install, one set of settings and one log. A cookie Chrome earns is already in the client’s jar. Blocks, retries and pages Chrome got past are all counted in the same stats.
It also means the parts can know about each other. The duplicate filter knows which cookie jar a request belongs to, so Bob never gets Ada’s page back from the cache. When Chrome gets past a block and the HTTP client carries on, it carries on with the same language, user agent and client hints, so the site sees one visitor rather than two. A spider that only declares what it wants (the links to follow, the fields to take) runs entirely in Rust, with no Python per page at all. On a local 1,050-page shop that took about two thirds of the time of the same spider written as callbacks.
What it doesn’t do yet
The browser driver drives Chrome. Firefox will follow when there’s a Firefox that can be driven without announcing it.
A parse timeout stops a page built to be slow, and a crawl reads at most 64 MiB of a response. Nothing in Netweir caps the parser’s own memory yet, though, so a small page built to abuse HTML’s rules for misnested tags can still make it allocate a lot.
Safari’s place for an Authorization header, and the requests it makes for images, are written from what Safari is known to send, and haven’t been checked against a recording yet. Chrome’s and Firefox’s have.
And it’s AGPL-3.0. If that doesn’t work for your company, there’s a commercial licence.
If you live in Scrapy
You don’t have to move to find out whether the fetching helps. Netweir ships a download handler for Scrapy, so two lines in settings.py send every request as Chrome would, and the rest of your project stays as it is. If you do want to move, your selectors, pipelines, middlewares and item loaders mostly come with you. Coming from Scrapy goes through it piece by piece, including what doesn’t carry over.
Try it
pip install netweir
netweir shell https://books.toscrape.com/That’s a Python prompt with the page in page, fetched as Chrome fetches it. The getting started guide goes from there to a crawl that writes a file in about ten minutes. If Netweir saves you a library or two, star the repo so the next person finds it, and tell us what you’d want next in Discussions.