Coming from Scrapy | Netweir blog

If you write Scrapy, you already think the way Netweir does. A spider with callbacks, requests that come back as responses, items that go through pipelines, middlewares in between. None of that goes away. What changes is what’s underneath: fetching that looks like Chrome, a parser in Rust, and an engine that keeps the queue out of Python.

This post is for you if your spiders came first. It goes through a Scrapy project piece by piece, says what each one becomes, and is honest about the parts that don’t carry over. Every snippet here runs on Netweir 0.4.0.

What carries over

Scrapy	Netweir
`scrapy.Spider`, `start_urls`, `parse`	`netweir.Spider`, `start_urls`, `parse`
`async def start()`	`async def start()`
`scrapy.Request`, `response.follow`	`netweir.Request`, `page.follow`
`response.css`, `response.xpath`	`page.css`, `page.xpath`
`ItemLoader`, `MapCompose`, `TakeFirst`	the same names, from `netweir.loader`
`ITEM_PIPELINES`	`pipelines = [...]` on the spider
`DOWNLOADER_MIDDLEWARES`, `SPIDER_MIDDLEWARES`	`middlewares = [...]`, `spider_middlewares = [...]`
`CrawlSpider` with `Rule` and `LinkExtractor`	`rules = [netweir.Follow(...)]` on any spider
`scrapy crawl`, `shell`, `genspider`…	`netweir crawl`, `shell`, `genspider`…

The rows that take real thought are settings and a handful of request details, and we’ll get to both.

The spider

Here’s a Scrapy spider you’ve probably written a version of:

import scrapy


class BooksSpider(scrapy.Spider):
    name = "books"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com/"]
    custom_settings = {"CONCURRENT_REQUESTS": 16, "DEPTH_LIMIT": 3}

    def parse(self, response):
        for book in response.css("article.product_pod"):
            yield {
                "title": book.css("h3 a::attr(title)").get(),
                "price": book.css(".price_color::text").get(),
            }
        yield from response.follow_all(css="li.next a", callback=self.parse)

And here it is in Netweir:

import netweir


class Books(netweir.Spider):
    name = "books"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com/"]
    settings = netweir.Settings(concurrency=16, max_depth=3)

    def parse(self, page):
        for book in page.css("article.product_pod"):
            yield {
                "title": book.css("h3 a::attr(title)").get(),
                "price": book.css(".price_color::text").get(),
            }
        yield from page.follow_all(page.css("li.next a::attr(href)"), callback=self.parse)

Three lines changed. The settings are a typed object rather than a dict of upper-case names, so a misspelt one is an error instead of a setting that quietly does nothing. The callback gets a page. And the link is selected down to its href, because page.follow and follow_all take URLs rather than <a> elements.

Run it the way you’d run it in Scrapy, and the file extension picks the format:

netweir crawl books.py -o books.jsonl

Out of the box it reads robots.txt, skips pages whose owners reserve text and data mining rights, paces each site by how fast it answers, and drops duplicate URLs, including ones that differ only by utm_ tags.

Responses and selectors

A Netweir Page is your Scrapy response. page.url, page.status, page.headers, page.body, page.text, page.meta and page.urljoin() all mean what you’d expect.

Your selectors should work unchanged. ::text and ::attr(), get() and getall(), re() and re_first(), .attrib, and the older extract() and extract_first() are all there. XPath is all of XPath 1.0 with parsel’s extras:

page.xpath("//p[has-class('price_color')]/text()").get()
page.xpath("//article//a[re:test(@href, '_\\d+/index\\.html$')]/@href").getall()
page.xpath("//li[@class=$cls]", cls="next").get()

Two things come out slightly differently, on purpose. get() on an element writes its HTML the way Chrome’s outerHTML does, so an attribute holding a quote comes out as " where lxml would switch to single quotes. The elements and their order are the same. And XML pages are read as XML, with register_namespace() and remove_namespaces() working as they do in Scrapy.

There’s no response.json(). For a JSON API, json.loads(page.text) does it, and with the netweir[jmespath] extra installed, page.jmespath("products[].name") queries it directly.

Items and loaders

A callback yields dicts or dataclasses. Loaders carry over with a change of import:

import netweir

# from scrapy.loader import ItemLoader
# from itemloaders.processors import MapCompose, TakeFirst
from netweir.loader import ItemLoader, MapCompose, TakeFirst


class BookLoader(ItemLoader):
    default_output_processor = TakeFirst()
    title_in = MapCompose(str.strip)
    price_in = MapCompose(netweir.parse_price)


def parse_book(self, page):
    loader = BookLoader(response=page)
    loader.add_css("title", "h1::text")
    loader.add_css("price", ".price_color::text")
    loader.add_value("url", page.url)
    yield loader.load_item()

add_css, add_xpath, add_value, nested loaders, context and every processor work as Scrapy’s do, into dicts, dataclasses, attrs classes or Scrapy Items. There’s nothing extra to install, and on a listing of a thousand products they take about three quarters of the time itemloaders and parsel take.

A few things differ, all deliberately. The ones you’re most likely to hit: a field that’s None stays None when nothing is added to it, where Scrapy’s loader turns it into []. And Pydantic models aren’t supported as items. The loaders guide has the full list.

Pipelines

A pipeline written for Scrapy works as it is. What changes is where you list it:

import sqlite3


class Database:
    def open_spider(self, spider):
        self.db = sqlite3.connect("books.db")
        self.db.execute("create table if not exists books (title, price)")

    def process_item(self, item, spider):
        self.db.execute("insert into books values (?, ?)", (item["title"], item["price"]))
        return item

    def close_spider(self, spider):
        self.db.commit()
        self.db.close()


class Books(netweir.Spider):
    pipelines = [Database]

A class in the list is made when the crawl starts, with from_crawler(crawler) if it has one. They run in list order, the order of their numbers in ITEM_PIPELINES. Raising DropItem, Netweir’s or Scrapy’s, drops the item. A plain function also works as a stage: it gets the item and returns it, changes it, or returns None to drop it.

Files and images come across too. netweir.files("store/") and netweir.images("covers/") read file_urls and image_urls as Scrapy’s pipelines do, store files in Scrapy’s layout, and download through the crawl itself, so they keep its robots.txt, delays and proxies.

Middlewares and signals

Downloader middlewares keep their three methods:

class Polite:
    def process_request(self, request, spider):
        request.headers["From"] = "[email protected]"


class Books(netweir.Spider):
    middlewares = [Polite()]

process_request can return a page of its own and skip the fetch, or another request to send in its place. IgnoreRequest, CloseSpider, DontCloseSpider and NotConfigured all work. Spider middlewares go in spider_middlewares, in the order of their numbers in SPIDER_MIDDLEWARES, lowest first.

Signals reach any middleware or pipeline with a method of the signal’s name: spider_opened, item_scraped, spider_idle, spider_closed and most of the rest. A component that connects them itself in from_crawler, with crawler.signals.connect(...) and Scrapy’s own scrapy.signals objects, carries over unchanged.

Following links by rule

A CrawlSpider becomes rules on an ordinary spider:

class Catalogue(netweir.Spider):
    start_urls = ["https://books.toscrape.com/"]
    rules = [
        netweir.Follow("li.next a"),
        netweir.Follow("article.product_pod h3 a", callback="parse_book"),
    ]

    def parse_book(self, page):
        yield {
            "title": page.css("h1::text").get(),
            "upc": page.xpath("//th[.='UPC']/following-sibling::td/text()").get(),
        }

Follow takes allow, deny, allow_domains and deny_domains, as LinkExtractor does, and follow defaults the way Scrapy’s Rule does. Swap the callback for extract= and a netweir.Item that declares its fields, and the pages are read and turned into items in Rust, with no Python per page. On a local 1,050-page shop that took about two thirds of the time.

Settings

Most of the settings you use have a Netweir setting that does the same job, under a lower-case name. These are the ones you’ll reach for first:

Scrapy	Netweir
`CONCURRENT_REQUESTS`	`concurrency` (64 by default)
`CONCURRENT_REQUESTS_PER_DOMAIN`	`per_domain` (8)
`ROBOTSTXT_OBEY`	`obey_robots` (on)
`AUTOTHROTTLE_ENABLED`	`throttle` (on)
`AUTOTHROTTLE_START_DELAY`, `AUTOTHROTTLE_MAX_DELAY`	`start_delay`, `max_delay`
`DEPTH_LIMIT`	`max_depth`
`DOWNLOAD_TIMEOUT`	`timeout`
`RETRY_TIMES`, `RETRY_HTTP_CODES`	`retries`, `retry_http_codes`
`REDIRECT_MAX_TIMES`	`max_redirects`
`CLOSESPIDER_ITEMCOUNT`, `CLOSESPIDER_TIMEOUT`	`max_items`, `max_time`
`JOBDIR`	`checkpoint`
`HTTPCACHE_ENABLED` and `HTTPCACHE_DIR`	`cache`
`LOGSTATS_INTERVAL`	`stats_interval`

They go on the spider as netweir.Settings(...), or on the command line as -s concurrency=16. A project’s shared settings live in spiders/settings.py as one Settings object that each spider takes whole or changes a field of. Every setting is listed with its default.

The request meta keys download_timeout, dont_redirect, redirect_ttl, http_user, http_pass, http_auth_domain and proxy do what they do in Scrapy. So do cookies= and meta["cookiejar"], for several logged-in users in one crawl.

The command line

Your hands already know it:

Scrapy	Netweir
`scrapy startproject shop`	`netweir startproject shop`
`scrapy genspider books books.toscrape.com`	`netweir genspider books books.toscrape.com`
`scrapy crawl books -o books.jsonl`	`netweir crawl books -o books.jsonl`
`scrapy runspider books.py`	`netweir runspider books.py`
`scrapy list`, `parse`, `check`, `version`	`netweir list`, `parse`, `check`, `version`
`scrapy shell`, `fetch`, `view`	`netweir shell`, `fetch`, `view`
scrapyd	`netweir serve`

-a category=fiction sets a spider attribute as Scrapy’s spider arguments do. netweir check runs the contracts in your docstrings, @url, @returns and @scrapes included, and exits 1 when one fails, so a redesign shows up in CI before it shows up in your data. genspider has the basic, crawl, xmlfeed and csvfeed templates, plus sitemap.

What doesn’t carry over

Most of these are a line or two of change. All of them are worth knowing before you start.

start_requests() isn’t called. Netweir has Scrapy 2.13’s async def start() instead, so rename the method. A spider that only has start_requests() fetches nothing at all.
There’s no cb_kwargs on a request. Pass what the callback needs in meta, and read it back from page.meta.
Components are listed as classes or objects on the spider, not as dotted paths with numbers in settings.
Scrapy’s setting names aren’t translated. There’s no spider.settings.get("CONCURRENT_REQUESTS"). Settings are attributes, as spider.settings.concurrency, and crawler.settings.get() reads only the spider’s custom_settings.
process_request runs when your code makes the request, before the duplicate filter, rather than just before the fetch, because the queue lives in Rust.
An errback gets (request, error), not a Twisted failure.
request_reached_downloader, headers_received and bytes_received aren’t sent. Rust hands Python whole responses, and stopping between chunks to call Python would cost every page what the engine is there to save.
DOWNLOAD_DELAY has no setting of its own. The throttle sets each site’s delay from how quickly it answers, never below min_delay, which is the closest match. And the delay counts from when a request actually leaves, after its connection is made, not from when it was handed out, so a slow handshake can’t let the next request overtake it.
There’s no telnet console. Read the stats from the metrics endpoint (metrics_port), pause with Ctrl-C and a checkpoint, and try selectors in netweir shell.
Files and images have no file_path() to override, and no FTP store. Names are always the SHA-1 of the URL.
scrapy bench, scrapy settings and scrapy edit have no Netweir counterpart.

Not ready to move?

Then don’t. Keep the project on Scrapy and let Netweir do the fetching. Two lines in settings.py:

DOWNLOAD_HANDLERS = {
    "http": "netweir.scrapy.DownloadHandler",
    "https": "netweir.scrapy.DownloadHandler",
}
NETWEIR_PROFILE = "chrome"  # or "firefox", "safari"

Every request then goes out as Chrome sends it, from the TLS handshake to the order of the headers. Scrapy’s redirect, cookie and retry middlewares carry on doing their jobs, its proxies from meta["proxy"] still work, and AutoThrottle still gets each response’s latency. One thing to know: the browser’s own User-Agent comes from the profile, so your USER_AGENT setting doesn’t reach the site. It needs Scrapy 2.15 or later, on its default asyncio reactor.

What you get that you didn’t have

Fetching that looks like Chrome without a second HTTP stack. A request from Netweir is what Chrome 155 sends, and a test holds it to a recording of the real browser.

Blocks you hear about. Block pages from Cloudflare, Akamai, DataDome, HUMAN, Kasada, Imperva and AWS WAF are recognised before your callback sees them. The crawl retries with a new session, slows the site down, and with browser="on_block" gives a stubborn request one go in Chrome, then hands Chrome’s cookies back to the HTTP client.

Selectors that survive a redesign. page.css(".price_color::text", track="price") finds its element again when the class is renamed, and tells you it had to.

A crawl that has to have worked. checks = [netweir.Checks(min_items=900, coverage={"price": 0.95})] makes netweir crawl exit 1 when the crawl finished but the prices didn’t come with it.

Speed. On the crawl benchmark in the repository, 10,000 pages from a local server answering 50 ms late, Netweir fetched 1,770 pages a second to Scrapy’s 890, with a fifth of the CPU per page. Why Netweir is one library has the tables.

Try it on your own spider

pip install netweir
netweir crawl books.py -o books.jsonl

Crawling has the full list of what maps and what doesn’t. If something from your Scrapy project doesn’t come across and you think it should, open an issue with the spider, or the smallest piece of it that shows the problem.