# netweir

> An ultra-fast, self-healing, undetectable web scraper for Python, with a Rust core. Open source under the AGPL-3.0; a commercial licence is available.

netweir 0.1.0 fetches pages with requests that match Chrome 154, Firefox 156 or Safari 27 on the wire, parses them with lexbor and runs CSS, XPath 1.0 and Beautiful Soup style queries in Rust. It crawls with spiders (callbacks or declarative rules), recognises and recovers from bot-protection blocks, resumes crawls after a crash from a checkpoint, finds tracked selectors again after a redesign, and drives Chrome for pages that need JavaScript.

## Documentation

- [Getting started](https://docs.netweir.com/getting-started/): install, fetch a page, write and run a spider.
- [Selecting](https://docs.netweir.com/selecting/): CSS, XPath and Beautiful Soup style queries.
- [Crawling](https://docs.netweir.com/crawling/): spiders, requests, pipelines, exporters and the command line.
- [Declarative spiders](https://docs.netweir.com/declarative/): items and rules that run in Rust.
- [Self-healing](https://docs.netweir.com/self-healing/): blocks, checkpoints, tracked selectors and repair proposals.
- [Browser](https://docs.netweir.com/browser/): drive Chrome for pages that need JavaScript.
- [Settings](https://docs.netweir.com/settings/): every setting and its default.
- [All docs as one Markdown file](https://docs.netweir.com/llms-full.txt)

## This site

- [Home](https://netweir.com/): what netweir is, with code and measured numbers.
- [Benchmarks](https://netweir.com/benchmarks/): parse, query and crawl times against other libraries, with the commands to rerun them.
- [Changelog](https://netweir.com/changelog/): every release and its notes.
- [Licence](https://netweir.com/licence/): AGPL-3.0, and how to ask about a commercial licence.

## Blog

- [netweir 0.1.0](https://netweir.com/blog/netweir-0-1-0/): The first release of netweir: Chrome-identical fetching, a Rust parser with Scrapy, XPath and Beautiful Soup queries, crawling that heals itself, and a Chrome driver. What it does today, and what comes next.

## Elsewhere

- [Source on GitHub](https://github.com/netweir/netweir)
- [Package on PyPI](https://pypi.org/project/netweir/)