Netweir reads HTML with its own parser now
Netweir used to parse with lexbor, the same C library selectolax uses. Now it parses with a parser written for it from the HTML Standard, and pages parse in about half the time. Here's how we checked it gives the same answers.
Netweir has its own HTML parser.
Until 0.5.0, every page went through lexbor, a good C parser and the same one selectolax uses. That was a fine place to start. It also put a ceiling on us. Parsing was about 80% of the time it took to read a page and run a query on it, and since selectolax parses with the same library, our lead over it came down to a few percent. Some machines swapped the order.
So I wrote a parser for Netweir, in Rust, from the WHATWG HTML Standard. It’s in the repository as crates/netweir-html, under the same licence as the rest of Netweir. You don’t have to change a line to use it: page.css(...), page.xpath(...) and everything else work as they did.
What half the time looks like
Pulling every price, title and link out of a generated shop page:
| 1 MB page | 10 MB page | |
|---|---|---|
| Netweir 0.4 (lexbor) | 5.0 ms | 61 ms |
| Netweir 0.5 | 3.1 ms | 32 ms |
| selectolax | 5.5 ms | 66 ms |
XPath moved too. Six queries on the 1 MB page took 9.9 ms and now take 7.7.
Synthetic pages flatter whoever wrote the generator, so the parser was also timed on 50 real ones: news front pages, Wikipedia articles, docs, shops, government sites and forums. In total it parses them 2.16 times as fast as lexbor at its highest optimisation level. Every single page got faster, the smallest gain being 1.27 times and the largest 6.43.
Where does the time go? Mostly into not copying. The document keeps one copy of the page, and text and attribute values in the tree point into it instead of being copied out. Tags people actually write take a short path through the tokenizer, and a test runs 20,000 inputs through both paths to make sure they agree.
Fast is easy. The same answers are harder
A parser that’s quick and wrong is worse than no parser, because the wrong answer comes back quietly. So it had to pass everything before it went anywhere near your spiders.
The HTML Standard has two public test suites, and it passes both. That’s all 7,045 of the html5lib tokenizer tests and all 3,898 tree-construction tests that don’t need JavaScript to run. Six tests need a script engine, and those are skipped.
Then it had to agree with lexbor on real pages. On all 50, it builds the same tree, and html() writes out the same bytes. 32 selectors were run against each page, and 1,595 of those 1,600 answers match. The five that don’t are on purpose, and the next section says why.
It’s been fuzzed for 7.3 million runs without a crash. Pages built to be slow, such as 200,000 unclosed <div>s or formatting tags misnested on purpose, parse in time that grows with their size, not with its square. A page bigger than 512 MB is refused with an error rather than parsed.
Where we picked the browser
Lexbor’s selector engine disagreed with browsers in a few places. Netweir now does what Chrome and Firefox do, because a selector you tried in DevTools should work the same in your spider:
#idand.classcompare exactly, unless the page is in quirks mode.- Attribute names ignore case on HTML elements only. On SVG,
[viewBox]matches and[viewbox]doesn’t, and the same goes for tag names such asclipPath. [href]means anhrefin no namespace, so it doesn’t match SVG’sxlink:href.[*|href]matches both. Those are the five answers that changed.- HTML’s case-insensitive attribute values, such as
typeandlang, compare without case. :is()and:where()skip a part they can’t parse, while:not()and:has()don’t.:scopeworks, and:hoverand the like never match.:lexbor-containsis gone with lexbor.
One thing stays as lexbor had it. node.attrs and XPath see SVG attributes in lowercase (viewbox) and a namespaced one by its local name, so //use/@href still finds xlink:href. That’s what lxml does too, and plenty of spiders depend on it.
What else changes
The build got simpler. There’s no lexbor submodule, no C shim and no FFI layer. A C compiler is still needed for BoringSSL, which isn’t going anywhere.
netweir version -v reports the parser instead of a lexbor version.
Try it
pip install -U netweirThen run the benchmarks on your own machine: python bench/parse.py and python bench/query.py in the repository. If a page parses differently from how your browser reads it, that’s a bug, and I’d like to see it. Open an issue with the URL or the HTML.