Crawlable, readable, traceable
Separate access rules, writing structure, and source records while keeping evidence appropriate to each claim.
Google Robots.txt Parser and Matcher LibraryRead article
Sources
Open original source ↗ · Official documentation
Separate access rules, writing structure, and source records while keeping evidence appropriate to each claim.
The complete project README describing parsing, building, and matching limits.