RegLensScraper crawler: operator documentation

This page describes an automated crawler, RegLensScraper, operated by applicable.ai (getapplicable.ai). It is operational documentation for site operators and network operators, not marketing material.

What it does

RegLensScraper reads publicly published government regulatory and rulemaking material (state and federal registers, administrative codes, and agency rulemaking notices) that is already open to the public on the open web. It does not log in, does not access anything behind authentication, and does not submit forms other than the ordinary pagination or filter controls a site itself exposes on its public pages.

The collected text feeds a compliance-preparation product used by compliance officers at community banks and credit unions to identify which rules may apply to their institution and to keep a record of where each rule comes from. The product prepares information for a human reviewer; it does not perform an audit, examination, attestation, or certification of anyone's compliance, and it is not affiliated with any government agency or regulator.

How it identifies itself

When operating in this identified mode, every request carries the following User-Agent header, unchanged from request to request:

RegLensScraper/1.0 (regulatory research; contact: [email protected])

This crawler never presents itself as a web browser and never uses a browser's identity. If a site's robots.txt names RegLensScraper specifically, that rule is read and honored in preference to any general rule.

How it behaves

• It fetches and honors robots.txt before requesting any page on a site, including any stated crawl-delay.
• It rate-limits its own requests per site and does not parallelize aggressively against a single host.
• It re-checks pages it has already read on a schedule, and only re-stores a page's content when that content has actually changed.
• It does not attempt to bypass CAPTCHAs, IP blocks, or bot-mitigation challenges. If a site declines to serve it, it stops and reports that, rather than retrying under a different identity.

Verifying our requests

RegLensScraper is verified by signature, not by IP address. Its outbound addresses are not fixed. Requests can carry an HTTP Message Signature (RFC 9421, the Web Bot Auth scheme) made with an Ed25519 key, with a Signature-Agent header that points to our public key directory:

https://app.getapplicable.ai/.well-known/http-message-signatures-directory

Hosts we read under a site-specific arrangement that departs from their robots.txt are never sent signed requests. If you need to identify our traffic another way, contact us at the address below.

Contact

Questions, takedown requests, or requests to be excluded from crawling: [email protected]. We will honor a specific, direct request not to crawl a site.