We offer a few methods for retrieving live web pages and data documents. Test live web pages and documents by using the "URL" feature on the playground or local retrieval from the CLI.
Browser Rendering
When used in a browser context, this agent may fetch additional resources such as scripts and other dynamic content to accurately render a web page for analysis. While rendering, we continue to respect robots.txt directives and may utilize cached resources to reduce unnecessary network requests.
Fetchbot
Our Schemamarkup-Fetchbot agent represents user-initiated retrieval of remote content from traditional websites. We respect the Robots Exclusion Protocol specification (i.e. robots.txt files) and will not retrieve content from sites that disallow automated access or target our Schemamarkup-Fetchbot agent.
The User-Agent header of all our web requests will look similar to one of the following lines (and may include additional client- or request-specific metadata).
Schemamarkup-Fetchbot/1.1 (+https://schemamarkup.app/fetch#v1)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36 Schemamarkup-Fetchbot/1.1 (+https://schemamarkup.app/fetch#v1)All client requests originate from IP addresses which can be verified through DNS. A reverse DNS lookup will return a subdomain of client.schemamarkup.app, and the host will have a reciprocal A record. These IP addresses may change over time, and a successful DNS verification may be cached for up to 28 days.
> host 203.0.113.47
47.113.0.203.in-addr.arpa domain name pointer ip-203-0-113-47.client.schemamarkup.app.
> host ip-203-0-113-47.client.schemamarkup.app.
ip-203-0-113-47.client.schemamarkup.app address 203.0.113.47