onyx_robots_check
InactiveTool of onyx-paid-mcp
Fetch a domain's robots.txt and report whether a given path is allowed for a given user-agent. Returns the raw robots.txt text, the matched rule, the crawl-delay if specified, and a clean allow/disallow verdict. Use when an agent does web scraping and wants to be polite — saves bans, saves CAPTCHAs, saves drama. ~50-200ms. (price: $0.001 USDC, tier: metered)
Parameters schema
{
"type": "object",
"required": [
"url"
],
"properties": {
"url": {
"type": "string",
"description": "Any URL on the target domain"
},
"user_agent": {
"type": "string",
"default": "*",
"description": "UA string to test"
}
}
}Parent server
onyx-paid-mcp
https://github.com/dimitrilaouanis-tech/onyx-mcp
2/7 registries