Crawler Documentation
StoreRouterBot
Last updated: October 21, 2026
StoreRouterBot is the public crawler identity used by Octogen Systems, Inc. Our customers are brands, retailers and agentic-shopping applications. StoreRouter's crawler reads ecommerce products. The platform then restructures those products into a common enriched schema to make it easier for LLMs to understand the crawled catalog.
User Agent
Requests signed with StoreRouter's Web Bot Auth identity use this user-agent string:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; StoreRouterBot/1.0; +https://storerouter.ai/botsUnsigned crawler traffic may use ordinary browser user agents. The StoreRouterBot identity is reserved for configured Web Bot Auth traffic.
Bot Details
Verification
StoreRouter uses Cloudflare Web Bot Auth for eligible crawls. Signed requests include HTTP Message Signature headers and a Signature-Agent header that points to StoreRouter's public key directory.
https://bots.storerouter.ai/.well-known/http-message-signatures-directorySite operators should prefer Web Bot Auth verification over static IP allowlists, since crawl egress can change as infrastructure changes.
Compatibility
StoreRouterBot replaces the former OctogenBot name. Older deployments and existing site rules may still refer to OctogenBot/1.0. During the transition, site operators should include both robots tokens and update any allowlist that matches the complete user-agent string. The operator remains Octogen Systems, Inc.
The legacy directory remains directly available with the same public key:
https://bots.octogen.ai/.well-known/http-message-signatures-directoryKey IDs are JWK thumbprints and do not change with the hostname. Verify signatures against the directory named in each request; do not substitute a different Signature-Agent value. Existing old documentation links remain available through the website redirect. The crawler contact is crawler@storerouter.ai.
Crawler Purpose
StoreRouterBot reads publicly available ecommerce pages so StoreRouter can understand how AI shopping agents interpret product catalogs, identify missing product data, and help merchants make their catalogs easier for agents to understand.
- Public product page URLs and page metadata
- Product names, descriptions, images, prices, availability, variants, and attributes
- Public structured data such as Schema.org Product markup
- Public sitemap and robots.txt signals used to scope crawl behavior
Boundaries
- StoreRouterBot does not try to access authenticated, paywalled, or private content.
- StoreRouterBot does not collect personal shopper account data from retailer sites.
- StoreRouterBot does not use signed bot identity unless the crawl is configured for Web Bot Auth.
- StoreRouterBot is intended for commercial product-data workflows, not search-engine indexing.
Robots.txt
To block StoreRouterBot and the legacy OctogenBot identity, add this group to your robots.txt file:
User-agent: StoreRouterBot
User-agent: OctogenBot
Disallow: /To limit crawling to public product pages, use path-specific rules:
User-agent: StoreRouterBot
User-agent: OctogenBot
Disallow: /account/
Disallow: /checkout/
Crawl-delay: 10Contact
For crawl questions, allowlist requests, robots.txt issues, or urgent operational concerns, email crawler@storerouter.ai.