InsertoraBot

How to identify it

InsertoraBot sends this user agent with every request:

InsertoraBot/1.0 (+https://insertora.com/bot)

What it does

• Reads robots.txt, XML sitemaps, RSS/Atom feeds and public HTML pages on the website a user asked to scan.
• Extracts article text (titles, headings and paragraphs) to measure relevance to a keyword.
• Stores a cached copy of article text for a limited time so repeat scans don’t re-crawl your site.

What it never does

• It never logs in, submits forms, or accesses account, cart or checkout pages.
• It never attempts to bypass CAPTCHAs, paywalls, rate limits or other access controls.
• It never modifies your website or inserts links.

Politeness

• Obeys robots.txt rules for InsertoraBot (or * if no specific group exists).
• Honours Crawl-delay (up to 10 seconds) and spaces requests by at least 750 ms.
• Crawls one page at a time per website and caps pages per scan.
• Backs off on HTTP 429 and server errors, and pauses crawling a site that keeps returning errors.

How to limit or block InsertoraBot

To block it entirely, add this to your robots.txt:

User-agent: InsertoraBot
Disallow: /

To slow it down, add a crawl delay:

User-agent: InsertoraBot
Crawl-delay: 5

If you would like your domain excluded from scanning altogether, contact us and we will add it to our blocked list.