The crawler
Our crawler, and how to turn it off.
If SeoBoostyBot turned up in your logs, this page says who runs it and how to stop it. Everything printed here describes the crawler as it runs today.
Identity
How to recognize it.
Two things identify the crawler: a user agent string that never changes, and a published list of the addresses it comes from. Anything calling itself SeoBoostyBot from an address outside that list did not come from us.
User agent
SeoBoostyBot/1.0 (+https://seoboosty.com/bot)Where it comes from
The addresses are published as a file rather than printed on this page, so anything on your side that keeps an allow list reads the current set instead of a copy that has gone stale.
The file is the source of truth. If an address ever changes, the file changes with it and nothing on your side needs editing by hand.
What it reads
It reads pages and writes down links.
SeoBoosty looks for expired domains that established sites still link to. To find them, the crawler reads publicly available pages and records which page links to which domain. That is the entire job.
What it does
- Requests pages that are already public, the same way a browser does
- Reads the links on the page, with their anchor text and rel attribute
- Keeps the link, the anchor text and the address of the page it sat on
What it never does
- Submit a form
- Attempt a login, with any credentials
- Buy anything or place an order
- Store your page text, your images or your files
What we keep is links and anchor text. The crawler reads your page once to find them, then drops the text.
How it behaves
What it does when your server pushes back.
Every line below describes the crawler running in production today, and the numbers come out of its own logs.
01It reads robots.txt first
The crawler fetches your robots.txt and applies the Disallow rules before it requests a page. In one recent ten minute window it skipped 3,554 URLs on robots.txt grounds alone.
02429 slows it down
The crawler takes an HTTP 429 at face value and eases off that host, rather than retrying at the same rate.
03A challenge cuts its rate to a quarter
When a response looks like a bot challenge or a WAF block, the crawler slows itself four times over on that host.
04It walks away from a failing host
A circuit breaker watches errors per host. Once it trips, the crawler abandons that host instead of hammering it.
05Seconds between requests
The crawler leaves seconds between requests to the same domain. It has no burst mode and opens no flood of parallel connections against one site.
How to control it
Lines you can paste.
Whatever you put in robots.txt for SeoBoostyBot takes effect on the crawler's next visit, and it re-reads robots.txt regularly. You do not have to write to us first.
Block it entirely
Once the crawler reads this, it requests nothing on the site.
User-agent: SeoBoostyBot
Disallow: /Slow it down
Ten seconds between requests to your site. Set whatever number suits your server.
User-agent: SeoBoostyBot
Crawl-delay: 10Block one part of the site
Ordinary path rules. Everything you leave out stays open.
User-agent: SeoBoostyBot
Disallow: /admin/
Disallow: /searchBlock it at the firewall
For anyone who would rather not wait for the next robots.txt read. Pull the current addresses and feed them straight to your firewall or WAF.
curl -s https://seoboosty.com/bot/ips.json | jq -r '.ipv4[]'Write the token SeoBoostyBot in the User-agent line, with no version and no URL after it.
Contact
Anything else, write to us.
One mailbox takes everything about the crawler and a person reads it. Ask us to stop crawling your site and we stop, and you do not have to explain why.
Open the contact formCrawler questions
[email protected]
Send the domain and a few log lines if you have them. That is usually enough to confirm the traffic was ours and to say what it was doing.