SocktBot

SocktBot is the crawler behind Sockt, part of Raq.com, run by Vu AI Ltd. Sockt records which public web technologies websites use, such as their content management system, analytics and online shop software, and public DNS facts such as which email provider a domain uses.

Most of Sockt's data comes from public web archives. SocktBot visits a site itself only when the archive has no usable copy or a customer is watching that domain for changes.

The robots.txt token is SocktBot.

User agent: Mozilla/5.0 (compatible; SocktBot/0.1; +https://sockt.com/bot)
Address: All requests come from one fixed IP address, 32.193.179.45, whose reverse DNS is socktbot.sockt.com.
robots.txt: Followed, including Disallow and Crawl-delay up to 60 seconds. SocktBot keeps a copy for up to 24 hours, so a change takes effect within a day.
What it fetches: robots.txt, the homepage and at most one other page of the same site: at most 3 requests per visit, one at a time and at least 5 seconds apart.
When refused: A 401 or 403 response or a bot challenge stops visits to that site for 30 days. SocktBot slows down on 429 and 503 and respects Retry-After.
Opting out: Block SocktBot in robots.txt, or remove your site from Sockt.

# robots.txt
User-agent: SocktBot
Disallow: /

What it fetches on a visit

robots.txt, then the homepage and at most one other page of the same site, usually About or Contact.
One request at a time per site, at least 5 seconds apart, and at most 3 requests per visit.
Public DNS records for the domain (MX, NS, TXT, the _dmarc record and the www CNAME), through its own resolver.

How it follows robots.txt

Rules for SocktBot apply to it. When no group names it, the rules for * apply. Rules for other crawlers do not.
Crawl-delay is respected up to 60 seconds. If you ask for a longer delay, SocktBot does not visit your site.
SocktBot keeps a copy of robots.txt for up to 24 hours and checks it before every request, so a change takes effect within a day.
If robots.txt cannot be read because of a server error, SocktBot does not visit.

Blocking or slowing SocktBot

To keep SocktBot off your whole site, add this to robots.txt:

User-agent: SocktBot
Disallow: /

To keep it out of part of a site and space out its visits:

User-agent: SocktBot
Disallow: /account/
Crawl-delay: 10

When a site refuses access (HTTP 401 or 403, or a bot challenge page), SocktBot stops visiting it for 30 days. It slows down on 429 and 503 responses and respects Retry-After.

What SocktBot never does

SocktBot never signs in, submits forms, solves CAPTCHAs, changes its user agent, rotates IP addresses, scans ports or fetches from private networks.

Removing your site

Blocking SocktBot stops new visits. To remove what Sockt already holds about your domain, use sockt.com/opt-out. You prove you control the domain with a sockt-verify= DNS TXT record or a link sent to a standard admin address at the domain. Removal is free.

Problems

If SocktBot causes a problem on your site, or something calling itself SocktBot ignores your robots.txt, email [email protected] with the time, your domain and the request lines from your logs.