About our crawler
UsefulWorkBot
If you saw UsefulWorkBot/0.1 in your logs, someone asked Useful Work to check that website for mistakes and broken links.
What it does
- It only visits websites someone has asked Useful Work to check, and only their public pages.
- However many people list a website, it reads it at most once every ten minutes.
- It reads pages with GET requests, at most one a second, and up to 100 pages per site.
- It never submits forms, logs in, buys anything or probes for security issues.
- It follows
robots.txtrules for*and forUsefulWorkBot.
Make it stop
Add these two lines to your robots.txt:
User-agent: UsefulWorkBot Disallow: /
Anyone can ask us to check a public website, the same way anyone can visit it. If you'd rather we didn't, the two lines above stop us: our next visit sees them and reads nothing more.