now also available for firefox

robots.txt details

06fa5ea4cbc42ed0c2c6272578a9f75f

saved: | size: 1047kb | checksum: 06fa5ea4cbc42ed0c2c6272578a9f75f

websites using this robots.txt

content

User-agent: *
Crawl-delay: 10

Disallow: /articles/print/
Disallow: /articles/email/
Disallow: /topclicks/
Disallow: /stats/
Disallow: /staff/
Disallow: /cgi/
Disallow: /temp/
Disallow: /andy/
Disallow: /media/
Disallow: /myOMC/user/
Disallow: /myOMC/talkback/
Disallow: /events/email.html
Disallow: /events/print.html
Disallow: /dps/index.html?a=getinfo
Disallow: /dps/desktop.html
Disallow: /myOMC/subscription/
Disallow: /business/print.html
Disallow: /business/email.html
Disallow: /search/
Disallow: /feedback/
Disallow: /events/submit.html
Disallow: /myOMC/blog_profile/
Disallow: /myOMC/blog/new/
Disallow: /myOMC/blog/edit/
Disallow: /myOMC/blog_avatar/
Disallow: /profile/
Disallow: /bots/

User-agent: Missigua Locator 1.9
Disallow: /

User-agent: aipbot
Disallow: /

User-agent: Snoopy
Disallow: /

User-agent: Baiduspider
User-agent: Baiduspider-video
User-agent: Baiduspider-image
Disallow: /

User-agent: 008
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: sogou spider
Disallow: /

User-agent: Yandex
Disallow: /