User-agent: * Disallow: /go/ Disallow: /gt/ Crawl-delay: 600 User-agent: Twiceler Disallow: /tools/info/ Disallow: /tools/keyword-density-analyzer/ User-agent: Yandex Disallow: /tools/info/ Disallow: /tools/keyword-density-analyzer/ User-agent: Speedy Spider Disallow: /tools/info/ Disallow: /tools/keyword-density-analyzer/ User-agent: Accelobot Disallow: /tools/info/ Disallow: /tools/keyword-density-analyzer/ User-agent: MJ12bot Disallow: /tools/info/ Disallow: /tools/keyword-density-analyzer/ User-agent: AhrefsBot Disallow: /tools/info/ Disallow: /tools/keyword-density-analyzer/ User-agent: Baiduspider Disallow: / User-agent: ia_archiver Disallow: / User-agent: AboutUsBot Disallow: / User-agent: panscient.com Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Wget Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: DBLBot Disallow: / User-agent: ScoutJet Disallow: / User-agent: netseer Disallow: / User-agent: bixolabs Disallow: / User-agent: dotbot Disallow: / User-agent: BLEXBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: XoviBot Disallow: / User-agent: BUbiNG Disallow: /