# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file # # To ban all spiders from the entire site uncomment the next two lines: Sitemap: https://akhbarak.net/sitemaps/static.xml Sitemap: https://akhbarak.net/sitemaps/tags.xml Sitemap: https://akhbarak.net/sitemaps/sections.xml Sitemap: https://akhbarak.net/sitemaps/sources.xml Sitemap: https://akhbarak.net/videos.xml Sitemap: https://akhbarak.net/galleries.xml User-agent: * Crawl-delay: 5 Disallow: /admin/ Disallow: /articles/ Disallow: /topics/limited/ Disallow: /tags/limited/ Disallow: /clusters/limited/ Disallow: /*/sort/* Disallow: /*/by_date Disallow: /*/by_score Disallow: /*/by_date/ Disallow: /*/by_score/ #Disallow: /*.js$ Disallow: /blog/tag/* Disallow: /blog/author/* #User-Agent: * #Disallow: /