source code of /robots.txt

Last modified
Lines 39

Parent directory Download CGIread sitemap Main page

Quick links: (none)

  1. User-Agent: PetalBot
  2. Disallow: /
  3. User-Agent: SemrushBot
  4. Disallow: /
  5. User-Agent: MJ12bot
  6. Disallow: /
  7. # Stop spamming my accesslog.
  8. User-Agent: W3C-checklink
  9. Disallow: /archive/
  10. Disallow: /sshin/
  11. # Don't waste. (Going up)
  12. Disallow: /read/?path=/&referer=/small-scripts/&title=Small+scripts
  13. # Don't waste.
  14. Disallow: /read/?path=/oops/
  15. Disallow: /read/?path=/projects/
  16. Disallow: /read/?path=/pages/
  17. Disallow: /read/?path=/read/
  18. Disallow: /read/?path=/tridecane/
  19. Disallow: /read/?path=/includes/
  20. User-Agent: RED
  21. Crawl-Delay: 30
  22. Disallow:
  23. User-Agent: Googlebot
  24. Disallow: /archive/*/*.*z
  25. User-Agent: FemtosearchBot
  26. Disallow: /archive/
  27. User-Agent: *
  28. Crawl-Delay: 3
  29. Disallow: /archive/
  30. Sitemap: https://oskog97.com/sitemap.xml