source code of /robots.txt

Last modified
Lines 39

Parent directory Download CGIread sitemap Main page

Quick links: (none) 

  1. User-Agent: PetalBot
  2. Disallow: /
    4. 
    
  4. User-Agent: SemrushBot
    5. 
    
  5. Disallow: /
    6. 
    
    7. 
    
  7. User-Agent: MJ12bot
    8. 
    
  8. Disallow: /
    9. 
    
  9. # Stop spamming my accesslog.
    10. 
    
    11. 
    
  11. User-Agent: W3C-checklink
    12. 
    
  12. Disallow: /archive/
    13. 
    
  13. Disallow: /sshin/
    14. 
    
  14. # Don't waste. (Going up)
    15. 
    
  15. Disallow: /read/?path=/&referer=/small-scripts/&title=Small+scripts
    16. 
    
  16. # Don't waste.
    17. 
    
  17. Disallow: /read/?path=/oops/
    18. 
    
  18. Disallow: /read/?path=/projects/
    19. 
    
  19. Disallow: /read/?path=/pages/
    20. 
    
  20. Disallow: /read/?path=/read/
    21. 
    
  21. Disallow: /read/?path=/tridecane/
    22. 
    
  22. Disallow: /read/?path=/includes/
    23. 
    
    24. 
    
  24. User-Agent: RED
    25. 
    
  25. Crawl-Delay: 30
    26. 
    
  26. Disallow:
    27. 
    
    28. 
    
  28. User-Agent: Googlebot
    29. 
    
  29. Disallow: /archive/*/*.*z
    30. 
    
    31. 
    
  31. User-Agent: FemtosearchBot
    32. 
    
  32. Disallow: /archive/
    33. 
    
    34. 
    
  34. User-Agent: *
    35. 
    
  35. Crawl-Delay: 3
    36. 
    
  36. Disallow: /archive/
    37. 
    
    38. 
    
  38. Sitemap: https://oskog97.com/sitemap.xml
    39. 
    
    40.