User-Agent: PetalBot Disallow: / User-Agent: SemrushBot Disallow: / User-Agent: MJ12bot Disallow: / # Stop spamming my accesslog. User-Agent: W3C-checklink Disallow: /archive/ Disallow: /sshin/ # Don't waste. (Going up) Disallow: /read/?path=/&referer=/small-scripts/&title=Small+scripts # Don't waste. Disallow: /read/?path=/oops/ Disallow: /read/?path=/projects/ Disallow: /read/?path=/pages/ Disallow: /read/?path=/read/ Disallow: /read/?path=/tridecane/ Disallow: /read/?path=/includes/ User-Agent: RED Crawl-Delay: 30 Disallow: User-Agent: Googlebot Disallow: /archive/*/*.*z User-Agent: FemtosearchBot Disallow: /archive/ User-Agent: * Crawl-Delay: 3 Disallow: /archive/ Sitemap: https://oskog97.com/sitemap.xml