photos.kennethreitz.org/templates/robots.txt at main - photos.kennethreitz.org - Gitea: Git with a cup of tea

kennethreitz/photos.kennethreitz.org

mirror of https://github.com/kennethreitz/photos.kennethreitz.org.git synced 2026-06-21 15:10:56 +00:00

Files

T

kennethreitz 961a0d45ea Bump VM memory to 2GB and add crawler rate limits

OOM crash took the site down — increase memory from 1GB to 2GB.
Add Crawl-delay to robots.txt to throttle aggressive bots.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-12 10:41:31 -04:00

24 lines

319 B

Plaintext

Raw Permalink Blame History

 User-agent: *
 Allow: /
 Crawl-delay: 2
 User-agent: GPTBot
 Crawl-delay: 5
 User-agent: Bytespider
 Crawl-delay: 10
 User-agent: AhrefsBot
 Crawl-delay: 10
 User-agent: SemrushBot
 Crawl-delay: 10
 User-agent: DotBot
 Crawl-delay: 10
 User-agent: MJ12bot
 Crawl-delay: 10
 Sitemap: https://photos.kennethreitz.org/sitemap.xml