mirror of
https://github.com/kennethreitz/photos.kennethreitz.org.git
synced 2026-06-21 15:10:56 +00:00
kennethreitz
961a0d45ea
OOM crash took the site down — increase memory from 1GB to 2GB. Add Crawl-delay to robots.txt to throttle aggressive bots. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
24 lines
319 B
Plaintext
24 lines
319 B
Plaintext
User-agent: *
|
|
Allow: /
|
|
Crawl-delay: 2
|
|
|
|
User-agent: GPTBot
|
|
Crawl-delay: 5
|
|
|
|
User-agent: Bytespider
|
|
Crawl-delay: 10
|
|
|
|
User-agent: AhrefsBot
|
|
Crawl-delay: 10
|
|
|
|
User-agent: SemrushBot
|
|
Crawl-delay: 10
|
|
|
|
User-agent: DotBot
|
|
Crawl-delay: 10
|
|
|
|
User-agent: MJ12bot
|
|
Crawl-delay: 10
|
|
|
|
Sitemap: https://photos.kennethreitz.org/sitemap.xml
|