mirror of
https://github.com/kennethreitz/photos.kennethreitz.org.git
synced 2026-06-05 06:46:13 +00:00
Bump VM memory to 2GB and add crawler rate limits
OOM crash took the site down — increase memory from 1GB to 2GB. Add Crawl-delay to robots.txt to throttle aggressive bots. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -41,6 +41,6 @@ primary_region = 'iad'
|
||||
DEBUG = '0'
|
||||
|
||||
[[vm]]
|
||||
memory = '1gb'
|
||||
memory = '2gb'
|
||||
cpu_kind = 'shared'
|
||||
cpus = 1
|
||||
|
||||
@@ -1,4 +1,23 @@
|
||||
User-agent: *
|
||||
Allow: /
|
||||
Crawl-delay: 2
|
||||
|
||||
User-agent: GPTBot
|
||||
Crawl-delay: 5
|
||||
|
||||
User-agent: Bytespider
|
||||
Crawl-delay: 10
|
||||
|
||||
User-agent: AhrefsBot
|
||||
Crawl-delay: 10
|
||||
|
||||
User-agent: SemrushBot
|
||||
Crawl-delay: 10
|
||||
|
||||
User-agent: DotBot
|
||||
Crawl-delay: 10
|
||||
|
||||
User-agent: MJ12bot
|
||||
Crawl-delay: 10
|
||||
|
||||
Sitemap: https://photos.kennethreitz.org/sitemap.xml
|
||||
|
||||
Reference in New Issue
Block a user