Bump VM memory to 2GB and add crawler rate limits

OOM crash took the site down — increase memory from 1GB to 2GB.
Add Crawl-delay to robots.txt to throttle aggressive bots.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-04-12 10:41:31 -04:00
parent 48cc7bf8b0
commit 961a0d45ea
2 changed files with 20 additions and 1 deletions
+1 -1
View File
@@ -41,6 +41,6 @@ primary_region = 'iad'
DEBUG = '0'
[[vm]]
memory = '1gb'
memory = '2gb'
cpu_kind = 'shared'
cpus = 1
+19
View File
@@ -1,4 +1,23 @@
User-agent: *
Allow: /
Crawl-delay: 2
User-agent: GPTBot
Crawl-delay: 5
User-agent: Bytespider
Crawl-delay: 10
User-agent: AhrefsBot
Crawl-delay: 10
User-agent: SemrushBot
Crawl-delay: 10
User-agent: DotBot
Crawl-delay: 10
User-agent: MJ12bot
Crawl-delay: 10
Sitemap: https://photos.kennethreitz.org/sitemap.xml