kennethreitz/mercury.kennethreitz.org
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
mercury.kennethreitz.org/docs/photos-to-objectstorage-migration.md
T
kennethreitz f613f79230 Photos migration complete (167G reclaimed); photos-backup cache on volume 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 13:00:34 -04:00

6.8 KiB
Raw Permalink Blame History

Migration: photos.kennethreitz.org media → Hetzner Object Storage

Goal: move the exiftree-media photo library (≈166 GiB, ~100k objects) off the expensive 1 TB Hetzner Cloud volume (MinIO) and onto cheap Hetzner Object Storage (kr-photos bucket), then reclaim ~168 GB of volume space. Serving option chosen: (a) serve straight from the Hetzner bucket URL (path-style, public-read).

Status started 2026-06-05. This doc is the resume-from-interruption runbook.

Why

The volume costs €76.70/mo (1000 GB @ €0.0767/GB) — more than the server. The photo media doesn't need fast block storage; it's HTTP-served static objects (it lived on Tigris originally). Object storage is ~€6/mo. ⚠️ The old Tigris fallback is GONE (bucket deleted) — so during this migration, the MinIO copy on the volume is the only complete copy of the photo library until the Storage Box archive finishes. Do not delete the MinIO copy until the Hetzner copy is verified complete.

Endpoints & creds (also in repo .env / /tmp/exiftree_creds.txt on the Mac)

  • Source — MinIO media (minio-q3xgqx-minio-1:9000, internal): bucket exiftree-media, key HU3BA5QMX2AZ67LYSKOQ, secret in Dokploy/creds file. Public today via https://img.kennethreitz.org (Traefik → MinIO on the volume).
  • Target — Hetzner Object Storage: endpoint https://hel1.your-objectstorage.com, region hel1 (MUST set region or S3 ops fail), bucket kr-photos, key 78BZ6Q8VF2I5IJF3PJEU, secret gQq8i4ZxpiGyaw7S4KvsF3utN79E421LUvDmSdWg. Bucket is set public-read (download). Public URL pattern (verified 200): https://hel1.your-objectstorage.com/kr-photos/<key>.

Steps (checklist)

  • 1. Copy MinIO exiftree-media → Hetzner kr-photos (IN PROGRESS). Runs as detached container photos-to-hz on mercury (rclone copy, idempotent — safe to restart; re-skips done objects). ~27 GiB/hr (volume disk-throughput capped; parallelism does not help), ~6h total.
    • Immich heavy jobs are PAUSED during the copy (they read the same MinIO objects and contend on volume throughput). Resume after — see step 6.
    • Restart command if it dies: see "Copy command" below.
  • 2. Verify copy complete: object count + bytes match between media:exiftree-media and hz:kr-photos (rclone size both). Must be equal before proceeding.
  • 3. Cutover photos app — set these env vars on the photos compose (composeId WICLbVwy5JEbHz2SPb4tR) and redeploy:
    • AWS_S3_ENDPOINT_URL=https://hel1.your-objectstorage.com
    • AWS_STORAGE_BUCKET_NAME=kr-photos
    • AWS_S3_REGION_NAME=hel1
    • AWS_ACCESS_KEY_ID=78BZ6Q8VF2I5IJF3PJEU
    • AWS_SECRET_ACCESS_KEY=gQq8i4ZxpiGyaw7S4KvsF3utN79E421LUvDmSdWg
    • AWS_S3_CUSTOM_DOMAIN=hel1.your-objectstorage.com/kr-photos
    • AWS_S3_ADDRESSING_STYLE=path
    • (keep DEBUG/SECRET_KEY/DATABASE_URL/CELERY_* unchanged)
  • 4. Verify photos.kennethreitz.org loads images (homepage img URLs should be https://hel1.your-objectstorage.com/kr-photos/... and return 200).
  • 5. Repoint Immich photo-library mount: the host systemd unit minio-photos-mount.service mounts m:exiftree-media/originals (MinIO). Re-point it to hz:kr-photos/originals (swap the rclone env in the unit to the Hetzner remote), systemctl daemon-reload && systemctl restart minio-photos-mount.
  • 6. Resume Immich jobs (paused in step 1): via API PUT /api/jobs/{thumbnailGeneration,smartSearch,faceDetection,metadataExtraction,library} with {"command":"resume"} (token = login me@kennethreitz.org / house pw).
  • 7. Decommission MinIO media ONLY after 3–4 verified: stop/remove the minio (media) compose UK8pWczw8d9GSmyLjZJiP, then rm -rf /mnt/objects/minio on the host → reclaims ~168 GB on the volume. Update the media-minio inventory entry. (Note: img.kennethreitz.org and the media-minio.kennethreitz.org console domains can be removed too.)
  • 8. Optional next: with 168 GB freed, the volume is over-sized; a later shrink (migrate to a smaller volume) saves more — see the volume-migration notes. Also drop the 291 GB iCloud Drive mirror off the volume once its Storage Box archive completes.

Copy command (to restart photos-to-hz if needed)

ssh root@mercury.kennethreitz.org "docker rm -f photos-to-hz 2>/dev/null
docker run -d --name photos-to-hz --network dokploy-network --entrypoint sh rclone/rclone:latest -c '
export RCLONE_CONFIG_MEDIA_TYPE=s3 RCLONE_CONFIG_MEDIA_PROVIDER=Minio RCLONE_CONFIG_MEDIA_ENDPOINT=http://minio-q3xgqx-minio-1:9000 RCLONE_CONFIG_MEDIA_ACCESS_KEY_ID=HU3BA5QMX2AZ67LYSKOQ RCLONE_CONFIG_MEDIA_SECRET_ACCESS_KEY=<media-secret>
export RCLONE_CONFIG_HZ_TYPE=s3 RCLONE_CONFIG_HZ_PROVIDER=Other RCLONE_CONFIG_HZ_REGION=hel1 RCLONE_CONFIG_HZ_LOCATION_CONSTRAINT=hel1 RCLONE_CONFIG_HZ_ENDPOINT=https://hel1.your-objectstorage.com RCLONE_CONFIG_HZ_ACCESS_KEY_ID=78BZ6Q8VF2I5IJF3PJEU RCLONE_CONFIG_HZ_SECRET_ACCESS_KEY=<hz-secret> RCLONE_CONFIG_HZ_NO_CHECK_BUCKET=true
rclone copy media:exiftree-media hz:kr-photos --transfers 32 --checkers 32 --s3-no-check-bucket -v --stats 1m --stats-one-line 2>&1
echo MIRROR_DONE_EXIT:\$?'"

Rollback (if cutover breaks the site)

The MinIO copy is untouched until step 7. To roll back: revert the step-3 env vars to the MinIO values (AWS_S3_ENDPOINT_URL=http://minio-q3xgqx-minio-1:9000, AWS_STORAGE_BUCKET_NAME=exiftree-media, AWS_S3_CUSTOM_DOMAIN=img.kennethreitz.org/exiftree-media, region us-east-1, MinIO key/secret) and redeploy. Photos serve from MinIO again.

COMPLETED 2026-06-06

Steps 1-7 done: copy verified (100,632 objects identical), photos app cut over to hel1.your-objectstorage.com/kr-photos (verified serving), Immich photo-library mount repointed to hz:kr-photos/originals, storagebox-archive source repointed to hz:kr-photos, MinIO media decommissioned, 167 GB reclaimed (volume 473G→306G used). Mid-migration the photos-backup vfs-cache filled root and broke Swarm — see the incident note in server.md; recovered, and photos-backup cache now lives on the volume.

Still pending (the actual €76.70 volume cost cut — step 8)

The volume is still 1 TB. To shrink it (option A: small volume for Immich), remaining:

  • iCloud Drive mirror (291 G) must move off — needs its Storage Box archive COMPLETE first (storagebox-archive reaches icloud-drive after kr-photos in its loop), then repoint Filebrowser (drive.kennethreitz.org) to a Storage Box mount.
  • photos-backup needs Apple re-auth (cookie corrupted in the incident): ssh -t root@mercury "docker run -it --rm -v /var/lib/icloudpd:/data icloudpd/icloudpd icloudpd --username me@kennethreitz.org --auth-only --cookie-directory /data/cookies"
  • Volume delete/recreate hits the 1 TB account quota — needs a Hetzner quota increase for a SAFE side-by-side migrate (vs risky delete-first). Recommended given the incident.
Reference in New Issue View Git Blame Copy Permalink