108 lines
6.8 KiB
Markdown
108 lines
6.8 KiB
Markdown
# Migration: photos.kennethreitz.org media → Hetzner Object Storage
|
||
|
||
**Goal:** move the `exiftree-media` photo library (≈166 GiB, ~100k objects) off the
|
||
expensive 1 TB Hetzner Cloud **volume** (MinIO) and onto cheap **Hetzner Object
|
||
Storage** (`kr-photos` bucket), then reclaim ~168 GB of volume space. Serving option
|
||
chosen: **(a) serve straight from the Hetzner bucket URL** (path-style, public-read).
|
||
|
||
*Status started 2026-06-05. This doc is the resume-from-interruption runbook.*
|
||
|
||
## Why
|
||
|
||
The volume costs €76.70/mo (1000 GB @ €0.0767/GB) — more than the server. The photo
|
||
media doesn't need fast block storage; it's HTTP-served static objects (it lived on
|
||
Tigris originally). Object storage is ~€6/mo. ⚠️ The old Tigris fallback is GONE
|
||
(bucket deleted) — so during this migration, the MinIO copy on the volume is the
|
||
**only complete copy** of the photo library until the Storage Box archive finishes.
|
||
**Do not delete the MinIO copy until the Hetzner copy is verified complete.**
|
||
|
||
## Endpoints & creds (also in repo `.env` / `/tmp/exiftree_creds.txt` on the Mac)
|
||
|
||
- **Source — MinIO media** (`minio-q3xgqx-minio-1:9000`, internal): bucket
|
||
`exiftree-media`, key `HU3BA5QMX2AZ67LYSKOQ`, secret in Dokploy/creds file.
|
||
Public today via `https://img.kennethreitz.org` (Traefik → MinIO on the volume).
|
||
- **Target — Hetzner Object Storage**: endpoint `https://hel1.your-objectstorage.com`,
|
||
region `hel1` (MUST set region or S3 ops fail), bucket `kr-photos`,
|
||
key `78BZ6Q8VF2I5IJF3PJEU`, secret `gQq8i4ZxpiGyaw7S4KvsF3utN79E421LUvDmSdWg`.
|
||
Bucket is set **public-read (download)**. Public URL pattern (verified 200):
|
||
`https://hel1.your-objectstorage.com/kr-photos/<key>`.
|
||
|
||
## Steps (checklist)
|
||
|
||
- [ ] **1. Copy** MinIO `exiftree-media` → Hetzner `kr-photos` (IN PROGRESS).
|
||
Runs as detached container `photos-to-hz` on mercury (rclone copy, idempotent —
|
||
safe to restart; re-skips done objects). ~27 GiB/hr (volume disk-throughput
|
||
capped; parallelism does not help), ~6h total.
|
||
- Immich heavy jobs are **PAUSED** during the copy (they read the same MinIO
|
||
objects and contend on volume throughput). Resume after — see step 6.
|
||
- Restart command if it dies: see "Copy command" below.
|
||
- [ ] **2. Verify** copy complete: object count + bytes match between
|
||
`media:exiftree-media` and `hz:kr-photos` (`rclone size` both). Must be equal
|
||
before proceeding.
|
||
- [ ] **3. Cutover photos app** — set these env vars on the photos compose
|
||
(`composeId WICLbVwy5JEbHz2SPb4tR`) and redeploy:
|
||
- `AWS_S3_ENDPOINT_URL=https://hel1.your-objectstorage.com`
|
||
- `AWS_STORAGE_BUCKET_NAME=kr-photos`
|
||
- `AWS_S3_REGION_NAME=hel1`
|
||
- `AWS_ACCESS_KEY_ID=78BZ6Q8VF2I5IJF3PJEU`
|
||
- `AWS_SECRET_ACCESS_KEY=gQq8i4ZxpiGyaw7S4KvsF3utN79E421LUvDmSdWg`
|
||
- `AWS_S3_CUSTOM_DOMAIN=hel1.your-objectstorage.com/kr-photos`
|
||
- `AWS_S3_ADDRESSING_STYLE=path`
|
||
- (keep DEBUG/SECRET_KEY/DATABASE_URL/CELERY_* unchanged)
|
||
- [ ] **4. Verify** photos.kennethreitz.org loads images (homepage img URLs should be
|
||
`https://hel1.your-objectstorage.com/kr-photos/...` and return 200).
|
||
- [ ] **5. Repoint Immich** photo-library mount: the host systemd unit
|
||
`minio-photos-mount.service` mounts `m:exiftree-media/originals` (MinIO). Re-point
|
||
it to `hz:kr-photos/originals` (swap the rclone env in the unit to the Hetzner
|
||
remote), `systemctl daemon-reload && systemctl restart minio-photos-mount`.
|
||
- [ ] **6. Resume Immich** jobs (paused in step 1): via API
|
||
`PUT /api/jobs/{thumbnailGeneration,smartSearch,faceDetection,metadataExtraction,library}`
|
||
with `{"command":"resume"}` (token = login me@kennethreitz.org / house pw).
|
||
- [ ] **7. Decommission MinIO media** ONLY after 3–4 verified: stop/remove the
|
||
`minio` (media) compose `UK8pWczw8d9GSmyLjZJiP`, then
|
||
`rm -rf /mnt/objects/minio` on the host → reclaims ~168 GB on the volume.
|
||
Update the media-minio inventory entry. (Note: `img.kennethreitz.org` and the
|
||
`media-minio.kennethreitz.org` console domains can be removed too.)
|
||
- [ ] **8. Optional next:** with 168 GB freed, the volume is over-sized; a later
|
||
shrink (migrate to a smaller volume) saves more — see the volume-migration notes.
|
||
Also drop the 291 GB iCloud Drive mirror off the volume once its Storage Box
|
||
archive completes.
|
||
|
||
## Copy command (to restart `photos-to-hz` if needed)
|
||
|
||
```sh
|
||
ssh root@mercury.kennethreitz.org "docker rm -f photos-to-hz 2>/dev/null
|
||
docker run -d --name photos-to-hz --network dokploy-network --entrypoint sh rclone/rclone:latest -c '
|
||
export RCLONE_CONFIG_MEDIA_TYPE=s3 RCLONE_CONFIG_MEDIA_PROVIDER=Minio RCLONE_CONFIG_MEDIA_ENDPOINT=http://minio-q3xgqx-minio-1:9000 RCLONE_CONFIG_MEDIA_ACCESS_KEY_ID=HU3BA5QMX2AZ67LYSKOQ RCLONE_CONFIG_MEDIA_SECRET_ACCESS_KEY=<media-secret>
|
||
export RCLONE_CONFIG_HZ_TYPE=s3 RCLONE_CONFIG_HZ_PROVIDER=Other RCLONE_CONFIG_HZ_REGION=hel1 RCLONE_CONFIG_HZ_LOCATION_CONSTRAINT=hel1 RCLONE_CONFIG_HZ_ENDPOINT=https://hel1.your-objectstorage.com RCLONE_CONFIG_HZ_ACCESS_KEY_ID=78BZ6Q8VF2I5IJF3PJEU RCLONE_CONFIG_HZ_SECRET_ACCESS_KEY=<hz-secret> RCLONE_CONFIG_HZ_NO_CHECK_BUCKET=true
|
||
rclone copy media:exiftree-media hz:kr-photos --transfers 32 --checkers 32 --s3-no-check-bucket -v --stats 1m --stats-one-line 2>&1
|
||
echo MIRROR_DONE_EXIT:\$?'"
|
||
```
|
||
|
||
## Rollback (if cutover breaks the site)
|
||
|
||
The MinIO copy is untouched until step 7. To roll back: revert the step-3 env vars to
|
||
the MinIO values (`AWS_S3_ENDPOINT_URL=http://minio-q3xgqx-minio-1:9000`,
|
||
`AWS_STORAGE_BUCKET_NAME=exiftree-media`, `AWS_S3_CUSTOM_DOMAIN=img.kennethreitz.org/exiftree-media`,
|
||
region `us-east-1`, MinIO key/secret) and redeploy. Photos serve from MinIO again.
|
||
|
||
|
||
## COMPLETED 2026-06-06
|
||
|
||
Steps 1-7 done: copy verified (100,632 objects identical), photos app cut over to
|
||
`hel1.your-objectstorage.com/kr-photos` (verified serving), Immich photo-library
|
||
mount repointed to `hz:kr-photos/originals`, storagebox-archive source repointed to
|
||
`hz:kr-photos`, MinIO media decommissioned, **167 GB reclaimed** (volume 473G→306G used).
|
||
Mid-migration the photos-backup vfs-cache filled root and broke Swarm — see the
|
||
incident note in server.md; recovered, and photos-backup cache now lives on the volume.
|
||
|
||
### Still pending (the actual €76.70 volume cost cut — step 8)
|
||
The volume is still 1 TB. To shrink it (option A: small volume for Immich), remaining:
|
||
- iCloud Drive mirror (291 G) must move off — needs its Storage Box archive COMPLETE
|
||
first (storagebox-archive reaches icloud-drive after kr-photos in its loop), then
|
||
repoint Filebrowser (drive.kennethreitz.org) to a Storage Box mount.
|
||
- photos-backup needs **Apple re-auth** (cookie corrupted in the incident):
|
||
`ssh -t root@mercury "docker run -it --rm -v /var/lib/icloudpd:/data icloudpd/icloudpd icloudpd --username me@kennethreitz.org --auth-only --cookie-directory /data/cookies"`
|
||
- Volume delete/recreate hits the 1 TB account quota — needs a Hetzner quota increase
|
||
for a SAFE side-by-side migrate (vs risky delete-first). Recommended given the incident.
|