sans-self.org / infrastructure
this repo has no description
fork atom

Fix backup script to prevent empty volumes from wiping S3 data

Hetzner volume loss left both PDS and knot running on empty volumes.
The backup cronjob then synced emptiness to S3 via rclone sync, which
deletes destination files not present in source. Two fixes:

- Add pre-flight check: abort if no DB files match BACKUP_DB_GLOB
- Switch rclone sync to rclone copy for directory backups (never deletes)

Both services restored from Feb 24 S3 snapshots.

sans-self.org c38693f3 0b5be594

+15 -2
+2
CHANGELOG.md
··· 16 16 - Add Tangled knot with Spindle CI/CD to k3s cluster (#1) 17 17 18 18 ### Fixed 19 + - Restore PDS and knot data from S3 backups (#34) 20 + - Fix backup script to prevent empty source from wiping S3 data (#33) 19 21 - Add PDS handle resolution for vesper and nyx subdomains (#32) 20 22 - Fix PDS SQLite locking errors causing daily outages (#15) 21 23 - Update PDS to v0.4.208 for OAuth metadata support (#13)
+13 -2
k8s/shared/backup.sh
··· 5 5 S3_OPTS="--s3-provider Other --s3-access-key-id ${S3_ACCESS_KEY} --s3-secret-access-key ${S3_SECRET_KEY} --s3-endpoint nbg1.your-objectstorage.com --s3-region nbg1 --s3-no-check-bucket --s3-acl private" 6 6 TIMESTAMP=$(date +%Y%m%d-%H%M%S) 7 7 8 + # Abort if no database files exist — the volume is likely empty or corrupt. 9 + # Running backups against an empty volume would overwrite good backups with nothing. 10 + db_found=0 11 + for db in ${BACKUP_DB_GLOB}; do 12 + [ -f "$db" ] && db_found=1 13 + done 14 + if [ "$db_found" -eq 0 ]; then 15 + echo "ABORT: no database files match ${BACKUP_DB_GLOB} — volume may be empty" >&2 16 + exit 1 17 + fi 18 + 8 19 # Back up SQLite databases matching BACKUP_DB_GLOB (e.g. "/data/*.sqlite") 9 20 # Uses copy-then-checkpoint to avoid write-locking the live database 10 21 for db in ${BACKUP_DB_GLOB}; do ··· 31 42 # Sync directories listed in BACKUP_SYNC_DIRS (newline-separated "local_path:remote_suffix") 32 43 echo "${BACKUP_SYNC_DIRS}" | while IFS=: read -r src dest; do 33 44 [ -z "$src" ] && continue 34 - rclone sync "$src" \ 45 + rclone copy "$src" \ 35 46 ":s3:${BACKUP_BUCKET}/${BACKUP_PREFIX}/${dest}" \ 36 47 ${S3_OPTS} 37 - echo "synced: ${dest}" 48 + echo "copied: ${dest}" 38 49 done 39 50 40 51 echo "backup complete: ${TIMESTAMP}"