Seeking help for backup strategy
I'm planning to purchase hardware/subscriptions to implement a backup strategy, which is entirely missing in my current setup, which is blocking me from migrating 100% from cloud services. Right now, I'm hosting my apps on a K3s cluster made of 3 Raspberry Pi 5 for the control plane, plus a worker RPi 4. Longhorn is configured with 2 replicas for high availability storage, every app uses its volumes (except for PSQL which handles replication by itself), and I'm quite happy with it. I now want to purchase/build a NAS (raid 5) and some cloud storage to get to 3 copies and 2 locations, but before the final call I made some considerations: Longhorn supports backing up directly to S3, however backups are not encrypted unless the volume itself is encrypted. My volumes aren't encrypted right now, mainly for performance reasons, however I can setup encryption on the most sensitive services (ie. Actual Budget), but cannot afford for Immich as the amount of data is massive and the RPis wouldn't probably keep up with it. Longhorn backups are convenient as the restore of an app volume (or the system itself) is almost a single click operation. At the same time it requires Longhorn to be set up. I would probably prefer to have plain files backed up for photos so if something happens to me (or to Longhorn), the data can still be accessed in a simple way. The cluster infra is entirely declared in YAML files, so setting up an completely new cluster is a bit more complex than running a few kubectl apply commands (I'm oversimplifying ofc). That said, I thought of a mixed solution: I can back up Longhorn volumes to the NAS (maybe putting a Garage instance in front of it), and then setup rsync or restic to push encrypted content from the pods to something like Hetzner BX11 (cheap and simple to access). So here are my doubts: Is having backups in different forms considered a good practice? (Longhorn + plain files) All of this covers HA and DR, what if my hardware is stolen? Can I somehow block access if the NAS? It would be amazing if I could do that and also be able to remotely start/restart it. How many backups should I keep for each volume and for how long? Should I keep multiple backups on both the NAS and the cloud or only one of them? I'm curious to know the details on how everyone solve this kind of problems!