Backing up OpenWrt hosts and their VMs to Proxmox Backup Server
Some of my OpenWrt boxes are x86 machines that also run a VM or two under plain QEMU. Everything else in the homelab is backed up to Proxmox Backup Server, so these should be too. Two packages in my OpenWrt feed do that.
The client
PBS backups need proxmox-backup-client, which is Rust and not in OpenWrt.
The feed’s proxmox-backup
package builds just the client from the upstream sources, with a few patches:
a client-only cargo build, no libsystemd, and the system zstd. So far I have
only tested it on x86_64.
pbs-helper: backing up the host
pbs-helper
adds pbs-backup, which backs up the router’s root filesystem as a .pxar
archive. Configuration is UCI:
# /etc/config/pbs
config proxmox_backup_server 'pbs_server'
option enabled '1'
option repository 'backup@pbs@pbs.example.org:datastore'
option password '...'
option backup_name 'rootfs'
option prefix '/'
list backup_arg '--exclude=/tmp'
option hc_ping_url 'https://hc-ping.com/<uuid>'
option gotify_url 'https://gotify.example.org'
option gotify_app_token '...'
and a cron line such as 0 1 * * * /usr/bin/pbs-backup.
A few details I care about:
- The password stays off the command line.
proxmox-backup-clientreads it throughPBS_PASSWORD_CMD, which runsuci get, so it never shows up inpsor in a temporary file. - Two independent reports. Healthchecks gets
/startbefore the backup and/<exit code>after it; Gotify gets a Markdown message. Either can be left out, and one failing never affects the other. The report carries the head and the tail of the backup log — what was backed up and how it ended — and drops the thousands of chunk-upload lines in between, which keeps it under Healthchecks’ 10 kB body limit. - Logs go to syslog under cron and to the terminal when run by hand.
The job runner behind it, pbs.helper, is a small ucode module shared with
the VM backups below. All side effects — running commands, HTTP, logging —
are injected, so it is unit-tested without a router.
qemu-instance-init: VMs as services
qemu-instance-init started from the init script on the OpenWrt wiki, ported to procd. Each VM is a UCI section:
# /etc/config/qemu
config instance 'vm1'
option vcpus '2'
option ram '2G'
option root_disk '/dev/vg0/vm1'
option root_disk_type 'lvm'
option net0_mac '52:54:00:12:34:56'
option net0_bridge 'br-lan'
option guest_agent '1'
option backup '1'
option backup_server 'pbs_server'
procd starts and respawns the guests. On stop the script sends an ACPI
system_powerdown over QMP and gives the guest time to shut down cleanly
before procd kills it. UEFI guests work with OVMF, and the guest agent socket
is one option away.
The package comes in two variants. qemu-instance-init is just the init
script. qemu-instance-init-pbs adds /etc/init.d/qemu backup [instance],
which runs through the same pbs.helper job runner, with the same
Healthchecks and Gotify reports.
Consistent backups of running VMs
Reading the disk of a running VM gives you a backup of a disk that was
changing while you read it. The -pbs variant keeps it still:
- Freeze the guest filesystems through the guest agent, so the guest flushes its caches and stops writing.
- Snapshot the disk:
- LVM disks get an
lvcreate -ssnapshot. OpenWrt’skmod-dmis built without snapshot support, so the feed has a smallkmod-dm-snapshotpackage for that. - qcow2 disks get a temporary overlay through QMP
blockdev-snapshot-sync: guest writes go to the overlay, and the base file stops changing.
- LVM disks get an
- Thaw right away — the guest is frozen only for the snapshot itself.
- Back up the snapshot or the base file.
- Clean up: remove the LVM snapshot, or merge the overlay back with
block-commitand delete it.
Missing pieces degrade with a warning in the report rather than failing: without the guest agent you get a crash-consistent snapshot, without a snapshot a flush and then the live disk. Leftovers are handled too — a stale LVM snapshot is removed by the next backup, and an overlay left behind by a killed backup is merged back by the next backup or, if the VM was stopped, on the next start.