← Notes

siphon: getting CloudNativePG's WAL from a bucket into Proxmox Backup Server

October 9, 2026 · backup, proxmox, cloudnativepg, postgres, s3, ceph, rust, kubernetes

Everything in my homelab is backed up to Proxmox Backup Server: retention, verification, and a copy on tape. In early October that stopped being true for the databases, and it took me a week and a small Rust tool to fix it. siphon streams an S3 bucket into a PBS snapshot and back; the README has the details. This note is about why it was needed and what got in the way.

How the databases fell out of PBS

October 3–7. I was moving services from VMs and compose stacks to small Kubernetes clusters — Forgejo first, then Healthchecks, Woodpecker, Immich, Paperless, Open Archiver. Their file data kept going to PBS: a generic CronJob mounts the volume read-only next to the running pod and runs the backup client, with an optional pre-backup step (Paperless exports its documents first).

The databases went a different way. Each app got a CloudNativePG cluster, and CloudNativePG backs up through the Barman Cloud plugin: WAL archived continuously plus a daily base backup, all into one shared bucket on Ceph RGW, a prefix per service, 30 days. That is the right tool — it gives point-in-time recovery, which a nightly pg_dump doesn’t. But the old backups had a postgresql.pxar with a dump in every PBS snapshot, and the new ones didn’t. Commit by commit the databases left PBS for a bucket that lives on the same Ceph cluster as the databases themselves: fine for a broken database, useless for a broken Ceph, and never on tape.

Why not just rclone

October 8. So the bucket had to get into PBS. PBS only backs up files, and the obvious ways around that all had a catch:

• rclone sync to CephFS, then proxmox-backup-client — the bucket would sit on the same Ceph twice.
• rclone mount and back up the mount — FUSE needs extra privileges in the pod, and long reads through FUSE are fragile.
• A third-party Go implementation of the PBS protocol exists, but I couldn’t judge how mature it is.

So: a small Rust tool on Proxmox’s own crates — the same chunker, index format and protocol code as the official client — streaming object bodies straight from S3 into PBS. Nothing touches local disk, which also keeps the CronJob simple: one per bucket, after the base backups, with a read-only bucket user.

Building on Proxmox crates

The crates I needed, pbs-client and pbs-datastore, aren’t on crates.io — they live in the proxmox-backup workspace on git.proxmox.com. That part took longer than the backup logic:

• No release tags to pin to. Proxmox doesn’t tag every release; there are only “bump version to 4.2.x-1” commits. siphon pins a commit by rev, picked to match the PBS server (or older).
• One revision for everything. pbs-client pulls a dozen more proxmox-* crates, plus pxar and pbs-api-types. To make my direct dependencies and its dependencies resolve to the same code, each repo is pinned once through [patch.crates-io] — which works only in the root workspace, so it lives in siphon’s Cargo.toml.
• Native libraries. pbs-client links libacl, libcrypt, libssl, libsystemd, libuuid and libzstd, with bindgen at build time. No static musl binary, then: a glibc build on Debian trixie, in a container.
• AGPL. proxmox-backup is AGPL-3.0, so siphon is too.

I also looked at building against Debian’s packaged librust-*-dev crates, and at a git submodule; both added weight without solving anything a pinned git dependency doesn’t.

An S3 client that drops headers

The first choice for S3 was proxmox-s3-client, already in the dependency graph. It doesn’t fit: it keeps only size, ETag, Content-Type and Last-Modified from a response, so no x-amz-meta-*; it can’t set metadata on upload; and it has no multipart upload. Restore needs all three, so siphon uses aws-sdk-s3. Two settings mattered with RGW: checksums only when an operation requires them (not every S3-compatible store understands the SDK’s newer defaults), and the CA from the system store.

Object metadata (Content-Type and friends, x-amz-meta-*, ETag, version ID) goes into the archive as user.s3.* xattrs — under user. so even a plain proxmox-backup-client restore to a directory keeps it.

Keys that aren’t paths

The archive tree is the bucket: key a/b/c is file c in directory a/b. S3 keys are just strings, though. a//b, a dir/ folder marker, .. as a component, a 300-byte component, or a next to a/b — none of these can be a file. Those objects are stored flat under .siphon/keys/<sha256> with the real key in an xattr. The rustfs server I use in the end-to-end test rejects most of these keys itself, so they are covered by unit tests only.

By the evening of October 8 backup and restore worked end to end in CI: rustfs and a containerised PBS 4.2, backup twice, PBS verify, restore, compare every body and header. The CronJob went into my GitOps repo the same night, waiting for its secrets.

A snapshot the web UI couldn’t open

October 9. The first real snapshot was in PBS, and the PBS web UI refused to browse it: unable to read dynamic index … catalog.pcat1.didx. The UI doesn’t read a .pxar archive directly; it browses the catalog that proxmox-backup-client uploads next to it, and siphon didn’t write one. Now it builds the catalog in memory while encoding (tens of bytes per object) and uploads it after the archive. The end-to-end test lists the catalog and downloads a single file through the PBS API, so this can’t come back silently. (That the first deployed run is what found it is my reading of the timestamps — the deploy commit and the fix are two hours apart.)

Not uploading the same chunks again

The same day: like the official client, siphon now passes the previous snapshot’s manifest to the upload, so chunks that are already in PBS aren’t sent again. PBS deduplicates on disk anyway, but this saves the transfer.

I wanted the reuse in the summary — archive 3.1 GiB, reused 3.09 GiB (99.6%) — but pbs-client keeps its counters private. siphon hashes each chunk once more and applies the same rule itself.

The test for it was flaky at first. The third backup adds one WAL file and should reuse some chunks, but with random test data the result jumped between 0% and 69% depending on where the chunker cut. The data now comes from fixed seeds, and the base object is 40 MiB — more than twice the chunker’s 16 MiB maximum chunk — so the chunks before the new object are always reused.

Last, siphon reports to Healthchecks itself: a start ping, then success with the summary or failure with the error chain, so a run that hangs or fails shows up there and goes to my usual notifications.

What’s left

Restore is the other half: siphon restore puts a snapshot back into a new bucket with its metadata, and a CloudNativePG cluster can recover from that bucket as from the original.

Every run still downloads the whole bucket; only the upload to PBS is incremental. The next step is the split archive the official client uses for change detection: objects whose key, size, ETag and timestamp match the previous snapshot would be copied from it without a single S3 request, and only new WAL and base backups would be read.