Notes on Vladimir Ermakov

obserwrt: flow export on OpenWrt after netobserv, pmacct and softflowd

Sat, 17 Oct 2026 00:00:00 +0300

In August I wanted to see what actually goes through the mesh: who talks to whom, over which tunnel, how much. The collector side was easy to choose — Akvorado takes IPFIX, polls the exporters over SNMP for interface names, and can take BGP routes over BMP. The hard part turned out to be the exporter on the OpenWrt routers.

obserwrt is where it ended: TC eBPF programs count flows in a kernel map, a small agent reads the map and sends IPFIX. The README describes it. This note is about how I got there — three ready-made exporters first, then my own, and what broke along the way.

Backing up OpenWrt hosts and their VMs to Proxmox Backup Server

Fri, 16 Oct 2026 00:00:00 +0300

Some of my OpenWrt boxes are x86 machines that also run a VM or two under plain QEMU. Everything else in the homelab is backed up to Proxmox Backup Server, so these should be too. Two packages in my OpenWrt feed do that.

The client

PBS backups need proxmox-backup-client, which is Rust and not in OpenWrt. The feed’s proxmox-backup package builds just the client from the upstream sources, with a few patches: a client-only cargo build, no libsystemd, and the system zstd. So far I have only tested it on x86_64.

siphon: getting CloudNativePG's WAL from a bucket into Proxmox Backup Server

Fri, 09 Oct 2026 00:00:00 +0300

Everything in my homelab is backed up to Proxmox Backup Server: retention, verification, and a copy on tape. In early October that stopped being true for the databases, and it took me a week and a small Rust tool to fix it. siphon streams an S3 bucket into a PBS snapshot and back; the README has the details. This note is about why it was needed and what got in the way.

pathosd: what it took to announce an address only while the service is up

Wed, 07 Oct 2026 00:00:00 +0300

Some services in my network run in several places at once — a DNS resolver on every hub, the S3 gateway on more than one storage node. Clients should see a single stable address, and routing should send them to the nearest instance that actually works. In a routed network that’s anycast: every instance announces the same address over BGP. The catch is the word works — a route has no idea whether the service behind it is alive.

Zone-o-Matic: how a DynDNS endpoint grew into the write side of my DNS

Mon, 05 Oct 2026 00:00:00 +0300

My authoritative DNS is about as simple as it gets: CoreDNS on the hubs serving plain zone files with the file plugin, which reloads them when the SOA serial changes. Simple to read, simple to back up, simple to diff in git.

Zone-o-Matic is the small Go service that writes to those files. It did not start out as a plan — every API in it arrived with a new client that needed to change a record.

Moving community.openwrt from shell to ucode

Fri, 02 Oct 2026 00:00:00 +0300

I manage a small fleet of OpenWrt routers with Ansible: cloud routers, a VPS and the boxes at home. The tool for that is community.openwrt, the successor of the old gekmihesg.openwrt role. Its trick is that the router needs no Python: the modules used to be shell scripts that the controller copied over SSH and ran with the router’s own BusyBox.

That works, but it is slow, and speed is what pushed me to rewrite it. In the shell uci module every option meant a few more forks of uci and jsonfilter, and my mesh needs a lot of options on every router — one task, one round of forks each. On top of that, JSON is assembled from shell strings, quoting is a minefield, and the modules are hard to test.

A homelab mesh over AmneziaWG

Sat, 26 Sep 2026 00:00:00 +0300

My routers live in several places: a few hubs on rented servers in different countries and a handful of sites — home, the homelab rack and a couple of others. They are all OpenWrt, and they are tied together with AmneziaWG, a WireGuard fork whose packets don’t look like WireGuard on the wire. This note is about the routing on top, not about the tunnels themselves.

Shape

One tunnel per link. Every link is its own point-to-point interface with exactly one peer: the hubs are joined into a mesh, every site connects to every hub. More interfaces, but each one is simple, has its own counters, and can be restarted on its own.
IPv6 inside the tunnels. Each link gets a /127 plus a link-local address. Every router has a loopback address that the rest of the network uses to reach it. Loopbacks live on Linux dummy interfaces, which stock OpenWrt can’t declare in /etc/config/network, so I packaged a small netifd protocol for them — netifd-proto-dummy, with a LuCI page in luci-proto-dummy.
Everything is in Ansible. Interfaces, keys, firewall zones and routing are generated from inventory with community.openwrt. The tunnels themselves — keys, shared PSKs, peers, cleanup — are handled by the wg_interface and wg_peer modules I wrote for exactly this mesh (#281), and the rest goes through batched uci operations — see the ucode note.

Routing

One BIRD daemon per router:

Watching BIRD on OpenWrt: metrics, ubus and a LuCI page

Sat, 12 Sep 2026 00:00:00 +0300

Every router in my mesh runs BIRD. birdc over SSH is fine for one box and tiring for ten, so I built three small packages that look at BIRD from three sides: Prometheus, ubus and LuCI. All of them live in my OpenWrt feed and all of them are written in ucode.

BIRD’s control socket

BIRD has a simple line protocol on a unix socket — the same one birdc uses. You connect, read a banner, send a command and read the reply; every reply line starts with a four-digit code followed by - (more lines follow) or a space (last line). That is easy to speak from ucode with the socket module, so none of the packages need birdc or a separate exporter binary.

inotify-rsync: push files to other routers the moment they change

Sun, 16 Aug 2026 00:00:00 +0300

Some files on my hubs are produced in one place: one hub renders a few static files and downloads a few big ones, and the other hubs serve copies. The copies should follow quickly. A cron job means either lag or a lot of pointless runs, so I wanted rsync to fire when something actually changes.

inotify-rsync does exactly that and nothing more: watch some directories, run rsync to a destination when they change.