← Notes

obserwrt: flow export on OpenWrt after netobserv, pmacct and softflowd

October 17, 2026 · ipfix, ebpf, openwrt, akvorado, ucode, cpp

In August I wanted to see what actually goes through the mesh: who talks to whom, over which tunnel, how much. The collector side was easy to choose — Akvorado takes IPFIX, polls the exporters over SNMP for interface names, and can take BGP routes over BMP. The hard part turned out to be the exporter on the OpenWrt routers.

obserwrt is where it ended: TC eBPF programs count flows in a kernel map, a small agent reads the map and sends IPFIX. The README describes it. This note is about how I got there — three ready-made exporters first, then my own, and what broke along the way.

netobserv-ebpf-agent: only templates

August 15. The first pick was the eBPF agent from the OpenShift Network Observability project — eBPF on TC hooks was what I wanted anyway, and my routers’ kernels have BTF. On the first deploy nothing started at all: the packaged init scripts read their config from a section named service, and I had named mine differently. The interface list also had to be a UCI list, not a comma-joined string.

Once it ran, Akvorado still got no flows. The agent sent small packets, about 100 bytes each — IPFIX templates, and nothing else. August 18 I switched the default to pmacct with a commit message saying “need to investigate later”. I never did, so the cause is still unknown.

pmacct: Akvorado threw the flows away

pmacct did send flows — and Akvorado dropped every one of them with input and output interfaces missing. Fixing that took a day and three separate problems:

1. No interface fields. Flows only carry the input/output interface if aggregate lists in_iface,out_iface, and the packaged init script did not pass the aggregate option from UCI to pmacctd at all. I shipped my own init script.
2. Wrong interface numbers. The script numbered the traced interfaces 1, 2, 3… in pmacct’s interface map. Akvorado learns the interfaces over SNMP, which reports the kernel ifIndex — so the numbers matched nothing. The script now reads /sys/class/net/<dev>/ifindex.
3. Wrong exporter address. Akvorado identifies an exporter by its address, and pmacct reported it as an IPv4-mapped IPv6 address (::ffff:192.0.2.1), which Akvorado doesn’t unmap. Setting nfprobe_source_ip to a plain IPv4 address, explicitly per hub, fixed it.

And all of that with sampling 1:100, since pmacct copies packets to user space.

softflowd: the same evening

August 19, evening. I added a third backend: softflowd, one instance per traced interface. It is the exporter OpenWrt already packages, and it is simple. Two days later I started my own instead. The reasons I wrote down afterwards in obserwrt’s README (so this is my later summary, not a note made at the time):

• softflowd reads packets through AF_PACKET: every packet is copied to user space and classified there, so its CPU cost grows with the packet rate — a bad fit for small MIPS routers.
• My tunnels are torn down and created again on every ifdown/ifup, with a new ifIndex each time. A sniffer bound at start keeps watching the old one.

obserwrt in ucode

August 21. The design: TC ingress/egress eBPF programs on the chosen devices update a flow map in the kernel; userspace work is proportional to the number of flows, not packets, so no sampling is needed. The agent was first written in ucode, OpenWrt’s scripting language, so the package would stay tiny.

Getting the first packets counted on a real router:

• libbpf 1.6 no longer guesses the program type from a classifier/ section name, so it has to be set explicitly; and an LRU hash map refuses BPF_F_NO_PREALLOC.
• AmneziaWG devices have no Ethernet header: the packet starts with the IP header. The program had to handle both kinds of framing (and how it did that came back in October).
• ucode is not JavaScript, even though it looks like it: push() is a global function, not a method; strings can’t be indexed; for…in over an array yields elements, not indices; there’s no throw and no function hoisting. Each of these broke the agent silently on the device. The hoisting one came back twice more — the “fix” with a forward declaration doesn’t work on the ucode version OpenWrt ships.
• The agent asked netifd for device status on every loop and kept it at 40% CPU. Devices now come from events, with a slow reconcile every 30 s.

August 23 the IPFIX exporter was in, checked first by decoding a tcpdump capture by hand and then by an end-to-end test against goflow2 in CI. August 25 the router role in my Ansible repo switched to obserwrt and dropped the other three exporters — and the pmacct init script with them. The exporter address stayed pinned to the same plain IPv4 so Akvorado kept seeing the same exporters. A day later the routers also traced the LAN bridge and the WAN.

Running it on the whole mesh found the next problems:

• The verifier rejected a program that walked past two VLAN tags until the IP header pointer got its own bounds check.
• A flow can be evicted from the LRU map between listing the keys and reading the value; the agent has to expect null there.
• The 5-second pass re-exported every live flow, including idle ones — zero-value records, and a core pinned once the map was near its limit (the busiest site runs up to 32k flows). Now a flow is sent only when it is new, has changed, or has expired.

Rewrite in C++

September 4–5. Even with that fixed, the ucode VM cost too much CPU and RAM on the low-end MIPS routers. The agent was rewritten in C++23. The eBPF program and the flow model didn’t change, and golden test vectors taken from the ucode version kept the IPFIX output byte-identical. netifd/ubus was replaced with rtnetlink: the kernel reports devices with their ifIndex directly. It also builds as a plain-Linux .deb.

Two bugs from that weekend are worth remembering:

• Link events never arrived. The startup dump of devices worked, but ifup/ifdown went unnoticed. The netlink socket joined the RTMGRP_LINK group with setsockopt and was then bound with nl_groups = 0 — and bind replaces the group set, so the membership was gone. The canonical way is to put the group into the bind address:

  struct sockaddr_nl sa = { .nl_family = AF_NETLINK, .nl_groups = RTMGRP_LINK };
  bind(fd, (struct sockaddr *)&sa, sizeof(sa));
  

• A map of size zero. After reflashing one router the daemon crash-looped with EINVAL. The OpenWrt Makefile passed -DFLOW_MAP_ENTRIES= with an empty variable, so the macro was defined but empty, the #ifndef default in the eBPF source never applied, and the object carried max_entries = 0. The define is gone; the size is set at load time (8192 by default, 32768 on the busiest site).

Beyond routers

October. The same exporter went onto the Proxmox nodes, and with it a bug that the routers had hidden. The program decided between IP and Ethernet by the first byte of the packet — the IP version nibble. On Ethernet that byte is the destination MAC, so every frame to a MAC beginning with 4x: or 6x: was parsed as IP garbage and its flow lost. Now each device gets an _eth or _l3 program pair, chosen by its link type when it is attached. The same release fixed the IPv6 extension-header length (off by 8 bytes, so for example MLD behind Hop-by-Hop was dropped) and counted GRO/TSO super-packets as one packet. These are now tested in CI by running the eBPF programs with BPF_PROG_TEST_RUN; the old object fails all 11 checks.

Around it, two small things in the router config: the firewall drops IPFIX going out to the WAN, so with the tunnels down flow data doesn’t leak upstream over the default route; and snmpd got a minimal read-only config, limited to the interface tables Akvorado polls.

And routes over BMP

October 17. The last piece was routes: BIRD sends them to Akvorado over BMP. Akvorado saw the peers come up but got no routes from one hub. With pre-policy monitoring and no import tables, BIRD before 3.3.3 skips sending routes entirely (BIRD issue #463); that hub runs 3.3.2. BMP is post-policy only until it is upgraded.