N.04 / SYSTEMS
Sodama datapath engineering note
The full engineering record of rewriting a production datapath in Rust: kernel boundaries, map contracts, performance remediation, long-running stability and evidence scope.
Engineering notes on rewriting a production datapath in Rust: kernel boundaries, map contracts, reproducible evidence, and the unglamorous work of making a system stay healthy.
01 / The problem is the boundary, not the language.
A rewrite becomes meaningful only when it changes how a system can be reasoned about. Sodama begins with three constraints: the datapath must be Rust end to end; rule compilation must be comparable byte for byte with the Go reference; and every performance or stability claim must have a reproducible experiment.
The result is both a production-tested network system and a research record of Linux kernel paths, eBPF lifecycle, protocol behavior and resource governance.
02 / One datapath, eight programs, fourteen maps.
The kernel side contains seven TC classifiers and one sk_skb offload-verdict program. Fourteen BTF maps carry shared state. Rust / aya-ebpf defines the programs; Rust + aya loads and coordinates them from userspace.
The design keeps packet decisions close to the kernel while moving configuration, protocol implementations and operations into a testable control plane. The map layouts and single-writer discipline are treated as architectural contracts rather than implementation details.
03 / The control plane carries the complexity.
Userspace parses the complete .dae configuration and compiles rules. It implements DNS caching, TCP, regex and EDNS behavior; TCP relay, UDP NAT and TCP offload; health probes, group policy, failure switching, runtime state and metrics export.
Protocol work covers Shadowsocks, Trojan, VLESS, VMess, TUIC, Hysteria2 and AnyTLS, including UDP and session behavior. TLS SNI and HTTP Host sniffing preserve domain information for remote resolution. Each feature is evaluated as part of the datapath, not as an isolated checklist item.
04 / Performance is a measured wait, not an adjective.
A reload should not inherit the latency of a slow subscription fetch. Decoupling fetch from reload reduced panel acknowledgement from an observed 19–34 seconds to 0.74 seconds, while preserving validation, persistence and acknowledgement.
The regression laboratory still combines netns topologies, fault injection, cross-implementation clients, soak tests and byte-exact parity checks. A faster path counts only when the same contracts continue to pass.
| Unit tests | 268 / 268 | 7 ignored |
|---|---|---|
| Standard / one-arm topology | 9 / 9 · 9 / 9 | netns verification |
| DNS / forwarding / operations | 6 / 6 · 6 / 6 · 5 / 5 | domain-specific suites |
| Protocol ladder | 17 / 17 | 15 sessions |
| Go rule-compiler parity | 5 / 5 | byte-exact |
| Kernel matrix | 9 / 9 · 9 / 9 | host + QEMU |
05 / Stability is the absence of drift over a long window.
Real workloads exposed failures that feature tests could not: blocking health probes, unbounded handshake threads, sticky IPs that were never re-resolved, and reloads coupled to slow network reads.
After hard deadlines, bounded probe concurrency, a dial watchdog, resolve-on-dial and per-source failure isolation, a three-month long-running production test kept RSS resident at 68 MB, with no observed memory growth or memory step.
| 553 blocked receive threads | Socket timeouts + probe cap 96 + dial watchdog | No recurrence observed |
|---|---|---|
| Relay threads 36 | Return group dialing to the reactor | Relay threads 0; slope +2.39 → −0.11 |
| Sticky address drift | Resolve on dial, preserve healthy source members | Self-recovery without source loss |
| Reload acknowledgement | 19–34 s → 0.74 s | Panel acknowledgement after validation, persistence and reload; not full request latency. |
|---|---|---|
| RSS | 68 MB | Three-month long-running test, one production workload. |
| Unit tests | 268 / 268 · 7 ignored | A specific revision; passing tests are not proof of all paths. |
| Not published here | Hardware · kernel · commit SHA · traffic mix · full sample log | Required before treating these numbers as a general benchmark. |
03′ / A console without engine changes.
The operations panel uses an Axum backend and a Leptos → WASM frontend, shipped as a single musl binary. It reads existing Unix-socket state and metrics contracts; configuration changes pass through validation, persistence and reload acknowledgement.
The binary is approximately 2.6 MB with 940 KB of static assets. Without an active browser session, it idles at roughly 3 MB RSS and three threads. Runtime cards expose RSS, threads, file descriptors, CPU and connections; data-age indicators distinguish a stalled panel from a genuinely stable engine.
06 / Decisions need an exit condition.
Sixty-three accepted ADRs record context, alternatives, rationale, consequences and—most importantly—the measurement that would justify revision. The archive extends from language migration and map layouts to the M4 reactor and zero-thread handshakes.
The remaining backlog is explicit: packaging and release CI, fuzzing, additional eBPF attachment modes, supply-chain pinning and broader protocol coverage. Stability is not a claim that the system is finished; it is a description of behavior that continues to be measured.
Memory safety is the starting line. Operational safety is the discipline of knowing every thread, descriptor, deadline and failure path.