For the past five years, my home network hasn't just been a gateway to the internet; it's been a miniature ISP backbone. While most residential setups rely on a single ISP router doing basic NAT, I wanted something more robust: carrier-grade reliability, deterministic traffic engineering, and seamless multi-homing using my own public Autonomous System Number (AS209792).
By applying hyperscale datacenter design principles—like eBGP-on-the-host and Infrastructure as Code—I’ve built a network that treats residential ISPs as mere transport utilities.
The High-Level Architecture
The core of the network is built on redundancy. I am connected to two primary fiber ISPs in Singapore, MyRepublic and ViewQwest, along with a 5G backup link. Instead of relying on these ISPs for routing, I use them as "underlay" transport to reach my "Upstream Edge Cloud"—a fleet of six VPS nodes (edge1.sin through edge6.sin) hosted across providers.
The edge nodes are the true gateways to the internet. They announce my public prefixes to the global BGP table, while my home Core Router manages the paths between them.
The Connectivity Fabric: WireGuard and Deterministic Addressing
To link my Core Router to the cloud, I use a mesh of WireGuard tunnels. Rather than managing a messy internal IPAM (IP Address Management) database, I use a deterministic addressing logic.
Every tunnel interface is assigned an IPv6 address and a specific UDP listen port generated procedurally based on the IDs of the two nodes. This ensures that the same two peers always get the same IP pair and port, making firewalling and BGP peering definitions completely predictable across the entire mesh. To simplify things further, I utilize RFC 5549, which allows me to carry IPv4 routing information over an IPv6 next-hop, eliminating the need for private IPv4 point-to-point links.
Design Choice: eBGP Everywhere
A common question in BGP design is whether to use internal BGP (iBGP) or external BGP (eBGP). While iBGP is standard for single-ASN networks, it introduces complexity with split-horizon rules and requires Route Reflectors.
I chose the "eBGP in the Datacenter" approach. Every node in my network—the Core Router, the VPS edges, the pfSense firewall, and the Kubernetes nodes—operates under its own unique private ASN (645xx range).
Why this works:
- Loop Prevention: BGP’s
AS_PATHattribute naturally prevents loops. If a node sees its own ASN in a path, it drops the route. - Observability: I can look at my routing table and instantly see the path a packet took by looking at the sequence of ASNs.
- A Clean Public Face: To the internet, I am a single entity. My edge nodes use BIRD (Internet Routing Daemon) with a strict export filter that "scrubs" these internal private ASNs, ensuring the world only sees a clean, single-hop path originating from AS209792.
The Brain: SaltStack & The Salt Mine
Orchestrating this manually would be impossible. I use SaltStack as the network’s "Control Plane." The secret sauce is the Salt Mine, a secure metadata store on the Salt Master.
- Metadata Exchange: When a node joins, it generates its own WireGuard keys locally and pushes them, along with its current ISP IP, into the Salt Mine.
- Automated Handshakes: All other nodes query the Mine and automatically "wire" themselves to the new peer.
- Full System State: Salt doesn't just configure BGP. It manages the entire stack:
- systemd-networkd: Generates
.netdevand.networkfiles for the tunnels. - Firewalling: Automatically opens the procedurally generated UDP ports only for authorized peers.
- BIRD: Templates the BGP sessions and filters.
- Kubernetes: Peering pfSense with the K8s nodes so that Pod/Service IPs are reachable across the entire network without NAT.
- systemd-networkd: Generates
Reliability and Convergence
With residential ISPs, "silent failures" (where the link is up but traffic is blackholed) are a reality. To counter this, every BGP session runs BFD (Bidirectional Forwarding Detection). If a link drops even a few packets, BFD detects the failure in milliseconds and BGP reroutes traffic to a healthy ISP.
As a final failover, the 5G link is configured with heavy AS-path prepending. It stays in standby, receiving no traffic, until every other fiber path is exhausted.
Conclusion
This setup has been rock-solid for over five years. By moving the "intelligence" into SaltStack and using BGP for what it does best—path selection—I've created a home network that behaves like a professional datacenter fabric. It’s stateless, modular, and can scale to a new cloud provider with a single state.apply.