For several years, a small group of friends and I have been considering building a private overlay network for operation of our more esoteric, old, or weird networking needs. The work once again fell by the wayside as life happens, but once again, the discussion came up. Rather than attempting to brainstorm the ideas up again, I’m writing it down this time so we can start thinking about setting up the bits again, this time more organized.
Additionally, as I’ve changed jobs and spend more of my time sysadmining, I need a network project to remind me how networking works.
Purpose
Provide a interconnect and transit of traffic between private networks used by a set of friends for interior operations. This is not a network designed to allow you to access the Internet from somebody else’s connection. A limited case where overlay network devices can access a general internet connection might be supported. Doing so would use one of the below defined hub points to provide PNAT access to the general internet.
For the purpose of supporting non-IP traffic, specifically for supporting reto-computing and experimental network traffic, The desire is to be able to move both IP frames routed, and ethernet frames, bridged, between two lan segments.
Not all paths possible in the network are necessarily desired. The ability to specifically enable traffic transit between end points is a desireable feature.
Existing and expected resources
It is assumed that anybody participating in the network has permanent infrastructure located in a residence, connected via residential internet service to the general internet. RFC1918 space (192.168.x.y) is expected to be serving the local network segment(s), with a single external IP address assigned to a router and NAT gateway, performing IPv4 PNAT operations.
Participants in the network are assumed to be able to acquire and operate a network switch that supports 802.1Q vlans. Participants are also expected to be able to host a virtualization platform. The likely platform for development would be SmartOS.
A subset of the participants are expected to be operating servers inside of internet connected datacenters.
Technology to support operations
The need to move arbitrary ethernet frames implies a need for l2 encapsulation. Options that are obvious included vxlan and mpls.
The need for an end point to live in an IPv4 Nat environment, where outbound connections to the general internet as the only likely functional operation leads to more complications. For the purpose of this discussion, I’m assuming inbound port forwarding can not be set in all cases..
It is assumed that outbound connections to a hosted server on arbitrary ports is possible for the residential connections.
Network design and topology
Managing a network of this scope and scale where the friends list can exceed n=5 makes a network where direct point to point links for each pairing is unmanageable. Reducing the network to a star topology from a full mesh makes handling the connections easier, but introduces a single point of failure, the hub. I suggest using a dual star topology, where there are two hub points, operated independently, but coordinated. This places the limits of configuration complexity (each node needs to negotiate a join to no more than 2 hubs), while providing diversity (independent operation of hubs).
Transport security
Implicitly we want to move raw ethernet frames around to support retro-computing on non IP protocols. It is likely that those packets would at some point carry information that while not private, is not intended to be public. More importantly, absent any signing of the packets, arbitrary hosts on the Internet could introduce packets into the connected layer 2 environments.
At a minimum, we require that the transport mechanism can authenticate the packets. A desirable property is encryption of the content of the packets while traversing the general internet.
Next steps
Well, I guess I need to sit down and build parts of this thing.