Problem description: Bring up a connection to the AMPR network, and provide a way for running Triton instances on addressing provided by ampr.org.
Long ago, in the stone age of the Internet, a block of IPv4 addresses were allocated for amateur radio use and experimentation. The current implementation of this is documented and supported at ampr.org. If you have appropriate standing, namely, you have an amateur radio license, you can have a block of addresses assigned to you for use. Once you have the addresses, routing them and using them is another matter.
Given a sufficiently large block and cooperation with your ISP, and coordinating with AMPR.org, you can do the normal thing and announce addresses by BGP. That's the easy way, which I'll now be ignoring. The second way is to join an IPIP overlay network, and not a simple one.
The system design is based on the theory that all of the nodes participating in the IPIP overlay network will build tunnels between all of the other nodes, generating a full mesh tunnel network. Data is provided via a number of feeds regarding the ip subnets and gateway ip addresses for the machines in question.
I run a SmartOS zone as a router connected to the Internet for my tiny datacenter needs. It provides NAT for the data closet, haproxy, name service, etc. I decided one day to setup the necessary stuff for it to also route my AMPR network.
To accomplish this on SmartOS, you end up having to build an individual tunnel with routing statements to each other endpoint in the IPIP mesh. In Linux, one of the most common ways to do this is the munge script. This works great for Linux, but obviously uses Linux kernel commands. illumos is not Linux, so a completely different set of commands is required.
In the end, I ended up building my own munge script for SmartOS. The following script is useful for implementations that have a small block of addresses that you wish to route for. It is expected that you've attached your SmartOS zone to an ethernet/ip network of your allocation, and your router SmartOS zone is the first address of that subnet. Additionally, you will need to enable ip spoofing for that interface from the global zone.
Using the api to pull the json encap file, this builds up tunnels and adds routes for everything in the encap file. It can be re-run on a new file and it will rebuild the whole set. Errors are emitted, expected and harmless for cases where the endpoint is already setup and all that's needed is another route. This has not been tested anywhere an unrelated existing IPIP tunnel is in operation or anywhere where you call tunnels ampr for some reason.
#!/bin/env bash
API=portal.ampr.org/api/v1/encap
USER=amprusername
APIKEY=ABCDEFGHIJKLMNOPQRSTUVWXYZ
LOCALGATEWAY=1.2.3.4
ip2int()
{
local a b c d
{ IFS=. read a b c d; } <<< $1
echo $(((((((a << 8) | b) << 8) | c) << 8) | d))
}
int2ip()
{
local ui32=$1; shift
local ip n
for n in 1 2 3 4; do
ip=$((ui32 & 0xff))${ip:+.}$ip
ui32=$((ui32 >> 8))
done
echo $ip
}
netmask()
# Example: netmask 24 => 255.255.255.0
{
local mask=$((0xffffffff << (32 - $1))); shift
int2ip $mask
}
broadcast()
# Example: broadcast 192.0.2.0 24 => 192.0.2.255
{
local addr=$(ip2int $1); shift
local mask=$((0xffffffff << (32 -$1))); shift
int2ip $((addr | ~mask))
}
network()
# Example: network 192.0.2.0 24 => 192.0.2.0
{
local addr=$(ip2int $1); shift
local mask=$((0xffffffff << (32 -$1))); shift
int2ip $((addr & mask))
}
# Teardown the existing AMPR interfaces
for iface in `ifconfig | grep ampr | cut -d : -f 1`;
do ifconfig $iface unplumb;
done;
# Collect the most recent encap.json
curl https://$USER:$APIKEY@$API > encap.json
#extract the list of all other gateways
cat encap.json | json -a gatewayIP | sort -n | uniq > gateways
LAST=`dladm show-iptun | grep ampr | tail -1 | sed -e 's/^ampr\(.\?\).*$/\1/'`
#Build an IPIP tunnel interface to every gateway in the file.
for gateway in `cat gateways `;
do LAST=`dladm show-iptun | grep ampr | tail -1 | sed -e 's/^ampr\([[:digit:]]\+\).*$/\1/'`;
NEXT=$((LAST+1));
dladm create-iptun -t -T ipv4 -a local=$LOCALGATEWAY -a remote=$gateway ampr$NEXT;
done;
cat encap.json | json -d : -a gatewayIP network maskLength > gwnm
for gwnm in `cat gwnm`;
do GW=`echo $gwnm | cut -d ':' -f 1`
NETWORK=`echo $gwnm | cut -d ':' -f 2`
NETINT=`ip2int $NETWORK`
FIRSTINT=$((NETINT+1))
FIRSTHOST=`int2ip $FIRSTINT`
MASKLEN=`echo $gwnm | cut -d ':' -f 3`
IFACE=`dladm show-iptun -p -o remote,link | grep $GW | cut -f 2 -d ':'`
if [ $MASKLEN -eq '32' ]
then
echo ifconfig $IFACE plumb 44.102.105.120 $NETWORK netmask `netmask $MASKLEN` up
ifconfig $IFACE plumb 44.102.105.120 $NETWORK netmask `netmask $MASKLEN` up
route add $NETWORK/$MASKLEN $NETWORK
else
echo ifconfig $IFACE plumb 44.102.105.121 $FIRSTHOST netmask `netmask $MASKLEN` up
ifconfig $IFACE plumb 44.102.105.121 $FIRSTHOST netmask `netmask $MASKLEN` up
route add $NETWORK/$MASKLEN $FIRSTHOST
fi
done;
An interesting aspect of the function, shown in these last few lines, is that in cases where a single address is being routed for, one needs to plumb and route the base address, rather than the address of the first host. The IP addresses shown here are my 44net allocation, and are left intact to illustrate the cases.
On my deployment, after running this script, I end up with 610 interfaces / tunnels, along with 1385 routes added to my routing table. Traffic moves as expected in this environment.
Having accomplished this, I'm tearing it all out. While SmartOS is really good at almost everything I want to do, it makes a very poor router. The shortcomings are in easy configuration (no commonality with the rest of the operators) and packet handling for hairpin traffic. Also, I essentially give up several potentially useful addresses to provide network and broadcast addresses for the ip network. The replacement will look different, but I do not know how yet.
This post closes out that experiment. I have written down the results. Do not follow me in this path. There is nothing worth while at the end of this road.