I’ve been planning a network expansion for a while now. For the next few weeks, I will be in total geek-mode as I travel to several European cities to deploy AS50869 on a european ring. At the same time, my buddy Fred from IP-Max has been wanting to go to Amsterdam. IP-Max’s network is considerably larger than mine, but it just never clicked with the right set of circumstances for them to deploy in the Netherlands, until the stars aligned …
Leadup to the Roadtrip
Usually, IP-Max deploys their routers by having them shipped into the destination location, but this time was special. We decided to make a roadtrip out of it, so Fred made his way from Geneva to Brüttisellen, stayed the night, and early on Monday May 17th, we packed up the car and started our trek.
It turns out we had estimated our risk profile completely wrong - we thought it would be hard to cross the border into Germany due to the ongoing pandemic, but actually that part was fine. The Germans had opened their borders for transit traffic and stays of up to 24hrs just a few days ago, and we both got a (negative) PCR test so we felt we had our bases covered.
The Border
Then when we arrived at the border, perhaps because we had Geneva license plates, we were asked about our trip, business or pleasure, and we shared that we had some equipment with us. Thus begun the four-and-a-half hour customs exercise that was necessary for us to safely send our equipment off to the European Union. One would think it should be easy, but it actually wasn’t quite that easy, considering we arrived at the border at 9am on a Monday, and the traffic into Switzerland was queueing up all expeditor and logistics companies, so nobody really was willing to help us out. But we made it and left again shortly after 1:30pm.
Frankfurt
We arrived at Frankfurt Equinix FR5 at the Kleyerstrasse at around 5pm. The IP-Max rack was quickly found, and while Fred was installing their corporate Xen host to run remote VMs for the Frankfurt area, I deployed the first router of the trip: defra0.ipng.ch.
IP-Max at this location has a respectable 30G of DWDM capacity from three different vendors into Zurich, 30G of LAG capacity towards DE-CIX, and a 10G DWDM wave into Anzin (France), which will be broken up for us in Amsterdam for a future blogpost - stay tuned :)
Making use of line card and route processor redundancy, we decided to use three line cards, reserving one TenGig ethernet port on each:
- Te0/0/0/4 – EoMPLS to NTT/eShelter Rumlang (chrma0.ipng.ch)
- Te0/1/0/4 – EoMPLS to Interxion ZUR1 (chgtg0.ipng.ch)
- Te0/2/0/4 – EoMPLS to Amsterdam NIKHEF (nlams0.ipng.ch)
At each site, specifically those that are a bit further away, I deploy a standard issue PCEngines APU with 802.11ac WiFi, serial, and IPMI access to any machine that may be there. If you ever visit a datacenter floor where I’m present, look for SSID AS50869 FRA in the case of Kleyerstrasse. The password is IPngGuest, you’re welcome to some bits of bandwidth in a pinch :)
You can see my router dangling off what looks like a fiber optic umbellical cord under er01.fra05.ip-max.net, right at the heart of the Frankfurt internet.
Logical Configuration
console.fra.ipng.nl At the top of the rack you can also see the blue APU3 with its WiFi antennas. It takes an IPv4 /29 and IPv6 /64 from IP-Max AS25091 which gives me access to my equipment even if bad things happen (and they will, it’s just a matter of time!). It also exposes a WireGuard so that I can access it even without the need for SSH which can come in useful if a KVM console is required. Note the logo :-)
On the inside of the APU, it configures one RFC1918 wifi segment and another RFC1918 wired segment. In this case, the wired segment is connected to the IPMI port of the Supermicro router. I have really gotten used to this style of deployment – I start with the OOB. Once the APU has power (and it does not need to have an uplink yet), I can already SSH to it from the wireless segment, and further configure it. Once it’s done, I make a habit of rebooting it to ensure it comes up. Then, I can easily configure (and even entirely install!!) the server behind it using IPMI serial-over-lan and HTML5 KVM if need be. It’s delicious. And, it has saved my ass several times over the years!
defra0.ipng.ch Making use of the line card redundancy, there is now 3x 10Gig connected to my router, which immediately makes it one of the better connected hosts in this facility. Logging in via IPMI, the DANOS image is quickly configured. There’s one link to Interxion ZUR1 in Glattbrugg, one link to eShelter in Rümlang, and one link up to Amsterdam. The interface towards Interxion ZUR1 doubles up as an egresspoint for now. There will be an IPv4/IPv6 transit session with AS25091, a DE-CIX connection and possibly but probably not a Kleyrex connection, were it not for the murderous cross connect costs at this facility.
The results
After the OSPF and OSPFv3 adjacencies came up, iBGP was next. For now, the machine is single-homed off of chrma0.ipng.ch but soon there will be as well a leg towards Amsterdam. So for now, all that we can do is test basic connectivity. So after finishing our trip to Amsterdam, and checking into our AirBnB ready to go through our quarantine song-and-dance, we spent a little time celebrating - we arrived at 1:30am, and turned in for the night at 3am. The next day, our groceries arrived, somehow unfortunately I had to be “well prepared” and ordered them to be delivered between 7-8am on Tuesday.
After a full day of regular work, we spent the evening taking a look at how my kit performs, and we are happy to report it’s absolutely great:
pim@defra0:~$ iperf3 -c chgtg0.ipng.ch -P 10
...
[SUM] 0.00-10.00 sec 11.2 GBytes 9.63 Gbits/sec 281 sender
[SUM] 0.00-10.02 sec 11.2 GBytes 9.56 Gbits/sec receiver
pim@defra0:~$ iperf3 -c chgtg0.ipng.ch -P 10 -R
...
[SUM] 0.00-10.01 sec 10.2 GBytes 8.73 Gbits/sec 550 sender
[SUM] 0.00-10.00 sec 10.1 GBytes 8.70 Gbits/sec receiver
pim@defra0:~$ ping4 chrma0.ipng.ch
PING chrma0.ipng.ch (194.1.163.0) 56(84) bytes of data.
...
--- chrma0.ipng.ch ping statistics ---
9 packets transmitted, 9 received, 0% packet loss, time 20ms
rtt min/avg/max/mdev = 5.864/6.022/6.173/0.072 ms
The roundtrip latency to Zurich is about 6.0ms, and the performance is north of 9Gbit in both directions for my router. Soon, we will go to Amsterdam, and deploy router number two (of four!) on this epic roadtrip: nlams0.ipng.ch which is a bucket list item of mine – to peer at Amsterdam Science Park.
More on that later!