I’ve been planning a network expansion for a while now. For the next few weeks, I will be in total geek-mode as I travel to several European cities to deploy AS50869 on a european ring. At the same time, my buddy Fred from IP-Max has been wanting to go to Amsterdam. IP-Max’s network is considerably larger than mine, but it just never clicked with the right set of circumstances for them to deploy in the Netherlands, until the stars aligned …
Leadup for IP-Max
Usually, if I were to go deploy somewhere with IP-Max, I settle down on top of (or underneath, or in some way physically close to) their router at a point of presence of theirs. In Amsterdam though, it was different … because IP-Max had not yet built a PoP here.
But I ask: why would that stop us? Fred told me last year that he had always wanted to build out a PoP in Amsterdam, but somehow he never really found the time. I offered to do the work to organize the local supplier chain, get a good spot in a well connected place, long haul to France and Germany, and otherwise exercise my (social) network to get it done.
In March 2021, I stumbled across rackspace at NIKHEF by working with the folks from ERITAP who got their hands on something that is less of a commodity: a full rack + power (the facility is always chronically oversubscribed).
A few chores on the tasklist:
- Sign for rackspace. Check.
- Order the IP-Max standard-issue small pop kit, which consists of:
- Procure an out-of-band provider for our APU4, easily found at NIKHEF (thanks, Arend)!
- Get connectivity in and out of Amsterdam!
Connectivity
The most important piece of planning is around the long haul connectivity. Considering IP-Max already operates a circuit from Frankfurt (Germany) to Anzin (France), I arranged for that link to be rerouted through Amsterdam and broken into two segments: Frankfurt-Amsterdam and Amsterdam-Anzin. I was like a kid in a candy store being able to meticulously choose the route that the fiber takes – over Düsseldorf, entering the Netherlands at Emmerik, over Arnhem and Ede, and to Amsterdam. A very direct route, using a 10Gig DWDM wave.
The other span goes from Amsterdam through Antwerp (Belgium) and Brussels and finally landing in Anzin (near Lille, France), which was the previous 10Gig DWDM wave, so there is no increased latency even though the link is broken up in Amsterdam. Yaay!
Delivery of the DWDM waves was ordered on March 30th, and although it should normally take 25 working days to deliver, for some awkward reason with the supplier it was going to take way longer than what we could afford, so a spot of VP style escalation took place, and oh look! Now it would take four weeks to turn up, was completed last Friday, which was just in time for our trip. Double yaay!
Staging Amsterdam
Because this is a completely new site for IP-Max as well as IPng, we’ll have to do a bit more work. And this suites us just fine, because after driving through Frankfurt (see my previous post), to the Netherlands, we have to stay in quarantine for five days (or, ten if we happen to fail our PCR test after five days!), which gives us plenty of time to stage and configure what will be our Cisco er01.ams01.ip-max.net and our Nexus as01.ams01.ip-max.net.
Of course, figuring out how all of this fits together is a nice exercise, and we planned to just plug and play the ASR9k, which worked out rather successfully by the way, so it had to be completely configured ahead of time. We created the interfaces, DNS, routing protocols like OSPF, OSPFv3, MPLS/LDP, BGP and all of the good stuff like ACLs, accounts and et cetera.
We staged the stuff in the laundry room of our AirBnB, being actually quite grateful once the staging was complete and we could turn the machines off.
For IPng, staging nlams0.ipng.ch was already done ahead of time. So all I really needed for it, was to ensure that the EoMPLS circuits were created ahead of time. I was really looking forward to seeing if we could beat 14ms to Amsterdam on the IP-Max network.
Extracurriculars
Besides the staging, we also ate some pretty delicious food:
- Mushroom risotto
- HotPot with Arend and Esther
- Chicken vegetable soup
- Tacos w/ Tapas
- Steak w/ broccoli and potatoes
- Red tuna w/ beans and herbs
But we also took the time to explore a little bit, for example on Kaz’s new boat through the canals and over the river Amstel. But mostly: we sat home and enjoyed our quarantine the best we could :-)
Deployment (day 1)
First before the day started, I drained the Frankfurt-Anzin link by raising OSPF cost on er01.fra01.ip-max.net and er01.lil02.ip-max.net while Fred notified customers and the IP-Max team of the impending update to the network.
We met up with ERITAP on Monday 24th, or target deploy date. We had labeled and packed up all of our gear, grabbed the car, and made our way to the Watergraafsmeer to the place where the Internet landed in Europe in 1982. Almost 40 years later, here we are: IP-Max is moving in!
The physical work was not very exciting. The Nexus, ASR, two APUs and my own Supermicro were racked in only a few minutes. But then the interesting bits begin – how do we connect all of this without making a Kabelsalat that you so often see in people’s racks.
After we connected the whole thing up and did extensive ping tests, we undrained the spans and saw a respectable 600Mbit of traffic traverse the new router. Because there were a few other folks tinkering in the rack (for example our friends from Coloclue we decided to adjourn for the day and visit Paul and Henrieke up in Almere for a fabulous homecooked meal (thanks again for the Picaña!) and we enjoyed being followed by the cops when driving back out of Almere – but we were not bothered/hassled by them.
Deployment (day 2)
So luckily we had already decided to go back for a day2 to complete the rack install, mostly for things like the fiber patch panel for IP-Max customers in the ERITAP rack, and to ensure that our power, serial and network cables would not come loose, because packets don’t like loose cables. Certainly we should avoid the electrons or photons falling onto the floor…
But the weird thing about my link errors (as seen by the ASR9k) was that usually the problem is either a duplex error (which was OK), or a dirty fiber or transciever (which was unlikely considering this link was a SFP+ DAC!). So that leaves either a faulty Cisco or a faulty Supermicro, neither of which are appealing.
On day two, after breakfast, we had to do a few chores first (like the claim for the VAT for imports, see our previous post, and as well get a corona PCR test for the way to France (which was absolutely horrible, by the way, I still feel my nose which was violated). So we hit NIKHEF at around 4pm to finish the job and take care of a few small favors for Coloclue, ERITAP and Byteworks, who are also in the same rack as IPng and IP-Max.
The results
After I replaced the DAC (ironically with an SFP+ optic), once OSPF and iBGP came back to life, this is what it looked like:
pim@chumbucket:~$ traceroute nlams0.ipng.ch
traceroute to nlams0.ipng.ch (194.1.163.32), 30 hops max, 60 byte packets
1 chbtl1.ipng.ch (194.1.163.67) 0.292 ms 0.216 ms 0.179 ms
2 chgtg0.ipng.ch (194.1.163.19) 0.599 ms 0.565 ms 0.531 ms
3 chrma0.ipng.ch (194.1.163.8) 0.873 ms 0.840 ms 0.806 ms
4 defra0.ipng.ch (194.1.163.25) 6.783 ms 6.751 ms 6.718 ms
5 nlams0.ipng.ch (194.1.163.32) 12.864 ms 12.831 ms 12.798 ms
pim@nlams0:~$ iperf3 -P 10 -c chgtg0.ipng.ch
...
[SUM] 0.00-10.00 sec 11.0 GBytes 9.49 Gbits/sec 95 sender
[SUM] 0.00-10.01 sec 11.0 GBytes 9.41 Gbits/sec receiver
pim@nlams0:~$ iperf3 -P 10 -c chgtg0.ipng.ch -R
...
[SUM] 0.00-10.01 sec 10.0 GBytes 8.62 Gbits/sec 339 sender
[SUM] 0.00-10.00 sec 9.98 GBytes 8.57 Gbits/sec receiver
So I spent the reminder of the night hanging out with Fred while pumping
9Gbit in both directions for 2 hours while traffic was low. It’s one thing
to do an iperf in your basement rack, but it’s an entirely different feeling
to do an iperf spanning three countries in Europe (CH, DE and NL). I will
note that the spans from Zurich to Frankfurt didn’t even get warm, although
the one from Frankfurt to Amsterdam kind of broke a sweat for a little while
there …
And the coolest thing yet? We’re not done with this trip.