I’ve been planning a network expansion for a while now. For the next few weeks, I will be in total geek-mode as I travel to several European cities to deploy AS50869 on a european ring. At the same time, my buddy Fred from IP-Max has been wanting to go to Amsterdam. IP-Max’s network is considerably larger than mine, but it just never clicked with the right set of circumstances for them to deploy in the Netherlands, until the stars aligned …
First a word
I just wanted to start with a note on how special it is to partner with IP-Max and having known the founder Fred for so long affords me this specific trip. It must be known that building a 10G pan-european ring is not an easy thing to do, and I do appreciate very much the kindness Fred has shown IPng even though we are under contract – the ability to travel to every point of presence and get the founder’s tour in each place, get to know the local DC ops, sales directors, field technicians and local customers, is simply golden. Thank you.
Deployment
The city of love, corny as it sounds, also happens to have quite a bit of fibre noire, happily lit by hundreds of local and international carriers. There are two places in the dead-center of the city, a cabaret voltaire called Telehouse TH2, and a new facility from KDDI which is in the same physical block, aptly addressed as 65 Rue Léon Frot, 75011 Paris and that is where my beloved router frpar0.ipng.ch will be. Note that in Lille, actually I had to make do with frggh0 which was in the town of Sainghin-en-Mélantois. This one lives in Paris, none of that suburban bullshit. This is the real deal. There’s probably more connectivity in this one block than in all of the Paris metro combined, maybe even more than all of France combined – peut être :)
After having visited the older location, we took the router, APU, cables and optics to Léon Frot. The rack was quickly found, and it is obvious that this is the location: A fridge was awaiting us, and I reserved two Tengig ports on the ASR9010, one towards Lille and one towards Zürich (which will be Geneva later on).
I’m getting the hang of this VLL stuff after our adventures, previously in Lille at CIV, which is a lazy 4.8ms away from this place (Fred already speaks of making that more like 3.7ms with a small call to his buddy Laurent). So I went about my business, racking first the WiFi enabled console.par.ipng.nl, connecting to WiFi with it, and finding my router on frpar0.ipng.ch over IPMI serial-over-lan. Configuring one Tengig port on the Intel X710 towards er01.lil01.ip-max.net and another Tengig port towards er01.zrh56.ip-max.net.
I have two Supermicros on backorder, one of which will go to Lille to replace the Dell R610 that I placed temporarily in that site, and the other will go to Geneva. At that point, I will break up this VLL to become one from here to Geneva and another from Geneva back home to Zürich. Seriously though I will have to stop Fred’s enthusiasm because he also mentioned somthing about LyonIX and another small stopover of 1HE and 35W over there… this is addictive, save me from myself!!1
Connectivity on the FLAP
All week, Fred has been talking about the FLAP, which I know to be a term but I really never bothered to ask about it – it turns out, he explained, that it stands for Frankfurt, London, Amsterdam, and Paris. I’m not in London (yet, please don’t dare me …), however I can legitimately claim I am on the FLAP because I have a router in Lille. So there’s that :)
Fred has ordered my FranceIX connection this afternoon, delivered from a 20Gig LAG on er02.par02.ip-max.net and directly into my router there. In the mean time, I will be busy configuring my DE-CIX port from a previous post.
The console server here (a standard issue APU3 with 802.11ac WiFi broadcasting AS50869 PAR with password IPngGuest, you’re welcome), connects to the router with IPMI, while the router itself connects via USB serial back to the APU for maximum resilience.
A hard knock life
It was not without troubles today. When configuring my VLL to Zurich, I had
misconfigured the er01.zrh56.ip-max.net side (a Cisco 7600, which is on its
way out), and the VLL would not come up. I could see traffic going in one
direction but not in the other … which typically does not make OSPF adjacencies
happen. After about an hour of messing around, I puppydog-eyed to Fred who
proceeded to find my bug within 30 seconds: I needed to do some VLAN gymnastics
by adding rewrite ingress tag pop 1 symmetric and as well adding 4 bytes
to the MTU (so mtu 9018 total, cuz packets gotta be sourced directly from the
jumbo-jumbo club!).
But, Fred was miserable as well because he had updated a Xen hypervisor which
ended up not being able to boot because of a broken LVM configuration. So we
literally swapped laptops and while he fixed my VLL, I fixed his volume group
by running update-initramfs -u -k all from a recovery Debian USB stick. For
an extra bonus, here’s a picture of Happy Fred at the local Cafe Leopard, where
pretty much every day you can find a host of locals and international nerds who
have emerged from the server floor.
And Mael also helped with the serial port - I had put it into 115200 baud
but not pinned it at 8N1 for the APU serial console. It shot into life
as soon as he gave that tip and I committed the config, bravo!
I tend to believe it will not be necessary for me to physically visit the facility that often – simple hardware, no spinning disks, an APU connecting to IPMI for full HTML5 based KVM control and serial-over-lan, and the router exposing a console back to the APU, which has aan OOB network connection from AS25091. Yeah, I think I’ll be good.
The results
But this was a special one indeed, because up until now, my traceroutes kept on getting longer and longer as I deployed in Frankfurt, Amsterdam, and Lille. Deploying in Paris therefore looked like this initially, as the packets took, let us say, the scenic route to my basement:
pim@frpar0:~$ traceroute chumbucket.ipng.nl
traceroute to chumbucket.ipng.nl (194.1.163.93), 30 hops max, 60 byte packets
1 frggh0.ipng.ch (194.1.163.30) 4.915 ms 4.885 ms 4.866 ms
2 nlams0.ipng.ch (194.1.163.28) 12.396 ms 12.398 ms 12.382 ms
3 defra0.ipng.ch (194.1.163.26) 18.536 ms 18.520 ms 18.541 ms
4 chrma0.ipng.ch (194.1.163.24) 24.572 ms 24.557 ms 24.542 ms
5 chgtg0.ipng.ch (194.1.163.9) 24.549 ms 24.510 ms 24.517 ms
6 chbtl1.ipng.ch (194.1.163.18) 24.707 ms 25.114 ms 25.038 ms
7 chumbucket.ipng.nl (194.1.163.93) 25.320 ms 25.564 ms 25.452 ms
That’s quite the scenic route indeed. But! On this glorious day, at exactly 16:34 UTC, the Tengig european IPv4 and IPv6 ring was closed, with one final set of OSPF adjacencies:
pim@frpar0:~$ show protocols ospfv3 neighbor
Neighbor ID Pri DeadTime State/IfState Duration I/F[State]
194.1.163.34 1 00:00:37 Full/PointToPoint 01:40:51 dp0p6s0f0.100[PointToPoint]
194.1.163.1 1 00:00:33 Full/PointToPoint 00:01:28 dp0p6s0f1.100[PointToPoint]
Which allowed the ring to hone in on shortest path at its best - East bound to Frankfurt and Amsterdam, and West bound to Paris and Lille. Link and equipment failures will not bother me that much, OSPF and OSPFv3 will take care of rerouting me around network problems, which considering the ASR9k at IP-Max, I do think will be the exception:
pim@chumbucket:~$ traceroute frggh0.ipng.ch
traceroute to frggh0.ipng.ch (194.1.163.34), 30 hops max, 60 byte packets
1 chbtl1.ipng.ch (194.1.163.67) 0.317 ms 0.238 ms 0.190 ms
2 chgtg0.ipng.ch (194.1.163.19) 0.619 ms 0.574 ms 0.531 ms
3 frpar0.ipng.ch (194.1.163.40) 15.271 ms 15.226 ms 15.174 ms
4 frggh0.ipng.ch (194.1.163.34) 20.059 ms 20.020 ms 19.977 ms
pim@chumbucket:~$ traceroute nlams0.ipng.ch
traceroute to nlams0.ipng.ch (194.1.163.32), 30 hops max, 60 byte packets
1 chbtl1.ipng.ch (194.1.163.67) 0.345 ms 0.198 ms 0.284 ms
2 chgtg0.ipng.ch (194.1.163.19) 0.610 ms 0.518 ms 0.538 ms
3 chrma0.ipng.ch (194.1.163.8) 0.732 ms 0.750 ms 0.716 ms
4 defra0.ipng.ch (194.1.163.25) 6.835 ms 6.802 ms 6.767 ms
5 nlams0.ipng.ch (194.1.163.32) 12.799 ms 12.765 ms 12.731 ms
Of course with impeccable throughput, bien sûr:
pim@frpar0:~$ iperf3 -c chgtg0.ipng.ch -R … [ 5] 0.00-10.01 sec 11.2 GBytes 9.42 Gbits/sec 1 sender [ 5] 0.00-10.00 sec 11.2 GBytes 9.42 Gbits/sec receiver
I'm tired, but ultimately satisfied on taking my private AS50869 across
_the FLAP_, with a physical presence in each, and an IXP connection and
TenGig bidirectional, and TenGig IP Transit at each location. I think this
network is good to go for the next few years at least.