Introduction
Many people maintain what is called a Bucketlist, a list of things they wish to do before they kick the bucket. I have one also, and although most of the items on that list are earthly and more on the emotional realm, and private, there is one specific thing that I have wanted to do ever since I first started working in IT in 1998: Peer at the Amsterdam Internet Exchange.
This post details striking this particular item off my bucketlist. It’s both indulgent, humblebraggy and incredibly nerdy and it talks a bit about mental health. If those are trigger words for you, skip ahead to another post, like my series on VPP ;-)
1998 - Netherlands
I started working when I was still at the TU/Eindhoven, and after a great sysadmin job at Radar Internet, which became Track and was sold to Wegener Arcade, I turned towards networking. After building Freeler (the first free ISP in the Netherlands) with Adrianus and co, and a small stint at their primary uplink Intouch with Rager (rest in peace, Brother), I joined BIT (AS12859) from 2000 to 2006, and it was here where I developed a true passion for that which makes the internet ’tick’: routing protocols.
I was secretly jealous that BIT could afford Junipers, F5 loadbalancers and large Cisco switches, and I loved working with and on those machines. BIT had a reseller relationship with BBNed, and were able to directly connect ADSL modems into their own infrastructure, and as such I could afford to get myself a subnet from 213.154.224.0/19 routed to my house in Wageningen. It was where I had a half-19" rack in a clothing closet in our guest bedroom, and it was there that I decided: I want to eventually participate in the BGP world and peer at AMS-IX (the only exchange at the time, NLIX was just starting up, thanks again, Jan!).
Pictured to the right was my first contribution to AS12859 - deploying a CWDM ring from Ede to Amsterdam and upgrading our backbone from an ATM E3 (34Mbit) and POS STM1 (155Mbit) leased line to Gigabit Ethernet on Juniper M5 routers, this was in 2001, 20 years ago almost to the month.
2008 - Switzerland
Fast forward to 2006, I moved to Switzerland and while I remained friendly with NLNOG and SWINOG (and a few other network operator groups), I did not pursue the whole internet exchange thing. I had operated networks for the greater part of a decade, and with my full time job, I spent a lot of time learning how to be a good Site Reliability Engineer. I still had three /24 PI space blocks, used for different purposes in the past, but I was much more comfortable letting the “real” ISPs announce them - in my case AS25091 IP-Max (thanks, Fred!) and AS13030 Init7 (thanks, Fredy!) and AS12859 BIT (thanks, Michel!). I cannot remember any meaningful downtime in any of those operators, of course there is always some, but due to the N+2 nature of my network deployment, I don’t think any global downtime for my internet presence has ever occured.
It’s not a coincidence that even Google for the longest time used my website at SixXS for their own monitoring, now that is cool. Although Jeroen and I did decide to retire the SixXS project (see my Sunset article on why), the website is still up and served off of three distinct networks, because I have to stay true to the SRE life.
Pictured to the right was one of the two racks at Deltalis DK2, a datacenter built into a mountain in the heart of the swiss Alps. Classic edge/core/border approach with (at the time) state of the art Cisco 7600 routers. One of these is destined to become my nightstand at some point, this was in 2013, which is now (almost) 10 years ago.
Corona Motivation
My buddy Fred from IP-Max would regularly ask me “why don’t you just announce your /24 yourself?” It’d be fun, he said. In 2007, we registered a /24 PI for SixXS, and I was always quite content to let him handle the routing. But it started to itch and a neighbor of mine inadvertently reminded me of this itch (thanks, Max) by asking me if I was interested to share an L2 ethernet link with him from our place in Brüttisellen to one of the datacenters in Zürich, a distance of about 7km as the photons fly.
I could not resist any longer. I was working long(er) than average hours due to the work-from-home situation: you easily chop off 45-60min of commute each day, but I noticed myself spending it in more meetings instead of in the train. I was slowly getting into a bad state, and my motivation was very low. I wanted to do something other than sleep-eat-work-sleep and even my jogging went to an all time minimum. I had very low emotional energy.
To put my mind off of things, I decided to reattach to my networking roots in a few ways: one was to build an AS and operate it for a while (maybe a few years until I get bored of it, and then re-parent my IP space to some friendly ISP, or who knows, cash in rich and sell my IP space to the highest bidder!), and the other was to continue my desire to have a competent replacement for silicon now that CPUs-of-now are just as fast as ASICs-of-then, and contribute to DANOS and VPP.
Step 1. Build a basement ISP
So getting a PC with Bird, or in my case, an appliance called DANOS which uses DPDK to implement wirespeed routing on commodity x86/64 hardware. So I happily announced my /24 and /48 from NTT’s datacenter, connected to the local internet exchange Swissix and rented an L2 circuit to my house via Openfactory. Also, I showed that a simple Supermicro (for example SYS-5018D-FN8T) could easily handle line rate 64 byte frames in both directions on its TenGigabit interfaces, that’s 29Mpps, and still have a responsive IPMI serial console. It reminded me of the early days of Juniper martini class routers, where Jean would say “.. and the chassis doesn’t even get warm”. That’s certainly correct today, cuz that Supermicro draws 35W, which is one microwatt per packet routed!
Step 2. Build a European Ring
Of course, I cannot end there, as I have a bucketlist item to work towards. I always wanted to peer in Amsterdam, ever since 2001 when I joined BIT. So I worked out a plan with Fred, who has also been wanting to go to Amsterdam with his Swiss ISP IP-Max.
So, in a really epic roadtrip full of nerd, Fred and I went into total geek-mode as we traveled to several European cities to deploy AS50869 on a european ring. I wrote about my experience extensively in these blog posts:
- Frankfurt: May 17th 2021.
- Amsterdam: May 26th 2021.
- Lille: May 28th 2021.
- Paris: June 1st 2021.
- Geneva: July 3rd 2021.
I think we can now say that I’m peering on the FLAP. It’s not that this AS50869 carries that much traffic, but it’s a very welcome relief of daily worklife to be able to do something fun and immediately rewarding like turn up a BGP session and see the traffic go from Zurich to any one of these cities at 10Gbit in any direction. No congestion, no packetlo, just pure horsepower performance.
Step 3. Build Linux CP in VPP
Next month, I plan to take VPP out for an elaborate spin. I’ve been running DANOS on my routers for a while now, and I’m pretty happy with it, but there are a few quirks that are annoying me more and more. Notably, the conversion of Vyatta style commands in the configuration into an FRR config, are often lossy. There’s a few key features (such as RPKI or LDP signalling for MPLS paths) that I’m missing, and the dataplane, although pretty stable, has crashed maybe three or four times over the last year. Note: One of IP-Max’s many Cisco ASR9k also had a few line card reboots in the last year so maybe these crashes are par for the course.
Ever since seeing Netgate and Cisco started work on the Linux Control Plane plugin, which takes interfaces in the VPP dataplane and exposes those as TAP interfaces in Linux, I’ve wanted to contribute to that. I’ve been determined to make use of VPP+LinuxCP in my own network. However, development has completely stalled on the plugin; the one that ships with VPP 21.06 is rudimentary at best: doesn’t do QinQ/QinAD; doesn’t apply any changes from the dataplane into the Linux network interface; and the plugin that mirrors netlink message has been stuck in limbo for a few months. So I reached out to the authors in May and offered to complete / rewrite the plugins. I find that writing code, compiling and testing it, and being able to immediately see the improvements in a live network incredibly motivating and energizing.
Expect to see a few posts in August/September about this work!
2021 - Switzerland
I can say that after making a few small tweaks and adjustments, and breaking the WFH regime into “work” from home and “play” from home, helps a lot. I now have a HDMI switch that flips my desk from my work Mac into my personal OpenBSD machine, and a 19" rack in my basement with equipment to loadtest and develop VPP, and I often do some small chores like establish a peering session and happily traceroute from my basement to Amsterdam.
I’ve spent some time in the mountains, in a family commitment to go to a new swiss canton every month. The picture on the right was taken from First in Grindelwald, looking south towards Eiger and Mönch. I live in an absolutely beautiful country. Thanks, Switzerland ;-)
On the Bucketlist front, I have the following to report. I waited a few months before writing the post, but I can confidently say that accomplishing this L2/L3 path from my workstation in Brüttisellen where I’m typing this blogpost, all the way over Frankfurt to Amsterdam and being able to reach my original colocation machine at AS8283 Coloclue using only switches, routers and IP addresses I own is a continual joy. Seeing that my work now affords me a straight gigabit bandwidth in each direction, makes me just fill with engineering pride and happiness.
pim@chumbucket:~$ traceroute ghoul.ipng.nl
traceroute to gripe.ipng.nl (94.142.241.186), 30 hops max, 60 byte packets
1 chbtl0.ipng.ch (194.1.163.66) 0.236 ms 0.178 ms 0.143 ms
2 chrma0.ipng.ch (194.1.163.17) 1.394 ms 1.363 ms 1.332 ms
3 defra0.ipng.ch (194.1.163.25) 7.275 ms 7.362 ms 7.213 ms
4 nlams0.ipng.ch (194.1.163.27) 12.905 ms 12.843 ms 12.844 ms
5 ghoul.ipng.nl (94.142.244.54) 13.120 ms 13.181 ms 13.044 ms
And as far as the actual bucketlist item goes, although I made a bit harder on myself because I moved to Switzerland, IP-Max also made it easier by giving me a great price on the backhaul connectivity to Amsterdam, so I can report that the bucket list item is indeed checked off the list:
pim@nlams0:~$ show protocols bgp address-family ipv6 unicast summary
IPv6 Unicast Summary:
BGP table version 689670802
RIB entries 251402, using 46 MiB of memory
Peers 67, using 1427 KiB of memory
Peer groups 32, using 2048 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
2a02:1668:a2b:5:869::1 4 51088 1561576 216485 0 0 0 08w4d03h 126136 5
2a02:1668:a2b:5:869::2 4 51088 1546990 216485 0 0 0 08w4d03h 126127 5
2a02:898::d1 4 8283 812846953 127814 0 0 0 08w6d20h 130590 6
2a02:898::d2 4 8283 828908332 127814 0 0 0 08w0d16h 130590 6
2a02:898:146::2 4 112 101560 228562328 0 0 0 06w2d15h 2 132437
2a07:cd40:1::4 4 212855 105513 238069267 0 0 0 2d14h12m 1 132437
2602:fed2:fff:ffff::1 4 137933 4180058 124978 0 0 0 04w4d10h 551 7
2602:fed2:fff:ffff::253 4 209762 2034724 125048 0 0 0 1d00h14m 618 7
2001:7f8:10f::205b:140 4 8283 137242 121460 0 0 0 08w5d17h 34 7
2001:7f8:10f::207b:145 4 8315 278651 274793 0 0 0 06w0d12h 34 7
2001:7f8:10f::500f:139 4 20495 117590 107877 0 0 0 04w3d00h 208 7
2001:7f8:10f::ac47:131 4 44103 152949 55010 0 0 0 05w1d13h 24 7
2001:7f8:10f::af36:129 4 44854 134969 146240 0 0 0 09w2d16h 1 7
2001:7f8:10f::afd1:133 4 45009 35438 35477 0 0 0 01w0d02h 3 7
2001:7f8:10f::e20a:148 4 57866 302505 280603 0 0 0 05w5d18h 161 7
2001:7f8:10f::e3bb:137 4 58299 1419455 104321 0 0 0 04w0d13h 531 7
2001:7f8:10f::ec8d:132 4 60557 120509 108071 0 0 0 01w4d20h 7 7
2001:7f8:10f::3:259e:143 4 206238 278960 272776 0 0 0 04w4d18h 2 7
2001:7f8:10f::3:3e9b:134 4 212635 823944 140075 0 0 0 08w5d17h 1 7
2001:7f8:10f::dc49:253 4 56393 5693179 157171 0 0 0 02w6d22h 26680 7
2001:7f8:10f::dc49:254 4 56393 5698910 162197 0 0 0 08w5d17h 26680 7
2a02:2528:1902::1 4 25091 9964126 137696 0 0 0 09w1d22h 113020 5
2001:7f8:8f::a500:6939:1 4 6939 8496149 138188 0 0 0 01w2d20h 48079 7
2001:7f8:8f::a500:8283:1 4 8283 23251 52823 0 0 0 03w3d02h Active 0
2001:7f8:8f::a501:3335:1 4 13335 3279 3199 0 0 0 1d02h35m 102 7
2001:7f8:8f::a502:495:1 4 20495 117248 107466 0 0 0 04w3d00h 208 7
2001:7f8:8f::a503:2934:1 4 32934 194428 193990 0 0 0 01w3d08h 30 7
2001:7f8:8f::a503:2934:2 4 32934 194035 194002 0 0 0 03w3d11h 30 7
2001:7f8:8f::a504:4854:1 4 44854 0 9052 0 0 0 never Idle (Admin) 0
2001:7f8:8f::a504:5009:1 4 45009 35433 35467 0 0 0 01w0d02h 3 7
2001:7f8:8f::a505:7866:1 4 57866 302602 276459 0 0 0 04w4d01h 161 7
2001:7f8:8f::a505:8299:1 4 58299 912125 141718 0 0 0 04w0d13h 531 7
2001:7f8:8f::a506:557:1 4 60557 120482 108067 0 0 0 01w4d20h 7 7
2001:7f8:8f::a521:2635:1 4 212635 622475 85332 0 0 0 02w5d10h 1 7
2001:7f8:8f::a504:9917:1 4 49917 8370930 158851 0 0 0 03w4d13h 25257 7
2001:7f8:8f::a504:9917:2 4 49917 8397150 160118 0 0 0 04w4d01h 25011 7
2001:7f8:13::a500:714:1 4 714 67722 66645 0 0 0 03w2d03h 146 7
2001:7f8:13::a500:714:2 4 714 68208 66645 0 0 0 03w2d03h 146 7
2001:7f8:13::a500:6939:1 4 6939 10980475 98099 0 0 0 07w0d10h 48079 7
2001:7f8:13::a502:495:1 4 20495 117773 107873 0 0 0 04w0d14h 208 7
2001:7f8:13::a503:4307:1 4 34307 10709086 100814 0 0 0 09w4d23h 23339 7
2001:7f8:13::a503:4307:2 4 34307 10694266 100814 0 0 0 09w4d23h 22137 7
2001:7f8:8f::a504:4103:1 4 44103 152932 55010 0 0 0 05w1d13h 24 7
2001:7f8:b7::a500:8283:1 4 8283 126035 98846 0 0 0 06w4d22h 34 7
2001:7f8:b7::a501:3335:1 4 13335 4277 4157 0 0 0 1d10h34m 102 7
2001:7f8:b7::a502:495:1 4 20495 117588 107871 0 0 0 04w3d00h 208 7
2001:7f8:b7::a504:5009:1 4 45009 35441 35504 0 0 0 01w0d02h 3 7
2001:7f8:b7::a506:557:1 4 60557 120546 108067 0 0 0 01w4d20h 7 7
2001:7f8:b7::a521:2635:1 4 212635 716031 94458 0 0 0 08w5d17h 1 7
2001:7f8:b7::a504:1441:1 4 41441 12911969 107363 0 0 0 08w2d12h 50606 7
2001:7f8:b7::a504:1441:2 4 41441 12733337 107304 0 0 0 08w2d12h 50606 7
Total number of neighbors 67
pim@nlams0:~$ show protocols ospfv3 neighbor
Neighbor ID Pri DeadTime State/IfState Duration I/F[State]
194.1.163.7 1 00:00:32 Full/PointToPoint 62d21:41:24 dp0p6s0f3.100[PointToPoint]
194.1.163.34 1 00:00:39 Full/PointToPoint 27d22:28:30 dp0p6s0f3.200[PointToPoint]
There are three full IPv4 and IPv6 transit providers: AS51088 (A2B Internet, thanks Erik!), AS8283 (Coloclue) and AS25091 (IP-Max, thanks Fred!). Also, the router is connected directly to Speed-IX, LSIX, FrysIX and NL-IX. Along with the many other internet exchanges I’ve connected to, it puts my humble AS50869 as #5 best connected ISP in Switzerland!
I mean, just look at that stability, BGP sessions often times up as long as the machine
has been there (remember, I deployed nlams0.ipng.ch only in May, so 9 weeks is all we can ask for!).
OSPF uptime (helpfully shown with duration with OSPFv3 on FRR) is impeccable as well. The link with 27d
of uptime is because I took out that router for maintenance 27 days ago to upgrade it to a preliminary
version of DANOS + Bird2, as I prepare the move to VPP + Bird2 later this year.
A note on mental health
Mental health includes our emotional, psychological, and social well-being. It affects how we think, feel, and act. It also helps determine how we handle stress, relate to others, and make choices. Mental health is important at every stage of life, from childhood and adolescence through adulthood.
If you’ve read so far, thanks! I can imagine that some find this story a mixture of nerd and brag, and that’s OK. I am writing these stories because I find happiness in writing about the small and large technical things that I perceive as important to my feelings of accomplishment and therefor my wellbeing.
I do many non-nerd and non-technical things, but I try to make it a habit of keeping my personal life off the internet (I’m not on social media and not often on digital messaging boards or chat apps). I could tell you equally enthusiastically about those hikes I took in Grindelwald, or those Bürli I baked, but that would have to be in person.
Well-being is a positive outcome that is meaningful for people and for many sectors of society, because it tells us that people perceive that their lives are going well. However, many indicators that measure living conditions fail to measure what people think and feel about their lives, such as the quality of their relationships, their positive emotions and resilience, the realization of their potential, or their overall satisfaction with life.
I find satisfaction in my modest dabbles with IPng Networks, both the software and the hardware and physical aspects of it. I encourage everybody to have a safe/fun place where they spend some meaningful time doing things that spark joy. To your health!