Introduction

Many people maintain what is called a Bucketlist, a list of things they wish to do before they kick the bucket. I have one also, and although most of the items on that list are earthly and more on the emotional realm, and private, there is one specific thing that I have wanted to do ever since I first started working in IT in 1998: Peer at the Amsterdam Internet Exchange.

This post details striking this particular item off my bucketlist. It’s both indulgent, humblebraggy and incredibly nerdy and it talks a bit about mental health. If those are trigger words for you, skip ahead to another post, like my series on VPP ;-)

1998 - Netherlands

The Kelvinstraat

I started working when I was still at the TU/Eindhoven, and after a great sysadmin job at Radar Internet, which became Track and was sold to Wegener Arcade, I turned towards networking. After building Freeler (the first free ISP in the Netherlands) with Adrianus and co, and a small stint at their primary uplink Intouch with Rager (rest in peace, Brother), I joined BIT (AS12859) from 2000 to 2006, and it was here where I developed a true passion for that which makes the internet ‘tick’: routing protocols.

I was secretly jealous that BIT could afford Junipers, F5 loadbalancers and large Cisco switches, and I loved working with and on those machines. BIT had a reseller relationship with BBNed, and were able to directly connect ADSL modems into their own infrastructure, and as such I could afford to get myself a subnet from 213.154.224.0/19 routed to my house in Wageningen. It was where I had a half-19” rack in a clothing closet in our guest bedroom, and it was there that I decided: I want to eventually participate in the BGP world and peer at AMS-IX (the only exchange at the time, NLIX was just starting up, thanks again, Jan!).

Pictured to the right was my first contribution to AS12859 - deploying a CWDM ring from Ede to Amsterdam and upgrading our backbone from an ATM E3 (34Mbit) and POS STM1 (155Mbit) leased line to Gigabit Ethernet on Juniper M5 routers, this was in 2001, 20 years ago almost to the month.

2008 - Switzerland

The Cavern

Fast forward to 2006, I moved to Switzerland and while I remained friendly with NLNOG and SWINOG (and a few other network operator groups), I did not pursue the whole internet exchange thing. I had operated networks for the greater part of a decade, and with my full time job, I spent a lot of time learning how to be a good Site Reliability Engineer. I still had three /24 PI space blocks, used for different purposes in the past, but I was much more comfortable letting the “real” ISPs announce them - in my case AS25091 IP-Max (thanks, Fred!) and AS13030 Init7 (thanks, Fredy!) and AS12859 BIT (thanks, Michel!). I cannot remember any meaningful downtime in any of those operators, of course there is always some, but due to the N+2 nature of my network deployment, I don’t think any global downtime for my internet presence has ever occured.

It’s not a coincidence that even Google for the longest time used my website at SixXS for their own monitoring, now that is cool. Although Jeroen and I did decide to retire the SixXS project (see my Sunset article on why), the website is still up and served off of three distinct networks, because I have to stay true to the SRE life.

Pictured to the right was one of the two racks at Deltalis DK2, a datacenter built into a mountain in the heart of the swiss Alps. Classic edge/core/border approach with (at the time) state of the art Cisco 7600 routers. One of these is destined to become my nightstand at some point, this was in 2013, which is now (almost) 10 years ago.

Corona Motivation

My buddy Fred from IP-Max would regularly ask me “why don’t you just announce your /24 yourself?” It’d be fun, he said. In 2007, we registered a /24 PI for SixXS, and I was always quite content to let him handle the routing. But it started to itch and a neighbor of mine inadvertently reminded me of this itch (thanks, Max) by asking me if I was interested to share an L2 ethernet link with him from our place in Brüttisellen to one of the datacenters in Zürich, a distance of about 7km as the photons fly.

The Virus

I could not resist any longer. I was working long(er) than average hours due to the work-from-home situation: you easily chop off 45-60min of commute each day, but I noticed myself spending it in more meetings instead of in the train. I was slowly getting into a bad state, and my motivation was very low. I wanted to do something other than sleep-eat-work-sleep and even my jogging went to an all time minimum. I had very low emotional energy.

To put my mind off of things, I decided to reattach to my networking roots in a few ways: one was to build an AS and operate it for a while (maybe a few years until I get bored of it, and then re-parent my IP space to some friendly ISP, or who knows, cash in rich and sell my IP space to the highest bidder!), and the other was to continue my desire to have a competent replacement for silicon now that CPUs-of-now are just as fast as ASICs-of-then, and contribute to DANOS and VPP.

Step 1. Build a basement ISP

So getting a PC with Bird, or in my case, an appliance called DANOS which uses DPDK to implement wirespeed routing on commodity x86/64 hardware. So I happily announced my /24 and /48 from NTT’s datacenter, connected to the local internet exchange Swissix and rented an L2 circuit to my house via Openfactory. Also, I showed that a simple Supermicro (for example SYS-5018D-FN8T) could easily handle line rate 64 byte frames in both directions on its TenGigabit interfaces, that’s 29Mpps, and still have a responsive IPMI serial console. It reminded me of the early days of Juniper martini class routers, where Jean would say “.. and the chassis doesn’t even get warm”. That’s certainly correct today, cuz that Supermicro draws 35W, which is one microwatt per packet routed!

Step 2. Build a European Ring

Staging Amsterdam

Of course, I cannot end there, as I have a bucketlist item to work towards. I always wanted to peer in Amsterdam, ever since 2001 when I joined BIT. So I worked out a plan with Fred, who has also been wanting to go to Amsterdam with his Swiss ISP IP-Max.

So, in a really epic roadtrip full of nerd, Fred and I went into total geek-mode as we traveled to several European cities to deploy AS50869 on a european ring. I wrote about my experience extensively in these blog posts:

I think we can now say that I’m peering on the FLAP. It’s not that this AS50869 carries that much traffic, but it’s a very welcome relief of daily worklife to be able to do something fun and immediately rewarding like turn up a BGP session and see the traffic go from Zurich to any one of these cities at 10Gbit in any direction. No congestion, no packetlo, just pure horsepower performance.

Step 3. Build Linux CP in VPP

Next month, I plan to take VPP out for an elaborate spin. I’ve been running DANOS on my routers for a while now, and I’m pretty happy with it, but there are a few quirks that are annoying me more and more. Notably, the conversion of Vyatta style commands in the configuration into an FRR config, are often lossy. There’s a few key features (such as RPKI or LDP signalling for MPLS paths) that I’m missing, and the dataplane, although pretty stable, has crashed maybe three or four times over the last year. Note: One of IP-Max’s many Cisco ASR9k also had a few line card reboots in the last year so maybe these crashes are par for the course.

Ever since seeing Netgate and Cisco started work on the Linux Control Plane plugin, which takes interfaces in the VPP dataplane and exposes those as TAP interfaces in Linux, I’ve wanted to contribute to that. I’ve been determined to make use of VPP+LinuxCP in my own network. However, development has completely stalled on the plugin; the one that ships with VPP 21.06 is rudimentary at best: doesn’t do QinQ/QinAD; doesn’t apply any changes from the dataplane into the Linux network interface; and the plugin that mirrors netlink message has been stuck in limbo for a few months. So I reached out to the authors in May and offered to complete / rewrite the plugins. I find that writing code, compiling and testing it, and being able to immediately see the improvements in a live network incredibly motivating and energizing.

Expect to see a few posts in August/September about this work!

2021 - Switzerland

Alpine Health

I can say that after making a few small tweaks and adjustments, and breaking the WFH regime into “work” from home and “play” from home, helps a lot. I now have a HDMI switch that flips my desk from my work Mac into my personal OpenBSD machine, and a 19” rack in my basement with equipment to loadtest and develop VPP, and I often do some small chores like establish a peering session and happily traceroute from my basement to Amsterdam.

I’ve spent some time in the mountains, in a family commitment to go to a new swiss canton every month. The picture on the right was taken from First in Grindelwald, looking south towards Eiger and Mönch. I live in an absolutely beautiful country. Thanks, Switzerland ;-)

On the Bucketlist front, I have the following to report. I waited a few months before writing the post, but I can confidently say that accomplishing this L2/L3 path from my workstation in Brüttisellen where I’m typing this blogpost, all the way over Frankfurt to Amsterdam and being able to reach my original colocation machine at AS8283 Coloclue using only switches, routers and IP addresses I own is a continual joy. Seeing that my work now affords me a straight gigabit bandwidth in each direction, makes me just fill with engineering pride and happiness.

pim@chumbucket:~$ traceroute ghoul.ipng.nl
traceroute to gripe.ipng.nl (94.142.241.186), 30 hops max, 60 byte packets
 1  chbtl0.ipng.ch (194.1.163.66)  0.236 ms  0.178 ms  0.143 ms
 2  chrma0.ipng.ch (194.1.163.17)  1.394 ms  1.363 ms  1.332 ms
 3  defra0.ipng.ch (194.1.163.25)  7.275 ms  7.362 ms  7.213 ms
 4  nlams0.ipng.ch (194.1.163.27)  12.905 ms  12.843 ms  12.844 ms
 5  ghoul.ipng.nl (94.142.244.54)  13.120 ms  13.181 ms  13.044 ms

And as far as the actual bucketlist item goes, although I made a bit harder on myself because I moved to Switzerland, IP-Max also made it easier by giving me a great price on the backhaul connectivity to Amsterdam, so I can report that the bucket list item is indeed checked off the list:

pim@nlams0:~$ show protocols bgp address-family ipv6 unicast summary

IPv6 Unicast Summary:
BGP table version 689670802
RIB entries 251402, using 46 MiB of memory
Peers 67, using 1427 KiB of memory
Peer groups 32, using 2048 bytes of memory

Neighbor                 V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
2a02:1668:a2b:5:869::1   4      51088   1561576    216485        0    0    0 08w4d03h       126136        5
2a02:1668:a2b:5:869::2   4      51088   1546990    216485        0    0    0 08w4d03h       126127        5
2a02:898::d1             4       8283 812846953    127814        0    0    0 08w6d20h       130590        6
2a02:898::d2             4       8283 828908332    127814        0    0    0 08w0d16h       130590        6
2a02:898:146::2          4        112    101560 228562328        0    0    0 06w2d15h            2   132437
2a07:cd40:1::4           4     212855    105513 238069267        0    0    0 2d14h12m            1   132437
2602:fed2:fff:ffff::1    4     137933   4180058    124978        0    0    0 04w4d10h          551        7
2602:fed2:fff:ffff::253  4     209762   2034724    125048        0    0    0 1d00h14m          618        7
2001:7f8:10f::205b:140   4       8283    137242    121460        0    0    0 08w5d17h           34        7
2001:7f8:10f::207b:145   4       8315    278651    274793        0    0    0 06w0d12h           34        7
2001:7f8:10f::500f:139   4      20495    117590    107877        0    0    0 04w3d00h          208        7
2001:7f8:10f::ac47:131   4      44103    152949     55010        0    0    0 05w1d13h           24        7
2001:7f8:10f::af36:129   4      44854    134969    146240        0    0    0 09w2d16h            1        7
2001:7f8:10f::afd1:133   4      45009     35438     35477        0    0    0 01w0d02h            3        7
2001:7f8:10f::e20a:148   4      57866    302505    280603        0    0    0 05w5d18h          161        7
2001:7f8:10f::e3bb:137   4      58299   1419455    104321        0    0    0 04w0d13h          531        7
2001:7f8:10f::ec8d:132   4      60557    120509    108071        0    0    0 01w4d20h            7        7
2001:7f8:10f::3:259e:143 4     206238    278960    272776        0    0    0 04w4d18h            2        7
2001:7f8:10f::3:3e9b:134 4     212635    823944    140075        0    0    0 08w5d17h            1        7
2001:7f8:10f::dc49:253   4      56393   5693179    157171        0    0    0 02w6d22h        26680        7
2001:7f8:10f::dc49:254   4      56393   5698910    162197        0    0    0 08w5d17h        26680        7
2a02:2528:1902::1        4      25091   9964126    137696        0    0    0 09w1d22h       113020        5
2001:7f8:8f::a500:6939:1 4       6939   8496149    138188        0    0    0 01w2d20h        48079        7
2001:7f8:8f::a500:8283:1 4       8283     23251     52823        0    0    0 03w3d02h       Active        0
2001:7f8:8f::a501:3335:1 4      13335      3279      3199        0    0    0 1d02h35m          102        7
2001:7f8:8f::a502:495:1  4      20495    117248    107466        0    0    0 04w3d00h          208        7
2001:7f8:8f::a503:2934:1 4      32934    194428    193990        0    0    0 01w3d08h           30        7
2001:7f8:8f::a503:2934:2 4      32934    194035    194002        0    0    0 03w3d11h           30        7
2001:7f8:8f::a504:4854:1 4      44854         0      9052        0    0    0    never Idle (Admin)        0
2001:7f8:8f::a504:5009:1 4      45009     35433     35467        0    0    0 01w0d02h            3        7
2001:7f8:8f::a505:7866:1 4      57866    302602    276459        0    0    0 04w4d01h          161        7
2001:7f8:8f::a505:8299:1 4      58299    912125    141718        0    0    0 04w0d13h          531        7
2001:7f8:8f::a506:557:1  4      60557    120482    108067        0    0    0 01w4d20h            7        7
2001:7f8:8f::a521:2635:1 4     212635    622475     85332        0    0    0 02w5d10h            1        7
2001:7f8:8f::a504:9917:1 4      49917   8370930    158851        0    0    0 03w4d13h        25257        7
2001:7f8:8f::a504:9917:2 4      49917   8397150    160118        0    0    0 04w4d01h        25011        7
2001:7f8:13::a500:714:1  4        714     67722     66645        0    0    0 03w2d03h          146        7
2001:7f8:13::a500:714:2  4        714     68208     66645        0    0    0 03w2d03h          146        7
2001:7f8:13::a500:6939:1 4       6939  10980475     98099        0    0    0 07w0d10h        48079        7
2001:7f8:13::a502:495:1  4      20495    117773    107873        0    0    0 04w0d14h          208        7
2001:7f8:13::a503:4307:1 4      34307  10709086    100814        0    0    0 09w4d23h        23339        7
2001:7f8:13::a503:4307:2 4      34307  10694266    100814        0    0    0 09w4d23h        22137        7
2001:7f8:8f::a504:4103:1 4      44103    152932     55010        0    0    0 05w1d13h           24        7
2001:7f8:b7::a500:8283:1 4       8283    126035     98846        0    0    0 06w4d22h           34        7
2001:7f8:b7::a501:3335:1 4      13335      4277      4157        0    0    0 1d10h34m          102        7
2001:7f8:b7::a502:495:1  4      20495    117588    107871        0    0    0 04w3d00h          208        7
2001:7f8:b7::a504:5009:1 4      45009     35441     35504        0    0    0 01w0d02h            3        7
2001:7f8:b7::a506:557:1  4      60557    120546    108067        0    0    0 01w4d20h            7        7
2001:7f8:b7::a521:2635:1 4     212635    716031     94458        0    0    0 08w5d17h            1        7
2001:7f8:b7::a504:1441:1 4      41441  12911969    107363        0    0    0 08w2d12h        50606        7
2001:7f8:b7::a504:1441:2 4      41441  12733337    107304        0    0    0 08w2d12h        50606        7

Total number of neighbors 67

pim@nlams0:~$ show protocols ospfv3 neighbor 
Neighbor ID     Pri    DeadTime    State/IfState         Duration I/F[State]
194.1.163.7       1    00:00:32     Full/PointToPoint 62d21:41:24 dp0p6s0f3.100[PointToPoint]
194.1.163.34      1    00:00:39     Full/PointToPoint 27d22:28:30 dp0p6s0f3.200[PointToPoint]

There are three full IPv4 and IPv6 transit providers: AS51088 (A2B Internet, thanks Erik!), AS8283 (Coloclue) and AS25091 (IP-Max, thanks Fred!). Also, the router is connected directly to Speed-IX, LSIX, FrysIX and NL-IX. Along with the many other internet exchanges I’ve connected to, it puts my humble AS50869 as #5 best connected ISP in Switzerland!

I mean, just look at that stability, BGP sessions often times up as long as the machine has been there (remember, I deployed nlams0.ipng.ch only in May, so 9 weeks is all we can ask for!). OSPF uptime (helpfully shown with duration with OSPFv3 on FRR) is impeccable as well. The link with 27d of uptime is because I took out that router for maintenance 27 days ago to upgrade it to a preliminary version of DANOS + Bird2, as I prepare the move to VPP + Bird2 later this year.

A note on mental health

Mental health includes our emotional, psychological, and social well-being. It affects how we think, feel, and act. It also helps determine how we handle stress, relate to others, and make choices. Mental health is important at every stage of life, from childhood and adolescence through adulthood.

If you’ve read so far, thanks! I can imagine that some find this story a mixture of nerd and brag, and that’s OK. I am writing these stories because I find happiness in writing about the small and large technical things that I perceive as important to my feelings of accomplishment and therefor my wellbeing.

I do many non-nerd and non-technical things, but I try to make it a habit of keeping my personal life off the internet (I’m not on social media and not often on digital messaging boards or chat apps). I could tell you equally enthusiastically about those hikes I took in Grindelwald, or those Bürli I baked, but that would have to be in person.

Well-being is a positive outcome that is meaningful for people and for many sectors of society, because it tells us that people perceive that their lives are going well. However, many indicators that measure living conditions fail to measure what people think and feel about their lives, such as the quality of their relationships, their positive emotions and resilience, the realization of their potential, or their overall satisfaction with life.

I find satisfaction in my modest dabbles with IPng Networks, both the software and the hardware and physical aspects of it. I encourage everybody to have a safe/fun place where they spend some meaningful time doing things that spark joy. To your health!