A Brief History
In January of 2003, my buddy Jeroen announced a project called the Ghost Route Hunters, after the industry had been plagued for a few years with anomalies in the DFZ - routes would show up with phantom BGP paths, unable to be traced down to a source or faulty implementation. Jeroen presented his findings at RIPE-46 and for years after this, the industry used the SixXS GRH as a distributed looking glass. At the time, one of SixXS’s point of presence providers kindly lent the project AS8298 to build this looking glass and underlying infrastructure.
After running SixXS for 16 years, Jeroen and I decided to Sunset it, which meant that in June of 2017, the Ghost Route Hunter project came to an end as well, and as we tore down the infrastructure, AS8298 became dormant.
Then in August of 2021, I was doing a little bit of cleaning on the IPng Networks serving infrastructure, and came across some old mail from RIPE NCC about that AS number. And while IPng Networks is running just fine on AS50869 today, it would be just that little bit cooler if it were to run on AS8298. So, I embarked on a journey to move a running ISP into a new AS number, which sounds like fun! This post describes the situation going in to this renumbering project, and there will be another post, likely in January 2022, that describes the retrospective (this future post may be either celebratory, or a huge postmortem, to be determined).
The Plan
Step 0. Acquire the AS
First off, I had to actually acquire the AS number. Back in the days (I’m speaking of 2002), RIPE NCC was a little bit less formal than it is today. As such, our loaned AS number was simply registered to SixXS, which is not a legal entity. Later, to do things properly, it was placed in Jeroen’s custody (by creating ORG-SIXX1-RIPE). But precisely because Jeroen nor SixXS was an LIR at that time, the previous holder (Easynet, with LIR de.easynet, later acquired by Sky, also called British Sky Broadcasting, trading under LIR uk.bskyb) became the sponsoring LIR. So I had to arrange two things:
- A Transfer Agreement between Jeroen and myself; that signaled his willingness to transfer the AS number to me. This is boilerplate stuff, and example contracts can be downloaded on the RIPE website.
- An agreement between Sky and IPng Networks; that signaled the transfer from sponsoring LIR to our LIR
ch.ipng. This is rather non-bureaucratic and a well traveled path; sponsoring LIRs and movements of resource between holders happen all the time.
With the permission of the previous holder, and with the help of the previous sponsoring LIR, the transfer itself was a matter of filing the correct paperwork at RIPE NCC, quoting the transfer agreement, and providing identification for the offering party (Jeroen) and the receiving party (Pim). And within a matter of a few days, the AS number was transfered to ORG-PVP9-RIPE, the ch.ipng LIR.
NOTE - In case you’re wondering, I registered the ch.ipng LIR a few months before I incorporated IPng Networks as a swiss limited liability company (called a GmbH, or Gesellschaft mit beschränkter Haftung in German), so for now I’m still trading as my natural person. RIPE has a cooldown period of two years before new LIRs can acquire/merge/rename. I do expect that some time in 2023 the PeeringDB page and bgp.he.net and friends to drop my personal name and take my company name :) For now, trading as Pim will have to do, slightly more business risk, but just as much fun!
Step 1. Split AS50869 into two networks
The autonomous system of IPng Networks spans two main parts. Firstly, in Zurich IPng Networks operates four sites and six routers:
- Two in a private colocation site at Daedalean (A) in Albisrieden called
ddln0andddln1, they are running DANOS - Two at our offices in Brüttisellen (C), called
chbtl0andchbtl1, they are running Debian - One at Interxion ZUR1 datacenter in Glattbrugg (D), called
chgtg0, running VPP, connecting to a public internet exchange CHIX-CH and taking transit from IP-Max and Openfactory. - One at NTT’s datacenter in Rümlang (E), called
chrma0, also running VPP, connecting to a public internet exchange SwissIX and taking transit from IP-Max and Meerfarbig.
NOTE: You can read a lot about my work on VPP in a series of VPP articles, please take a look!
There’s a few downstream IP Transit networks and lots of local connected networks, such as in the DDLN colo. Then, from chrma0, we connect to our european ring northwards (towards Frankfurt), and from chgtg0 we connect to our european ring south-westwards (towards Geneva).
That ring, then, consists of five additional sites and five routers, all running VPP:
- Frankfurt:
defra0, connecting to four DE-CIX exchangepoints in Frankfurt itself directly, and remotely to Munich, Düsseldorf and Hamburg - Amsterdam:
nlams0, connecting to NL-IX, SpeedIX, FrysIX (our favorite!), and LSIX; we also pick up two transit providers (A2B and Coloclue). - Lille:
frggh0, connecting to the northern france exchange called LillIX - Paris:
frpar0, connecting to two FranceIX exchange points, directly in Paris, and remotely to Marseille - Geneva:
chplo0, connecting to our very own Free IX
Every one of these sites has an upstream session with AS25091 (IP-Max). Considering these folks are organizationally very close to me, it is easy for me to rejigger any one of those sessions between AS50869 (current) and AS8298 (new). And considering AS8298 has been a member of our as-set AS-IPNG for a while, it’ll also be a natural propagation to rely on IP-Max, even if some peering sessions might be down.
So before I start, IPng Networks’ view from AS25091 in Amsterdam looks like this:
Network Next Hop Metric LocPrf Weight Path
* 92.119.38.0/24 46.20.242.210 0 50869 i
* 176.119.215.0/24 46.20.242.210 0 50869 60557 i
* 185.36.229.0/24 46.20.242.210 0 50869 212855 i
* 185.173.128.0/24 46.20.242.210 0 50869 57777 i
* 185.209.12.0/24 46.20.242.210 0 50869 212323 i
* 192.31.196.0/24 46.20.242.210 0 50869 112 i
* 192.175.48.0/24 46.20.242.210 0 50869 112 i
* 194.1.163.0/24 46.20.242.210 0 50869 i
* 194.126.235.0/24 46.20.242.210 0 50869 i
Network Next Hop Metric LocPrf Weight Path
* 2001:4:112::/48 2a02:2528:1902::210 0 50869 112 i
* 2001:678:3d4::/48 2a02:2528:1902::210 0 50869 201723 i
* 2001:678:ce4::/48 2a02:2528:1902::210 0 50869 60557 i
* 2001:678:ce8::/48 2a02:2528:1902::210 0 50869 60557 i
* 2001:678:cec::/48 2a02:2528:1902::210 0 50869 60557 i
* 2001:678:cf0::/48 2a02:2528:1902::210 0 50869 60557 i
* 2001:678:d78::/48 2a02:2528:1902::210 0 50869 i
* 2001:67c:6bc::/48 2a02:2528:1902::210 0 50869 201723 i
* 2620:4f:8000::/48 2a02:2528:1902::210 0 50869 112 i
* 2a07:cd40::/29 2a02:2528:1902::210 0 50869 212855 i
* 2a0b:dd80::/29 2a02:2528:1902::210 0 50869 i
* 2a0b:dd80::/32 2a02:2528:1902::210 0 50869 i
* 2a0d:8d06::/32 2a02:2528:1902::210 0 50869 60557 i
* 2a0e:fd40:200::/48 2a02:2528:1902::210 0 50869 60557 i
* 2a0e:fd45:da0::/48 2a02:2528:1902::210 0 50869 60557 i
* 2a10:d200::/29 2a02:2528:1902::210 0 50869 212323 i
* 2a10:fc40::/29 2a02:2528:1902::210 0 50869 57777 i
Step 2. Restrict the routers that originate our prefixes
As a preparation to actually starting to use AS8298, I’ll create RPKI records of authorization for all of our prefixes in both AS50869 and AS8298, and I’ll add route: objects for all of them in both as well.
Now, I’m ready to make my first networking topology change: instead of originating our prefixes in all routers, I will originate our prefixes in AS50869 only from two routers: chbtl0 and chbtl1. Nobody on the internet will notice this change, the as-path will remain ^50869$ for all prefixes.
I’ll also prepare the two routers to speak to eachother with an iBGP session (rather than to IPng Networks’ route-reflectors, which are still in AS50869).
Step 3. Convert Brüttisellen to AS8298
Now, I’ll switch these routers chbtl0 and chbtl1 out of AS50869 and into AS8298. The only damage at this point might be that my personal Spotify and Netflix stop working, and my family yells at me (but they do that all the time anyway, so it’s a wash…). If things go poorly, the backout plan is to switch back to AS50869 and return things to normal. But if things go well, from this point onwards everybody will see our own IPng Networks prefixes via as-path ^50869_8298$ and effectively, AS50869 will become a transit provider for AS8298, which will be singlehomed for the moment.
My buddy Max runs a small /29 and /64 exchangepoint in Brüttisellen with only three members on it - IPng Networks, Stucchinet and Openfactory. I will ask both of them to be my canary, and change their peering session from AS50869 to AS8298. If things go bad, that’s no worries, I can drop/disable these peering sessions as I have sessions with both as well in other places. But it’d be a good place to see and test if things work as expected.
Step 4. Convert DDLN to AS8298
Now that our IPv4 and IPv6 prefixes have moved and AS50869 does not originate prefixes anymore, you may wonder ‘what about the colo?’. Indeed, the colo runs at Daedalean behind ddln0 and ddln1, both of which are still in AS50869. However, the only way to ever be able to reach those prefixes, would be to find an entrypoint in AS50869 (as it is the only uplink of AS8298). All routers in AS50869 and AS8298 share an underlying OSPF and OSPFv3 interior gateway protocol, which means that if anything is destined to 194.1.163.0/24 or 2001:678:d78::/48, those packets will find their way to the correct location using the IGP. That’s neat, because it means that even though ddln0 is speaking BGP in AS50869, it will happily forward traffic to its more specific prefixes from AS8298.
Considering there’s only one IP transit network in DDLN, those two routers will be the first for me to convert. After converting them to AS8298, they will receive transit from AS50869 just like the ones in Brüttisellen. I’ll rejigger Jeroen’s AS57777 to receive transit directly from AS8298, and it will then be the first to transition. Jeroen’s prefixes will briefly become ^50869_8298_57777$, which will be the only change, but this will validate that, indeed, AS8298 can provide transit. Apart from the longer as-path, the physical path that IP packets take will remain the same because Jeroen’s network is currently cough singlehomed at DDLN.
Now that I have four routers in AS8298, I’ll add a iBGP sessions directly only amongst the pairs, rather than in a full mesh on these routers. I’ve now created two islands of AS8298, interconnected by AS50869. You’d think this is bad, but it’s actually a fine situation to be in and there will be no loss of service, because:
- we’ve already established that AS50869 has reachability to all more-specifics in AS8298 (Step 3)
- AS50869 has reachability to AS57777 via its downstream AS8298
- AS50869 is the only way in and out of the DDLN or Brüttisellen routers
Nevertheless, I’d rather swiftly continue with the next step, and reconnect these two islands. It’s is a good time for me to give a headsup to the larger internet exchanges (notably SwissIX) so folks can prepare for what’s coming. I think many NOC teams know how to establish/remove a peering session, but I do expect the ITIL-automated teams to not have a playbook for “peer on IP 192.0.2.1 has changed from AS666 to AS42”. I’ll observe their performance on this task and take notes, as there’s quite a few public IXPs to go…
Step 5. Convert Rümlang and Glattbrugg to AS8298
After four of our routers (two redundant pairs) have been operating in AS8298 for a few days, I’m ready to renumber our first machine connected to a public internet exchange: chgtg0 is connected to CHIX-CH. I’ll contact the members with whom IPng Networks has a direct peering, and of course the internet exchange folks, to ask them to renumber our AS50869 to AS8298. After restarting our router into the new AS, one by one I’ll establish sessions with our peers there - this is an important exercise because I’ll be doing this later in Step 5 on every peering/border router in the european ring, and there will be in total ~1'850 BGP adjacencies that have to be renumbered.
At CHIX-CH, IPng has in total four BGP sessions with the two routeservers in AS212100, and in total 38 direct BGP sessions; two of which are somewhat important: AS13030 (Init7) and AS15169 (Google), to which a large fraction of our traffic flows. While upgrading this router, I’ll also switch my one downstream network (Daedalean itself, operating AS212323) to receive transit from AS8298. Because I’ve canaried this with Jeroen’s AS57777 in the DDLN colo previously, I’ll be reasonably certain at this point that it’ll work well. If not, they have alternative uplinks (notably AS174), so they should be fine without me.
At SwissIX, IPng has as well four BGP sessions with the two routeservers in AS42476, and in total 132 direct BGP sessions. I think that, once these two peering routers are complete, I’ll checkpoint and let things run like this for a while. Let’s take a few weeks off, giving me a while to hunt down peers at SwissIX and CH-IX to catch up and re-establish their sessions with the new AS8298 :)
After this step, phase one of the transition is complete, and AS8298 (and its networks AS57777 and AS212323) will be directly visible in Switzerland, and still tucked away behind ^50869_8298_212323$ for the international traffic. I will however have 2 transit sessions from IP-Max (AS25091), 2 transit sessions from Openfactory (AS58299) and 1 transit session from Meerfarbig (AS34549); so it is expected to be a quite stable network at this point, which is good.
Step 6. Convert European Ring to AS8298
For this task, I’ll swap my iBGP sessions to all use the new AS8298. I do this by first dismantling rr0.chbtl0.ipng.ch and bring it back up as an iBGP speaker in the new AS; then one by one push all routers to speak to that route reflector (in addition to the existing route reflectors in AS50869). After that stabilizes, rinse and repeat with rr0.chplo0.ipng.ch; and finally finish the job with rr0.frggh0.ipng.ch. The two pairs of routers who were by themselves in AS8298 (chbtl0/chbtl1 in Brüttisellen; and ddln0/ddln1 at Daedalean) can now be reattached to the iBGP route-reflectors as well.
It will be a bit of a schlepp, but now comes the notification of all international peers (there are an additional ~250 direct peerings) and downstreams (there are three left: Raymon, Jelle and Krik), and upstreams (there are two additional ones: Coloclue, and A2B). While normally this is a matter of merely swapping the AS number, it has to be done on both sides - on my side, I can do it with a one-line change to the git repository, and it’ll be pushed by Kees (the network build and push automation that was inspired by Coloclue’s Kees), on the remote side it will be a matter of patience. One by one, folks will (or.. won’t) update their peering session. The only folks I’ll actively chase is the DE-CIX, FranceIX, NL-IX, LSIX and FrysIX routeserver operators, as the vast majority of adjacencies are learned via those. By means of a headsup one week in advance, and a few reminders on the day of, and the day after maintenance, I should minimize downtime. But, in this case, because I already have two transit providers in Switzerland (AS25091 IP-Max, and AS58299 Openfactory) who provide me full tables in AS8298, it should be operationally smooth sailing.
At the end of this exercise, the as-path will be ^8298$ for my own prefixes and ^8298_.*$ for my downstream networks, and AS50869 will no longer be in any as-path in the DFZ. Rolling back is tricky, although Bird can do individual peering sessions with differing AS numbers, I don’t think this is a good idea; as it’ll mean many (many) changes into the repository; so in the interest of simplicity and don’t break things that work, I will do them router-by-router rather than session-by-session; and send a few reminders to folks to update their records to match my peeringdb entries.
Step 7. Repurpose/retire AS50869
There’s really not that much to do – delete the route: and route6: objects and remove RPKI ROAs for the old announcments; but mostly it’ll be a matter of hunting down peering partners who have not (yet) updated their records and sessions. I imagine lots of folks will hesitate and be unfamiliar with this type of operation (even though it literally is an s/50869/8298/g for them). I’ll take most of December to remind folks a few times, and ultimately just clean up broken peering sessions in January 2022.
And of course, then lick my wounds and count pokemon - on October 24th, the day this post was published, Hurricane Electric showed 1'845 adjacencies in total for AS50869, of which 1'653 IPv4 and 1'430 IPv6. I will consider it a success if I lose less than 200 adjacencies. I’ll keep AS50869 around, as a test AS number to do a few experiments.
The Timeline
Most all intrusive maintenances will be done in maintenance windows between 22:00 - 03:00 UTC from Thursday to Friday. The tentative planning for the project, which starts on October 22nd and lasts through end of year (10 weeks):
- 2021-10-22 - RIPE updates for
route:,route6:and RPKI ROAs - 2021-10-24 - Originate prefixes from chbtl0/chbtl1 (no-op for the DFZ)
- 2021-10-28 - Move chbtl0/chbtl1 at Brüttisellen to AS8298
- 2021-10-29 - Update Brüttisellen IXP (AS58299 and AS58280), canary upstream AS58299
- 2021-10-29 - Headsup to CHIX-CH and SwissIX peers and Transit partners, announcing move
- 2021-11-04 - Move ddln0/ddln1 at Daedalean to AS8298, canary downstream AS57777
- 2021-11-11 - Move chgtg0 Glattbrugg to AS8298, add downstream AS212323
- 2021-11-12 - Move CHIX-CH peers to AS8298
- 2021-11-15 - Move chrma0 Rümlang to AS8298
- 2021-11-16 - Move SwissIX peers to AS8298
- Two week cooldown period, start to move IXPs to AS8298
- 2021-11-29 - Headsup to all european IXPs and IP Transit partners, announcing move
- 2021-12-02 - Move defra0, nlams0, frggh0, frpar0, chplo0 to AS8298
- 2021-12 - Move european IXPs to AS8298
- 2021-12-06 - First reminder for peerings that did not re-establish
- 2021-12-13 - Second reminder for peerings that did not re-establish
- 2021-12-20 - Third (final) reminder for peerings that did not re-establish
- 2022-01-10 - Remove peerings that did not re-establish
Appendix
This blogpost turned into a talk at RIPE #83, in case you wanted to take a look at the recording and a few questions.