I’ve been planning a network expansion for a while now. For the next few weeks, I will be in total geek-mode as I travel to several European cities to deploy AS50869 on a european ring. At the same time, my buddy Fred from IP-Max has been wanting to go to Amsterdam. IP-Max’s network is considerably larger than mine, but it just never clicked with the right set of circumstances for them to deploy in the Netherlands, until the stars aligned …


Quai du Rhône

After our adventure in Frankfurt, Amsterdam, Lille, and Paris came to an end, I still had a few loose ends to tie up. In particular, in Lille I had dropped an old Dell R610 while waiting for new Supermicros to be delivered. There is benefit to having one standard footprint setup, in my case an PCEngines APU2, Supermicro 5018D-FN8T and Intel X710-DA4 expansion NIC. They run fantastic with DANOS and VPP applications.

Of course, we mustn’t forget home base, Geneva, where IP-Max has its headquarters in a beautiful mansion pictured here. At the same time, my family likes to take one trip per month to a city we don’t usually go, sort of to keep up with real life as we are now more and more able to travel. Marina has a niece in Geneva, who has lived and worked there for 20+ years, so we figured we’d combine these things and stay the weekend at her place.

After making our way from Zurich to Geneva, a trip that took us just short of six hours (!) by car, we arrived at the second half of the Belgium:Italy eurocup soccer match. It was perhaps due to our tardiness and lack of physical supportering, that the belgians lost the match that day. Sorry!


Geneva Rack

My current circuit runs from Paris (Leon Frot), frpar0.ipng.ch over a direct DWDM wave to Zurich where I pick it up on chgtg0.ipng.ch at Interxion Glattbrugg. So what we’ll do is break open this VLL at the IP-Max side, insert the new router chplo0.ipng.ch, and reconfigure the Paris side to go to the new router, and the new router to create another VLL back to Zurich, which due to the toplogy of IP-Max’s underlying DWDM network will traverse Paris - Lyon - Geneva instead (shaving off ~1.5ms of latency at the same time).

I hung up the APU2 OOB server and the 5018D-FN8T router, and another Dell R610 to run virtual machines at Safehost SH1 in Plan-les-Ouates, a southern suburb of Geneva. I connected one 10G port to er01.gva20.ip-max.net and another 10G port to er02.gva20.ip-max.net to obtain maximum availability benefits. As an example of what the configuration on the ASR9k platform looks like for this type of operation, here’s what I committed on er01.gva20.

Of course, first things first: let’s ensure that the OOB machine has connectivity, by allocating a /64 IPv6 and /29 IPv4. I usually configure myself a BGP transit session in the same subnet, which means we’ll want to bridge the 1G UTP connection of the APU with the 10G fiber connection of the Supermicro router, like so:

interface BVI911
 description Cust: IPng OOB and Transit
 ipv4 address
 ipv4 unreachables disable
 ipv6 nd suppress-ra
 ipv6 address 2a02:2528:ff05::1/64
 ipv6 enable
 load-interval 30
interface GigabitEthernet0/7/0/38
 description Cust: IPng APU (OOB)
 mtu 9064
 load-interval 30
interface TenGigE0/1/0/3
 description Cust: IPng (VLL and Transit)
 mtu 9014
interface TenGigE0/1/0/3.911 l2transport
 encapsulation dot1q 911 exact
 rewrite ingress tag pop 1 symmetric
 mtu 9018

 bridge group BG_IPng
  bridge-domain BD_IPng911
   interface Te0/1/0/3.911
   interface GigabitEthernet0/7/0/38
   routed interface BVI911

After this, we pulled UTP cable and configured the APU2, which then has an internal network towards the IPMI port of the Supermicro, and from there on, the configuration becomes much easier. Of course, all config can be done wirelessly, because the APU console.plo.ipng.nl acts as a WiFi access point, so I connect to it and commit the network configs.

Once that’s online and happy, the router chplo0.ipng.ch is next. For this, on er02.par02.ip-max.net, I reconfigure the current VLL to point to the loopback of this router er01.gva20.ip-max.net using the same pw-id. Then, I can configure this router as follows:

interface TenGigE0/1/0/3.100 l2transport
 description Cust: IPng VLL to par02
 encapsulation dot1q 100
 rewrite ingress tag pop 1 symmetric
 mtu 9018

  encapsulation mpls
   transport-mode ethernet
 xconnect group IPng
  p2p IPng_to_par02
   interface TenGigE0/1/0/3.100
   neighbor ipv4 pw-id 210535705
    pw-class EOMPLS-PW-CLASS

The results

And with that, the pseudowire is constructed, and the original interface on frpar0.ipng.ch directly sees the interface here on chplo0.ipng.ch using jumboframes of 9000 bytes (+14 bytes of ethernet overhead and +4 bytes of VLAN tag on the ingress interface). It is as if the routers are directly connected by a very long ethernet cable, a pseudo-wire if you wish. Super low pingtimes are observed between this new router in Geneva and the existing two in Paris and Zurich:

pim@chplo0:~$ /bin/ping -4 -c5 frpar0
PING frpar0.ipng.ch ( 56(84) bytes of data.
64 bytes from frpar0.ipng.ch ( icmp_seq=1 ttl=64 time=8.78 ms
64 bytes from frpar0.ipng.ch ( icmp_seq=2 ttl=64 time=8.80 ms
64 bytes from frpar0.ipng.ch ( icmp_seq=3 ttl=64 time=8.81 ms
64 bytes from frpar0.ipng.ch ( icmp_seq=4 ttl=64 time=8.82 ms
64 bytes from frpar0.ipng.ch ( icmp_seq=5 ttl=64 time=8.85 ms

--- frpar0.ipng.ch ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 10ms
rtt min/avg/max/mdev = 8.783/8.810/8.846/0.104 ms
pim@chplo0:~$ /bin/ping -6 -c5 chgtg0
PING chgtg0(chgtg0.ipng.ch (2001:678:d78::1)) 56 data bytes
64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=1 ttl=64 time=4.51 ms
64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=2 ttl=64 time=4.44 ms
64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=3 ttl=64 time=4.36 ms
64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=4 ttl=64 time=4.47 ms
64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=5 ttl=64 time=4.41 ms

--- chgtg0 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 10ms
rtt min/avg/max/mdev = 4.362/4.436/4.506/0.077 ms

For good measure I’ve also connected to FreeIX, a new internet exchange project I’m working on, that will span the Geneva, Zurich and Lugano areas. More on that in a future post!

pim@chplo0:~$ iperf3 -4 -c ## chgtg0.ipng.ch
Connecting to host, port 5201
[  5] local port 46872 connected to port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   809 MBytes  6.78 Gbits/sec    4   11.4 MBytes       
[  5]   1.00-2.00   sec   869 MBytes  7.29 Gbits/sec    0   11.4 MBytes       
[  5]   2.00-3.00   sec   865 MBytes  7.25 Gbits/sec    0   11.4 MBytes       
[  5]   3.00-4.00   sec   868 MBytes  7.28 Gbits/sec    0   11.4 MBytes       
[  5]   4.00-5.00   sec   836 MBytes  7.01 Gbits/sec    0   11.4 MBytes       
[  5]   5.00-6.00   sec   852 MBytes  7.15 Gbits/sec    0   11.4 MBytes       
[  5]   6.00-7.00   sec   865 MBytes  7.26 Gbits/sec    0   11.4 MBytes       
[  5]   7.00-8.00   sec   865 MBytes  7.26 Gbits/sec    0   11.4 MBytes       
[  5]   8.00-9.00   sec   861 MBytes  7.22 Gbits/sec    0   11.4 MBytes       
[  5]   9.00-10.00  sec   860 MBytes  7.22 Gbits/sec    0   11.4 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  8.35 GBytes  7.17 Gbits/sec    4             sender
[  5]   0.00-10.01  sec  8.35 GBytes  7.16 Gbits/sec                  receiver

iperf Done.

You kind of get used to performance stats like this, but that said, it’s nice to see that performance over FreeIX is slightly lower than performance over the IPng backbone, and this is because on my VLLs, I can make use of jumbo frames, which gives me 20% or so better performance (currently 9.62 Gbits/sec).

Currently I’m busy at work in the background completing the configuration, the management environment and physical infrastructure for the internet exchange. I’m planning to make a more complete post about the FreeIX project in a few weeks once it’s ready for launch. Stay tuned!