I’ve been planning a network expansion for a while now. For the next few weeks, I will be in total geek-mode as I travel to several European cities to deploy AS50869 on a european ring. At the same time, my buddy Fred from IP-Max has been wanting to go to Amsterdam. IP-Max’s network is considerably larger than mine, but it just never clicked with the right set of circumstances for them to deploy in the Netherlands, until the stars aligned …
After our adventure in Frankfurt,
and Paris came to an end, I still had a few
loose ends to tie up. In particular, in Lille I had dropped an old Dell R610
while waiting for new Supermicros to be delivered. There is benefit to having
one standard footprint setup, in my case an PCEngines
5018D-FN8T and Intel
X710-DA4 expansion NIC. They run fantastic with
DANOS and VPP applications.
Of course, we mustn’t forget home base, Geneva, where IP-Max has its headquarters in a beautiful mansion pictured here. At the same time, my family likes to take one trip per month to a city we don’t usually go, sort of to keep up with real life as we are now more and more able to travel. Marina has a niece in Geneva, who has lived and worked there for 20+ years, so we figured we’d combine these things and stay the weekend at her place.
After making our way from Zurich to Geneva, a trip that took us just short of six hours (!) by car, we arrived at the second half of the Belgium:Italy eurocup soccer match. It was perhaps due to our tardiness and lack of physical supportering, that the belgians lost the match that day. Sorry!
My current circuit runs from Paris (Leon Frot),
frpar0.ipng.ch over a direct
DWDM wave to Zurich where I pick it up on
chgtg0.ipng.ch at Interxion
Glattbrugg. So what we’ll do is break open this VLL at the IP-Max side,
insert the new router
chplo0.ipng.ch, and reconfigure the Paris side
to go to the new router, and the new router to create another VLL back
to Zurich, which due to the toplogy of IP-Max’s underlying DWDM network
will traverse Paris - Lyon - Geneva instead (shaving off ~1.5ms of latency
at the same time).
I hung up the
APU2 OOB server and the
5018D-FN8T router, and another Dell
R610 to run virtual machines at Safehost SH1 in Plan-les-Ouates, a southern
suburb of Geneva. I connected one 10G port to
another 10G port to
er02.gva20.ip-max.net to obtain maximum availability
benefits. As an example of what the configuration on the ASR9k platform looks
like for this type of operation, here’s what I committed on
Of course, first things first: let’s ensure that the OOB machine has connectivity, by allocating a /64 IPv6 and /29 IPv4. I usually configure myself a BGP transit session in the same subnet, which means we’ll want to bridge the 1G UTP connection of the APU with the 10G fiber connection of the Supermicro router, like so:
interface BVI911 description Cust: IPng OOB and Transit ipv4 address 22.214.171.124 255.255.255.248 ipv4 unreachables disable ipv6 nd suppress-ra ipv6 address 2a02:2528:ff05::1/64 ipv6 enable load-interval 30 ! interface GigabitEthernet0/7/0/38 description Cust: IPng APU (OOB) mtu 9064 load-interval 30 l2transport ! ! interface TenGigE0/1/0/3 description Cust: IPng (VLL and Transit) mtu 9014 ! interface TenGigE0/1/0/3.911 l2transport encapsulation dot1q 911 exact rewrite ingress tag pop 1 symmetric mtu 9018 ! l2vpn bridge group BG_IPng bridge-domain BD_IPng911 interface Te0/1/0/3.911 ! interface GigabitEthernet0/7/0/38 ! routed interface BVI911 ! ! !
After this, we pulled UTP cable and configured the
APU2, which then has an
internal network towards the IPMI port of the Supermicro, and from there on,
the configuration becomes much easier. Of course, all config can be done
wirelessly, because the APU
console.plo.ipng.nl acts as a WiFi access
point, so I connect to it and commit the network configs.
Once that’s online and happy, the router
chplo0.ipng.ch is next. For this,
er02.par02.ip-max.net, I reconfigure the current VLL to point to the
loopback of this router
er01.gva20.ip-max.net using the same
I can configure this router as follows:
interface TenGigE0/1/0/3.100 l2transport description Cust: IPng VLL to par02 encapsulation dot1q 100 rewrite ingress tag pop 1 symmetric mtu 9018 ! l2vpn pw-class EOMPLS-PW-CLASS encapsulation mpls transport-mode ethernet ! ! xconnect group IPng p2p IPng_to_par02 interface TenGigE0/1/0/3.100 neighbor ipv4 126.96.36.199 pw-id 210535705 pw-class EOMPLS-PW-CLASS ! ! !
And with that, the pseudowire is constructed, and the original interface on
frpar0.ipng.ch directly sees the interface here on
jumboframes of 9000 bytes (+14 bytes of ethernet overhead and +4 bytes of VLAN
tag on the ingress interface). It is as if the routers are directly connected
by a very long ethernet cable, a pseudo-wire if you wish. Super low pingtimes
are observed between this new router in Geneva and the existing two in Paris
pim@chplo0:~$ /bin/ping -4 -c5 frpar0 PING frpar0.ipng.ch (188.8.131.52) 56(84) bytes of data. 64 bytes from frpar0.ipng.ch (184.108.40.206): icmp_seq=1 ttl=64 time=8.78 ms 64 bytes from frpar0.ipng.ch (220.127.116.11): icmp_seq=2 ttl=64 time=8.80 ms 64 bytes from frpar0.ipng.ch (18.104.22.168): icmp_seq=3 ttl=64 time=8.81 ms 64 bytes from frpar0.ipng.ch (22.214.171.124): icmp_seq=4 ttl=64 time=8.82 ms 64 bytes from frpar0.ipng.ch (126.96.36.199): icmp_seq=5 ttl=64 time=8.85 ms --- frpar0.ipng.ch ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 10ms rtt min/avg/max/mdev = 8.783/8.810/8.846/0.104 ms pim@chplo0:~$ /bin/ping -6 -c5 chgtg0 PING chgtg0(chgtg0.ipng.ch (2001:678:d78::1)) 56 data bytes 64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=1 ttl=64 time=4.51 ms 64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=2 ttl=64 time=4.44 ms 64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=3 ttl=64 time=4.36 ms 64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=4 ttl=64 time=4.47 ms 64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=5 ttl=64 time=4.41 ms --- chgtg0 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 10ms rtt min/avg/max/mdev = 4.362/4.436/4.506/0.077 ms
For good measure I’ve also connected to FreeIX, a new internet exchange project I’m working on, that will span the Geneva, Zurich and Lugano areas. More on that in a future post!
pim@chplo0:~$ iperf3 -4 -c 188.8.131.52 ## chgtg0.ipng.ch Connecting to host 184.108.40.206, port 5201 [ 5] local 220.127.116.11 port 46872 connected to 18.104.22.168 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 809 MBytes 6.78 Gbits/sec 4 11.4 MBytes [ 5] 1.00-2.00 sec 869 MBytes 7.29 Gbits/sec 0 11.4 MBytes [ 5] 2.00-3.00 sec 865 MBytes 7.25 Gbits/sec 0 11.4 MBytes [ 5] 3.00-4.00 sec 868 MBytes 7.28 Gbits/sec 0 11.4 MBytes [ 5] 4.00-5.00 sec 836 MBytes 7.01 Gbits/sec 0 11.4 MBytes [ 5] 5.00-6.00 sec 852 MBytes 7.15 Gbits/sec 0 11.4 MBytes [ 5] 6.00-7.00 sec 865 MBytes 7.26 Gbits/sec 0 11.4 MBytes [ 5] 7.00-8.00 sec 865 MBytes 7.26 Gbits/sec 0 11.4 MBytes [ 5] 8.00-9.00 sec 861 MBytes 7.22 Gbits/sec 0 11.4 MBytes [ 5] 9.00-10.00 sec 860 MBytes 7.22 Gbits/sec 0 11.4 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 8.35 GBytes 7.17 Gbits/sec 4 sender [ 5] 0.00-10.01 sec 8.35 GBytes 7.16 Gbits/sec receiver iperf Done.
You kind of get used to performance stats like this, but that said, it’s nice to see that performance over FreeIX is slightly lower than performance over the IPng backbone, and this is because on my VLLs, I can make use of jumbo frames, which gives me 20% or so better performance (currently 9.62 Gbits/sec).
Currently I’m busy at work in the background completing the configuration, the management environment and physical infrastructure for the internet exchange. I’m planning to make a more complete post about the FreeIX project in a few weeks once it’s ready for launch. Stay tuned!