I’ve been planning a network expansion for a while now. For the next few weeks, I will be in total geek-mode as I travel to several European cities to deploy AS50869 on a european ring. At the same time, my buddy Fred from IP-Max has been wanting to go to Amsterdam. IP-Max’s network is considerably larger than mine, but it just never clicked with the right set of circumstances for them to deploy in the Netherlands, until the stars aligned …
Deployment
After our adventure in Frankfurt,
Amsterdam, Lille,
and Paris came to an end, I still had a few
loose ends to tie up. In particular, in Lille I had dropped an old Dell R610
while waiting for new Supermicros to be delivered. There is benefit to having
one standard footprint setup, in my case an PCEngines APU2, Supermicro
5018D-FN8T and Intel X710-DA4 expansion NIC. They run fantastic with
DANOS and VPP applications.
Of course, we mustn’t forget home base, Geneva, where IP-Max has its headquarters in a beautiful mansion pictured here. At the same time, my family likes to take one trip per month to a city we don’t usually go, sort of to keep up with real life as we are now more and more able to travel. Marina has a niece in Geneva, who has lived and worked there for 20+ years, so we figured we’d combine these things and stay the weekend at her place.
After making our way from Zurich to Geneva, a trip that took us just short of six hours (!) by car, we arrived at the second half of the Belgium:Italy eurocup soccer match. It was perhaps due to our tardiness and lack of physical supportering, that the belgians lost the match that day. Sorry!
Connectivity
My current circuit runs from Paris (Leon Frot), frpar0.ipng.ch over a direct
DWDM wave to Zurich where I pick it up on chgtg0.ipng.ch at Interxion
Glattbrugg. So what we’ll do is break open this VLL at the IP-Max side,
insert the new router chplo0.ipng.ch, and reconfigure the Paris side
to go to the new router, and the new router to create another VLL back
to Zurich, which due to the toplogy of IP-Max’s underlying DWDM network
will traverse Paris - Lyon - Geneva instead (shaving off ~1.5ms of latency
at the same time).
I hung up the APU2 OOB server and the 5018D-FN8T router, and another Dell
R610 to run virtual machines at Safehost SH1 in Plan-les-Ouates, a southern
suburb of Geneva. I connected one 10G port to er01.gva20.ip-max.net and
another 10G port to er02.gva20.ip-max.net to obtain maximum availability
benefits. As an example of what the configuration on the ASR9k platform looks
like for this type of operation, here’s what I committed on er01.gva20.
Of course, first things first: let’s ensure that the OOB machine has connectivity, by allocating a /64 IPv6 and /29 IPv4. I usually configure myself a BGP transit session in the same subnet, which means we’ll want to bridge the 1G UTP connection of the APU with the 10G fiber connection of the Supermicro router, like so:
interface BVI911
description Cust: IPng OOB and Transit
ipv4 address 46.20.250.105 255.255.255.248
ipv4 unreachables disable
ipv6 nd suppress-ra
ipv6 address 2a02:2528:ff05::1/64
ipv6 enable
load-interval 30
!
interface GigabitEthernet0/7/0/38
description Cust: IPng APU (OOB)
mtu 9064
load-interval 30
l2transport
!
!
interface TenGigE0/1/0/3
description Cust: IPng (VLL and Transit)
mtu 9014
!
interface TenGigE0/1/0/3.911 l2transport
encapsulation dot1q 911 exact
rewrite ingress tag pop 1 symmetric
mtu 9018
!
l2vpn
bridge group BG_IPng
bridge-domain BD_IPng911
interface Te0/1/0/3.911
!
interface GigabitEthernet0/7/0/38
!
routed interface BVI911
!
!
!
After this, we pulled UTP cable and configured the APU2, which then has an
internal network towards the IPMI port of the Supermicro, and from there on,
the configuration becomes much easier. Of course, all config can be done
wirelessly, because the APU console.plo.ipng.nl acts as a WiFi access
point, so I connect to it and commit the network configs.
Once that’s online and happy, the router chplo0.ipng.ch is next. For this,
on er02.par02.ip-max.net, I reconfigure the current VLL to point to the
loopback of this router er01.gva20.ip-max.net using the same pw-id. Then,
I can configure this router as follows:
interface TenGigE0/1/0/3.100 l2transport
description Cust: IPng VLL to par02
encapsulation dot1q 100
rewrite ingress tag pop 1 symmetric
mtu 9018
!
l2vpn
pw-class EOMPLS-PW-CLASS
encapsulation mpls
transport-mode ethernet
!
!
xconnect group IPng
p2p IPng_to_par02
interface TenGigE0/1/0/3.100
neighbor ipv4 46.20.255.33 pw-id 210535705
pw-class EOMPLS-PW-CLASS
!
!
!
The results
And with that, the pseudowire is constructed, and the original interface on
frpar0.ipng.ch directly sees the interface here on chplo0.ipng.ch using
jumboframes of 9000 bytes (+14 bytes of ethernet overhead and +4 bytes of VLAN
tag on the ingress interface). It is as if the routers are directly connected
by a very long ethernet cable, a pseudo-wire if you wish. Super low pingtimes
are observed between this new router in Geneva and the existing two in Paris
and Zurich:
pim@chplo0:~$ /bin/ping -4 -c5 frpar0
PING frpar0.ipng.ch (194.1.163.33) 56(84) bytes of data.
64 bytes from frpar0.ipng.ch (194.1.163.33): icmp_seq=1 ttl=64 time=8.78 ms
64 bytes from frpar0.ipng.ch (194.1.163.33): icmp_seq=2 ttl=64 time=8.80 ms
64 bytes from frpar0.ipng.ch (194.1.163.33): icmp_seq=3 ttl=64 time=8.81 ms
64 bytes from frpar0.ipng.ch (194.1.163.33): icmp_seq=4 ttl=64 time=8.82 ms
64 bytes from frpar0.ipng.ch (194.1.163.33): icmp_seq=5 ttl=64 time=8.85 ms
--- frpar0.ipng.ch ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 10ms
rtt min/avg/max/mdev = 8.783/8.810/8.846/0.104 ms
pim@chplo0:~$ /bin/ping -6 -c5 chgtg0
PING chgtg0(chgtg0.ipng.ch (2001:678:d78::1)) 56 data bytes
64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=1 ttl=64 time=4.51 ms
64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=2 ttl=64 time=4.44 ms
64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=3 ttl=64 time=4.36 ms
64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=4 ttl=64 time=4.47 ms
64 bytes from chgtg0.ipng.ch (2001:678:d78::1): icmp_seq=5 ttl=64 time=4.41 ms
--- chgtg0 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 10ms
rtt min/avg/max/mdev = 4.362/4.436/4.506/0.077 ms
For good measure I’ve also connected to FreeIX, a new internet exchange project I’m working on, that will span the Geneva, Zurich and Lugano areas. More on that in a future post!
pim@chplo0:~$ iperf3 -4 -c 185.1.205.1 ## chgtg0.ipng.ch
Connecting to host 185.1.205.1, port 5201
[ 5] local 185.1.205.2 port 46872 connected to 185.1.205.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 809 MBytes 6.78 Gbits/sec 4 11.4 MBytes
[ 5] 1.00-2.00 sec 869 MBytes 7.29 Gbits/sec 0 11.4 MBytes
[ 5] 2.00-3.00 sec 865 MBytes 7.25 Gbits/sec 0 11.4 MBytes
[ 5] 3.00-4.00 sec 868 MBytes 7.28 Gbits/sec 0 11.4 MBytes
[ 5] 4.00-5.00 sec 836 MBytes 7.01 Gbits/sec 0 11.4 MBytes
[ 5] 5.00-6.00 sec 852 MBytes 7.15 Gbits/sec 0 11.4 MBytes
[ 5] 6.00-7.00 sec 865 MBytes 7.26 Gbits/sec 0 11.4 MBytes
[ 5] 7.00-8.00 sec 865 MBytes 7.26 Gbits/sec 0 11.4 MBytes
[ 5] 8.00-9.00 sec 861 MBytes 7.22 Gbits/sec 0 11.4 MBytes
[ 5] 9.00-10.00 sec 860 MBytes 7.22 Gbits/sec 0 11.4 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 8.35 GBytes 7.17 Gbits/sec 4 sender
[ 5] 0.00-10.01 sec 8.35 GBytes 7.16 Gbits/sec receiver
iperf Done.
You kind of get used to performance stats like this, but that said, it’s nice to see that performance over FreeIX is slightly lower than performance over the IPng backbone, and this is because on my VLLs, I can make use of jumbo frames, which gives me 20% or so better performance (currently 9.62 Gbits/sec).
Currently I’m busy at work in the background completing the configuration, the management environment and physical infrastructure for the internet exchange. I’m planning to make a more complete post about the FreeIX project in a few weeks once it’s ready for launch. Stay tuned!