Introduction to IPng Networks

At IPng Networks, we run a modest network with European reach. With our home base in Zurich, Switzerland, we are pretty well connected into the Swiss internet scene. We operate four sites in Zurich, and an additional set of sites in European cities, each of which are described on this post. If you’re curious as to how the network runs, you can find two main pieces here: Firstly, the physical parts, where exactly are IPng’s routers and switches, what types of kit does the ISP use, and so on. Secondly, the logical parts, what operating systems and configurations are in use.

Physical

Zurich Metropolitan Area

The Canton of Zurich, Switzerland is our home-base, and it’s where IPng Networks GmbH is registered. The local commercial datacenter scene is dominated by Interxion, NTT and Equinix. The small town of Brüttisellen (zipcode CH-8306), is where our founder lives and, due to the ongoing Corona pandemic, where he works from home.

In Brüttisellen, marked with C, we have our first two routers, chbtl0.ipng.ch and chbtl1.ipng.ch, racked in our office. There are only two fiber operators in this town - UPC and Swisscom. The orange trace (C to D) is a leased line from UPC, which we rent from Openfactory and it gets terminated at Interxion Glattbrugg, where our first router called chgtg0.ipng.ch is located. From there, Openfactory rents darkfiber to multiple locations - but notably the dark purple trace (D to E) that connects from Interxion Glattbrugg to NTT Rümlang, where our second router called chrma0.ipng.ch is located.

We rent a 10G CWDM wave between these two datacenters, directly connecting these two routers. Now, Equinix also has a sizable footprint in Zürich, and operating ZH04 (B where we only have passive optical presence) in the Industriekwartier (our local internet exchange SwissIX was born in the now defunct Equinix ZH01 office building). From the neighboring building Equinix ZH04, our partner IP-Max rents dark fiber to Equinix ZH05 in the Zurich Allmend area (the light purple trace B to F), and from there, IP-Max rents dark fiber to NTT Rümlang again (F to E), completing the ring. We rent a 10G circuit on that path, to redundantly connect our routers chgtg0 and chrma0. If at any time we’d need to connect partners or customers, we can do so at a moment’s notice, as rackspace is available in all Equinix sites for IPng Networks.

The green link (D to B) is a 10G carrier ethernet circuit between Interxion, over the light purple path (B to A) on its last mile to Albisrieden, where we built a very small colocation site, which you can read about in more detail in our informational post - the colo is open for private individuals and small businesses (contact us for details!).

European Ring

At IPng, we are strong believers in a free and open Internet. Having seen the shakeout of internet backbone providers over the last two decades, it seems to be a race to the bottom, with mergers, acquisitions and takeovers of datacenters and network carriers. Prices are going lower, and small fish traffic (let’s be honest, IPng Networks is definitely a small provider), to the point that purchasing IP transit is cheaper than connecting to local Internet exchange points. We’ve decided specifically to go the extra mile, quite literally, and plot a path to several continental european internet hubs.

Frankfurt - Connected from NTT’s datacenter at Rümlang (Zurich) with a first 10G circuit, and from Interxion’s datacenter at Glattbrugg (Zurich) with a second 10G circuit, this is our first hop into the world. Here, we connect to DE-CIX from Equinix FR5 at the Kleyerstrasse. More details in our post IPng Arrives in Frankfurt.

Amsterdam - The Amsterdam Science Park is where European Internet was born. NIKHEF is where we rent rackspace that connects with a 10G circuit to Frankfurt, and a 10G circuit onwards towards Lille. We connect to Speed-IX, LSIX, NL-IX, and an exchange point we help run called FrysIX. More details in our post IPng Arrives in Amsterdam.

Lille - IP-Max does lots of business in this region, with presence in both local datacenters here, one in Lille and one in Anzin. IPng has a point of presence here too, at the CIV1 facility, with a northbound 10G circuit to Amsterdam, and a southbound 10G circuit to Paris. Here, we connect to LillIX. More details in our post IPng Arrives in Lille.

Paris - Where two large facilities are placed back-to-back in the middle of the city, originally Telehouse TH2, with a new facility at Léon Frot, where we pick up a 10G circuit from Lille and further on the ring with a 10G circuit to Geneva. Here, we connect to FranceIX. More details in our post IPng Arrives in Paris.

Geneva - The home-base of IP-Max is where we close our ring. From Paris, IP-Max has two redundant paths back to Switzerland, the first being a DWDM link from to Zurich, and the second being a DWDM link to Lyon and then into Geneva. Here, at SafeHost in Plan les Ouates, is where we have our fourth Swiss point of presence, with a connection to our very own Free-IX and a 10G circuit to Interxion at Glattbrugg (Zurich), and of course to Paris. More details in our post IPng Arrives in Geneva.

Logical

As a small operator, we’d love to be able to boast the newest Juniper PTX10016 routers but we neither have the rack space, the power budget, and to be perfectly honest, the monetary budget to run these at IPng Networks. But it turns out, we know a fair bit about hardware silicon, architecture and the controlplane software running on commercial routers.

We’ve decided to go a different route. In our opinion, at speeds under 100Gbit, it’s perfectly viable to use software routers on off-the-shelf hardware, notably Intel network cards and CPUs, notably those that have support for the Dataplane Development Kit (aka DPDK), which offers libraries to accelerate packet processing workloads, which turn ordinary servers into very performant routers. Two notable applications are VPP and Danos.

VPP

VPP originally comes from the house of Cisco [ref] and looks quite a bit like the commercial ASR9k platform. In development since 2002, VPP is production code currently running in shipping products. It runs in user space on multiple architectures including x86, ARM, and Power architectures on both x86 servers and embedded devices. The design of VPP is hardware, kernel, and deployment (bare metal, VM, container) agnostic. It runs completely in userspace.

We’ve contributed a little bit to the Control Plane abstraction [ref], which allows users to combine the throughput of a dataplane with usual routing software like Bird or FRR. We’ve been running it in production since December 2020 on chbtl1.ipng.ch. It’s our ultimate goal to run VPP and Linux Control Plane on the entire network, as the design and architecture really resonates with us as software and systems engineers.

DANOS

The Disaggregated Network Operating System (DANOS) project originally comes from AT&T’s “dNOS” software framework and provides an open, cost-effective and flexible alternative to traditional networking equipment. As part of The Linux Foundation, it now incorporates contributions from complementary open source communities in building a standardized distributed Network Operating System (NOS) to speed the adoption and use of white boxes in a service provider’s infrastructure.

We’ve been using DANOS since its first release in August 2019, and it’s currently our routing platform of choice – it combines the sheer speed of DPDK with a Vyatta command line interface. As an appliance, care was taken to complete the whole package, with SNMP, YANG interface, image and upgrade management, interface monitoring with wireshark semantics, et cetera. Performing easily at wire speed 10G workloads (including 64byte ethernet frames), and being completely open source, it fits very well with our philosophy of an open and free internet.