Network and Security Design

From the beginning, it was clear to us that we needed a central identity provider (IdP) for user authentication and authorizations. This separates user authentication from services, and login credentials are only processed at the IdP. A compromised single service would thus not leak any credentials or sessions of other services connected to the IdP - except if the IdP itself is exploited. The IdP is therefore the most important service in terms of availability and security. We have decided not to use a cloud-based IdP, because we dont out-source security.

We encourage our users to use passwordless WebAuthn-based authentication with user verification and require 2FA for password-based login as a fallback.

At the beginning, we used Keycloak as IdP. While Keyclock is very flexible and almost feature complete, it requires a lot of configuration and does not support RADIUS. We later changed to Authentik because it is more lightweight and provides a RADIUS Provider, which we use to authenticate on network devices.

How do you want to operate all these services? A single server to rule them all? Just a bunch of VMs? K8s? your level of distribution? And then you have to decide whether you treat your servers as Pets or Cattles 😉.

Those questions are not easy to answer and depend not only on the computer resources available, but also on the knowledge (and habits) of the staff operating the IX. At DD-IX, these deliberations influenced our choice of software, which we would like to highlight in the following sections:

We isolate all services using hypervisors to minimise the risk of lateral movement in case a service is exploited. It may sound like a lot of effort to run all services in dedicated virtual machines, but it is not if the concept of MicroVMs is used. The fundamental idea of MicroVMs is to put each services into dedicated VMs, similar to containers. These VMs avoid the emulation of common hardware but rely on a large extent on VirtIO drivers to minimize the virtualisation overhead.

At DD-IX, we decided to use NixOS with Astro’s microvm.nix. This setup only takes a few additional nix-config options to put a service into a microvm.

The NixOS store of the base system is mapped read-only into the microvm containers using the virtiofs filesystem, which results in almost no storage overhead compared to a single machine setup. With the exception of memory and IP addresses, all resources can be shared efficiently with this setup. We make our nix-config publicly available.

Originally, since we started with a greenfield deployment, we were optimistic deploying an IPv6-only network internally. This should be possible in 2024, shouldn’t it?

We have failed several times to deploy an IPv6-only network. There are still leaf switches being sold whose silicon can not provide all features in IPv6 (underlays). The switch model we use have been launched around 2018, and so we have IPv4 addresses in our MP-BGP EVPN underlay. What we did not expect was that our core software (NixOS, IXP Manager, and arouteserver) requires also IPv4. Unfortunately, the NixOS infrastructure relies heavily on GitHub and, even in 2024, GitHub still does not provide AAAA resource records for github.com. Some of the online lookups that our IXP tool chains perform are still offered only via IPv4.

We looked at the available IPv6-only transitions. All transitions use some NAT(-like) mechanism which are not implemented in the vanilla linux kernel running on our firewall and for some of the transitions we would need to tamper recursive resolution inside of our network. We do not like any kind of NAT and we don’t like tampering.

We have therefore decided to still use IPv4 addresses in VLANs, but only if it is required. The non-public VLANs use an RFC1918 setup with NAT, we can’t have everything.

The “IXP” zone contains all devices and services that are directly attached to the peering LAN. This includes dedicated switches for the peering lan and connected route servers.

The second and largest zone “SVC” contains all devices and services that are necessary for the association and its business operations.

In the future, we are also planning to have a “LAB” zone for a full-stack IXP testing environment.

This is a somewhat vague definition and should contain from where can this service be accessed and to which application tier does the service belong to (if applicable)? A Web application such as the IXP Manager uses three services, each of them assigned to a different zone:

If we require more than a single VLAN within a zone a distinguisher is appended. This might be a counter or a location abbreviation. At the moment only the IXP zone is distributed over more than one PoP and we avoid to have PoP spanning broadcast domains if appropriate. So while the peering LAN is spanned over all PoPs the management and quarantine VLANs are of course not and so their name need to get distinguisher appended.

Defining zones helps to get some criteria for a more objective decision on which services should be separated from others.