VPP Linux CP - Part3

Posted: 2021-08-15

About this series

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation services router), VPP will look and feel quite familiar as many of the approaches are shared between the two. One thing notably missing, is the higher level control plane, that is to say: there is no OSPF or ISIS, BGP, LDP and the like. This series of posts details my work on a VPP plugin which is called the Linux Control Plane, or LCP for short, which creates Linux network devices that mirror their VPP dataplane counterpart. IPv4 and IPv6 traffic, and associated protocols like ARP and IPv6 Neighbor Discovery can now be handled by Linux, while the heavy lifting of packet forwarding is done by the VPP dataplane. Or, said another way: this plugin will allow Linux to use VPP as a software ASIC for fast forwarding, filtering, NAT, and so on, while keeping control of the interface state (links, addresses and routes) itself. When the plugin is completed, running software like FRR or Bird on top of VPP and achieving >100Mpps and >100Gbps forwarding rates will be well in reach!

In this third post, I’ll be adding a convenience feature that I think will be popular: the plugin will now automatically create or delete LIPs for sub-interfaces where-ever the parent has a LIP configured.

My test setup

I’ve extended the setup from the first post. The base configuration for the enp66s0f0 interface remains exactly the same, but I’ve also added an LACP bond0 interface, which also has the whole kitten kaboodle of sub-interfaces defined, see below in the Appendix for details, but here’s the table again for reference:

Name	type	Addresses
enp66s0f0	untagged	10.0.1.2/30 2001:db8:0:1::2/64
enp66s0f0.q	dot1q 1234	10.0.2.2/30 2001:db8:0:2::2/64
enp66s0f0.qinq	outer dot1q 1234, inner dot1q 1000	10.0.3.2/30 2001:db8:0:3::2/64
enp66s0f0.ad	dot1ad 2345	10.0.4.2/30 2001:db8:0:4::2/64
enp66s0f0.qinad	outer dot1ad 2345, inner dot1q 1000	10.0.5.2/30 2001:db8:0:5::2/64
bond0	untagged	10.1.1.2/30 2001:db8:1:1::2/64
bond0.q	dot1q 1234	10.1.2.2/30 2001:db8:1:2::2/64
bond0.qinq	outer dot1q 1234, inner dot1q 1000	10.1.3.2/30 2001:db8:1:3::2/64
bond0.ad	dot1ad 2345	10.1.4.2/30 2001:db8:1:4::2/64
bond0.qinad	outer dot1ad 2345, inner dot1q 1000	10.1.5.2/30 2001:db8:1:5::2/64

The goal of this post is to show what code needed to be written and which changes needed to be made to the plugin, in order to automatically create and delete sub-interfaces.

Startingpoint

Based on the state of the plugin after the second post, operators must create LIP instances for interfaces as well as each sub-interface explicitly:

DBGvpp# lcp create TenGigabitEthernet3/0/0 host-if e0
DBGvpp# create sub TenGigabitEthernet3/0/0 1234
DBGvpp# lcp create TenGigabitEthernet3/0/0.1234 host-if e0.1234
DBGvpp# create sub TenGigabitEthernet3/0/0 1235 dot1q 1234 inner-dot1q 1000 exact-match
DBGvpp# lcp create TenGigabitEthernet3/0/0.1235 host-if e0.1235
DBGvpp# create sub TenGigabitEthernet3/0/0 1236 dot1ad 2345 exact-match
DBGvpp# lcp create TenGigabitEthernet3/0/0.1236 host-if e0.1236
DBGvpp# create sub TenGigabitEthernet3/0/0 1237 dot1ad 2345 inner-dot1q 1000 exact-match
DBGvpp# lcp create TenGigabitEthernet3/0/0.1237 host-if e0.1237

But one might ask – is it really useful to have L3 interfaces in VPP without a companion interface in an appropriate Linux namespace? I think the answer might be ‘yes’ for individual interfaces (for example, in a mgmt VRF that has no need to run routing protocols), but I also think the answer is probably ’no’ for sub-interfaces, once their parent has a LIP defined.

Configuration

The original plugin (the one that ships with VPP 21.06) has a configuration flag that seems promising by defining a flag interface-auto-create, but its implementation was never finished. I’ve removed that flag and replaced it with a new one. The main reason for this decision is that there are actually two kinds of auto configuration: the first one is detailed in this post, but in the future, I will also make it possible to create VPP interfaces by creating their Linux counterpart (eg. ip link add link e0 name e0.1234 type vlan id 1234 with a configuration statement that might be called netlink-auto-subint), and I’d like for the plugin to individually enable/disable both types. Also, I find the name unfortunate, as the feature should create and delete LIPs on sub-interfaces, not just create them. So out with the old, in with the new :)

I have to acknowledge that not everybody will want automagically created interfaces, similar to the original configuration, so I define a new configuration flag called lcp-auto-subint which goes into the linux-cp module configuration stanza in VPP’s startup.conf, which might look a little like this:

linux-cp {
  default netns dataplane
  lcp-auto-subint
}

Based on this config, I set the startup default in lcp_set_lcp_auto_subint(), but I realize that an administrator may want to turn it on/off at runtime, too, so I add a CLI getter/setter that interacts with the flag in this [commit]:

DBGvpp# show lcp
lcp default netns dataplane
lcp lcp-auto-subint on
lcp lcp-sync off

DBGvpp# lcp lcp-auto-subint off
DBGvpp# show lcp
lcp default netns dataplane
lcp lcp-auto-subint off
lcp lcp-sync off

The prep work for the rest of the interface syncer starts with this [commit], and for the rest of this blog post, the behavior will be in the ‘on’ position.

The code for the configuration toggle is in this [commit].

Auto create/delete sub-interfaces

The original plugin code (that ships with VPP 21.06) made a start by defining a function called lcp_itf_phy_add() and registering an intent with VNET_SW_INTERFACE_ADD_DEL_FUNCTION(). I’ve moved the function to the source file I created in Part 2 (called lcp_if_sync.c), specifically to handle interface syncing, and gave it a name that matches the VPP callback, so lcp_itf_interface_add_del().

The logic of that function is pretty straight forward. I want to only continue if lcp-auto-subint is set, and I only want to create or delete sub-interfaces, not parents. This way, the operator can decide on a per-interface basis if they want it to participate in Linux (eg, issuing lcp create BondEthernet0 host-if be0). After I’ve established that (a) the caller wants auto-creation/auto-deletion, and (b) we’re fielding a callback for a sub-int, all I must do is:

On creation: does the parent interface sw->sup_sw_if_index have a LIP? If yes, let’s create a LIP for this sub-interface, too. We determine that Linux interface name by taking the parent name (say, be0), and sticking the sub-int number after it, like be0.1234.
On deletion: does this sub-interface we’re fielding the callback for have a LIP? If yes, then delete it.

I noticed that interface deletion had a bug (one that I fell victim to as well: it does not remove the netlink device in the correct network namespace), which I fixed.

The code for the auto create/delete and the bugfix is in this [commit].

Further Work

One other thing I noticed (and this is actually a bug!) is that on BondEthernet interfaces, upon creation a temporary MAC is assigned, which is subsequently overwritten by the first physical interface that is added to the bond, which means that when a LIP is created before the first interface is added, its MAC will be the temporary MAC. Compare:

vppctl create bond mode lacp load-balance l2
vppctl lcp create BondEthernet0 host-if be0
## MAC of be0 is now a temp/ephemeral MAC

vppctl bond add BondEthernet0 TenGigabitEthernet3/0/2
vppctl bond add BondEthernet0 TenGigabitEthernet3/0/3
## MAC of the BondEthernet0 device is now that of TenGigabitEthernet3/0/2
## MAC of TenGigabitEthernet3/0/3 is that of BondEthernet0 (ie. Te3/0/2)

In such a situation, be0 will not be reachable unless it’s manually set to the correct MAC. I looked around but found no callback of event handler for MAC address changes in VPP – so I should add one probably, but in the mean time, I’ll just add interfaces to the bond before creating the LIP, like so:

vppctl create bond mode lacp load-balance l2
vppctl bond add BondEthernet0 TenGigabitEthernet3/0/2
## MAC of the BondEthernet0 device is now that of TenGigabitEthernet3/0/2

vppctl lcp create BondEthernet0 host-if be0
## MAC of be0 is now that of BondEthernet0

vppctl bond add BondEthernet0 TenGigabitEthernet3/0/3
## MAC of TenGigabitEthernet3/0/3 is that of BondEthernet0 (ie. Te3/0/2)

.. which is an adequate workaround for now.

Results

After this code is in, the operator will only have to create a LIP for the main interfaces, and the plugin will take care of the rest!

pim@hippo:~/src/lcpng$ grep 'create' config3.sh
vppctl lcp lcp-auto-subint on
vppctl lcp create TenGigabitEthernet3/0/0 host-if e0
vppctl create sub TenGigabitEthernet3/0/0 1234
vppctl create sub TenGigabitEthernet3/0/0 1235 dot1q 1234 inner-dot1q 1000 exact-match
vppctl create sub TenGigabitEthernet3/0/0 1236 dot1ad 2345 exact-match
vppctl create sub TenGigabitEthernet3/0/0 1237 dot1ad 2345 inner-dot1q 1000 exact-match

vppctl create bond mode lacp load-balance l2
vppctl lcp create BondEthernet0 host-if be0
vppctl create sub BondEthernet0 1234
vppctl create sub BondEthernet0 1235 dot1q 1234 inner-dot1q 1000 exact-match
vppctl create sub BondEthernet0 1236 dot1ad 2345 exact-match
vppctl create sub BondEthernet0 1237 dot1ad 2345 inner-dot1q 1000 exact-match

And as an end-to-end functional validation, now extended as well to ping the Ubuntu machine over the LACP interface and all of its subinterfaces, works like a charm:

pim@hippo:~/src/lcpng$ sudo ip netns exec dataplane ip link | grep e0
1063: e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UNKNOWN mode DEFAULT group default qlen 1000
1064: be0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UNKNOWN mode DEFAULT group default qlen 1000
209: e0.1234@e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
210: e0.1235@e0.1234: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
211: e0.1236@e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
212: e0.1237@e0.1236: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
213: be0.1234@be0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
214: be0.1235@be0.1234: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
215: be0.1236@be0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
216: be0.1237@be0.1236: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000

# The TenGigabitEthernet3/0/0 (e0) interfaces
pim@hippo:~/src/lcpng$ fping 10.0.1.2 10.0.2.2 10.0.3.2 10.0.4.2 10.0.5.2
10.0.1.2 is alive
10.0.2.2 is alive
10.0.3.2 is alive
10.0.4.2 is alive
10.0.5.2 is alive

pim@hippo:~/src/lcpng$ fping6 2001:db8:0:1::2 2001:db8:0:2::2 \
  2001:db8:0:3::2 2001:db8:0:4::2 2001:db8:0:5::2
2001:db8:0:1::2 is alive
2001:db8:0:2::2 is alive
2001:db8:0:3::2 is alive
2001:db8:0:4::2 is alive
2001:db8:0:5::2 is alive

## The BondEthernet0 (be0) interfaces
pim@hippo:~/src/lcpng$ fping 10.1.1.2 10.1.2.2 10.1.3.2 10.1.4.2 10.1.5.2
10.1.1.2 is alive
10.1.2.2 is alive
10.1.3.2 is alive
10.1.4.2 is alive
10.1.5.2 is alive

pim@hippo:~/src/lcpng$ fping6 2001:db8:1:1::2 2001:db8:1:2::2 \
  2001:db8:1:3::2 2001:db8:1:4::2 2001:db8:1:5::2
2001:db8:1:1::2 is alive
2001:db8:1:2::2 is alive
2001:db8:1:3::2 is alive
2001:db8:1:4::2 is alive
2001:db8:1:5::2 is alive

Credits

I’d like to make clear that the Linux CP plugin is a great collaboration between several great folks and that my work stands on their shoulders. I’ve had a little bit of help along the way from Neale Ranns, Matthew Smith and Jon Loeliger, and I’d like to thank them for their work!

Appendix

Ubuntu config

# Untagged interface
ip addr add 10.0.1.2/30 dev enp66s0f0
ip addr add 2001:db8:0:1::2/64 dev enp66s0f0
ip link set enp66s0f0 up mtu 9000

# Single 802.1q tag 1234
ip link add link enp66s0f0 name enp66s0f0.q type vlan id 1234
ip link set enp66s0f0.q up mtu 9000
ip addr add 10.0.2.2/30 dev enp66s0f0.q
ip addr add 2001:db8:0:2::2/64 dev enp66s0f0.q

# Double 802.1q tag 1234 inner-tag 1000
ip link add link enp66s0f0.q name enp66s0f0.qinq type vlan id 1000
ip link set enp66s0f0.qinq up mtu 9000
ip addr add 10.0.3.2/30 dev enp66s0f0.qinq
ip addr add 2001:db8:0:3::2/64 dev enp66s0f0.qinq

# Single 802.1ad tag 2345
ip link add link enp66s0f0 name enp66s0f0.ad type vlan id 2345 proto 802.1ad
ip link set enp66s0f0.ad up mtu 9000
ip addr add 10.0.4.2/30 dev enp66s0f0.ad
ip addr add 2001:db8:0:4::2/64 dev enp66s0f0.ad

# Double 802.1ad tag 2345 inner-tag 1000
ip link add link enp66s0f0.ad name enp66s0f0.qinad type vlan id 1000 proto 802.1q
ip link set enp66s0f0.qinad up mtu 9000
ip addr add 10.0.5.2/30 dev enp66s0f0.qinad
ip addr add 2001:db8:0:5::2/64 dev enp66s0f0.qinad

## Bond interface
ip link add bond0 type bond mode 802.3ad
ip link set enp66s0f2 down
ip link set enp66s0f3 down
ip link set enp66s0f2 master bond0
ip link set enp66s0f3 master bond0
ip link set enp66s0f2 up
ip link set enp66s0f3 up
ip link set bond0 up

ip addr add 10.1.1.2/30 dev bond0
ip addr add 2001:db8:1:1::2/64 dev bond0
ip link set bond0 up mtu 9000

# Single 802.1q tag 1234
ip link add link bond0 name bond0.q type vlan id 1234
ip link set bond0.q up mtu 9000
ip addr add 10.1.2.2/30 dev bond0.q
ip addr add 2001:db8:1:2::2/64 dev bond0.q

# Double 802.1q tag 1234 inner-tag 1000
ip link add link bond0.q name bond0.qinq type vlan id 1000
ip link set bond0.qinq up mtu 9000
ip addr add 10.1.3.2/30 dev bond0.qinq
ip addr add 2001:db8:1:3::2/64 dev bond0.qinq

# Single 802.1ad tag 2345
ip link add link bond0 name bond0.ad type vlan id 2345 proto 802.1ad
ip link set bond0.ad up mtu 9000
ip addr add 10.1.4.2/30 dev bond0.ad
ip addr add 2001:db8:1:4::2/64 dev bond0.ad

# Double 802.1ad tag 2345 inner-tag 1000
ip link add link bond0.ad name bond0.qinad type vlan id 1000 proto 802.1q
ip link set bond0.qinad up mtu 9000
ip addr add 10.1.5.2/30 dev bond0.qinad
ip addr add 2001:db8:1:5::2/64 dev bond0.qinad

VPP config

## No more `lcp create` commands for sub-interfaces.
vppctl lcp default netns dataplane
vppctl lcp lcp-auto-subint on

vppctl lcp create TenGigabitEthernet3/0/0 host-if e0
vppctl set interface state TenGigabitEthernet3/0/0 up
vppctl set interface mtu packet 9000 TenGigabitEthernet3/0/0
vppctl set interface ip address TenGigabitEthernet3/0/0 10.0.1.1/30
vppctl set interface ip address TenGigabitEthernet3/0/0 2001:db8:0:1::1/64

vppctl create sub TenGigabitEthernet3/0/0 1234
vppctl set interface mtu packet 9000 TenGigabitEthernet3/0/0.1234
vppctl set interface state TenGigabitEthernet3/0/0.1234 up
vppctl set interface ip address TenGigabitEthernet3/0/0.1234 10.0.2.1/30
vppctl set interface ip address TenGigabitEthernet3/0/0.1234 2001:db8:0:2::1/64

vppctl create sub TenGigabitEthernet3/0/0 1235 dot1q 1234 inner-dot1q 1000 exact-match
vppctl set interface state TenGigabitEthernet3/0/0.1235 up
vppctl set interface mtu packet 9000 TenGigabitEthernet3/0/0.1235
vppctl set interface ip address TenGigabitEthernet3/0/0.1235 10.0.3.1/30
vppctl set interface ip address TenGigabitEthernet3/0/0.1235 2001:db8:0:3::1/64

vppctl create sub TenGigabitEthernet3/0/0 1236 dot1ad 2345 exact-match
vppctl set interface state TenGigabitEthernet3/0/0.1236 up
vppctl set interface mtu packet 9000 TenGigabitEthernet3/0/0.1236
vppctl set interface ip address TenGigabitEthernet3/0/0.1236 10.0.4.1/30
vppctl set interface ip address TenGigabitEthernet3/0/0.1236 2001:db8:0:4::1/64

vppctl create sub TenGigabitEthernet3/0/0 1237 dot1ad 2345 inner-dot1q 1000 exact-match
vppctl set interface state TenGigabitEthernet3/0/0.1237 up
vppctl set interface mtu packet 9000 TenGigabitEthernet3/0/0.1237
vppctl set interface ip address TenGigabitEthernet3/0/0.1237 10.0.5.1/30
vppctl set interface ip address TenGigabitEthernet3/0/0.1237 2001:db8:0:5::1/64

## The LACP bond
vppctl create bond mode lacp load-balance l2
vppctl bond add BondEthernet0 TenGigabitEthernet3/0/2
vppctl bond add BondEthernet0 TenGigabitEthernet3/0/3
vppctl lcp create BondEthernet0 host-if be0
vppctl set interface state TenGigabitEthernet3/0/2 up
vppctl set interface state TenGigabitEthernet3/0/3 up
vppctl set interface state BondEthernet0 up
vppctl set interface mtu packet 9000 BondEthernet0
vppctl set interface ip address BondEthernet0 10.1.1.1/30
vppctl set interface ip address BondEthernet0 2001:db8:1:1::1/64

vppctl create sub BondEthernet0 1234
vppctl set interface mtu packet 9000 BondEthernet0.1234
vppctl set interface state BondEthernet0.1234 up
vppctl set interface ip address BondEthernet0.1234 10.1.2.1/30
vppctl set interface ip address BondEthernet0.1234 2001:db8:1:2::1/64

vppctl create sub BondEthernet0 1235 dot1q 1234 inner-dot1q 1000 exact-match
vppctl set interface state BondEthernet0.1235 up
vppctl set interface mtu packet 9000 BondEthernet0.1235
vppctl set interface ip address BondEthernet0.1235 10.1.3.1/30
vppctl set interface ip address BondEthernet0.1235 2001:db8:1:3::1/64

vppctl create sub BondEthernet0 1236 dot1ad 2345 exact-match
vppctl set interface state BondEthernet0.1236 up
vppctl set interface mtu packet 9000 BondEthernet0.1236
vppctl set interface ip address BondEthernet0.1236 10.1.4.1/30
vppctl set interface ip address BondEthernet0.1236 2001:db8:1:4::1/64

vppctl create sub BondEthernet0 1237 dot1ad 2345 inner-dot1q 1000 exact-match
vppctl set interface state BondEthernet0.1237 up
vppctl set interface mtu packet 9000 BondEthernet0.1237
vppctl set interface ip address BondEthernet0.1237 10.1.5.1/30
vppctl set interface ip address BondEthernet0.1237 2001:db8:1:5::1/64

Final note

You may have noticed that the [commit] links are all git commits in my private working copy. I want to wait until my previous work is reviewed and submitted before piling on more changes. Feel free to contact vpp-dev@ for more information in the mean time :-)