Networking with libvirt - The next level
In a previous article, I mentioned that I wanted to have a separate network as a DMZ. So on the host, I got a second bridge, and given that it was using a different NIC, I could use differnt VLANs using the managed switch behind it.
This worked quite well for a while. But as I wanted to extend this possibility to separate the networks even further, in smaller VLANs, this method proved to be quite problematic.
The initial requirement
My initial requirement was simple:
have several VLANs on one machine, each separated from each other, to prevent certain VMs to talk to others
I wanted to get this working using libvirt given that is what I use to manage my VMs. This may seem like an odd requirement to put, but I found that quite some articles will explain you how to use virtual interfaces or bridges with qemu, not everything applies out of the box with libvirt.
The research
So I went out there and tried to figure out what would be required to allow multiple VLANs on a single machine, have libvirt recognize it all, and have well separated networks.
Well… this wasn’t an easy ride.
All started with this RedHat article. I thought that I could make the “one bridge with VLAN filtering” work. I kind of did but I soon realized that it wasn’t really for me. My issue was that I couldn’t filter the traffic as I wanted, it was a “all pass or nothing does” kind of deal. Maybe it would have been possible if I was to implement the rules on the host firewall, but I realized that I didn’t want to maintain my global firewall and a firewall on the host.
So came my refined requirement!
The refined requirement
same as above, but I want to be able to manage the traffic through the global firewall
This implies a lot of things! The traffic exiting the physical interface(s) must be already tagged and the rest of the network appliances must follow!
The implementation
So after much research, trial and errors, I managed to have the following working:
+---------+ +---------+ +---------+
| VM1 | | VM2 | | VM3 |
| +-----+ | | +-----+ | | +-----+ |
| | et1 | | | | et2 | | | | et3 | |
+-+-----+-+ +-+-----+-+ +-+-----+-+
| +---------+ | |
+------| brd1 |------+ +-------+
+---------+ | brd2 |
| +-------+
| |
vlan200 vlan201
| |
+-----------------------------------+
|
bond0
| |
eno1-----+ +-----eno2
So, I have a bond interface (bond0) on top of two NICs (eno1 and eno2) in a “active-backup” mode (so if one NIC dies, the other one takes over).
On top of this bond, I am creating virtual interfaces with a tagged VLAN ID (vlan200 and vlan201).
I’m attaching a bridge on this VLAN interface so that I can have several VMs attached “in the same VLAN” (e.g. VM1 and VM2).
I’m not managin the et1, et2 or et3, those are created dynamically by libvirt when starting the VM in bridge mode.
Netplan
I’m using Ubuntu on my host (because ZFS) so I used netplan to manage all this. I’m not a huge fan of it but I managed to get it working, so why not. The nice thing though was that I could separate in different files what I needed.
Therefore, for bond creation, I have a file explaining the interfaces to bond (0002-interface-bonding.yaml):
1network:
2 version: 2
3 renderer: networkd
4 bonds:
5 bond0:
6 dhcp4: false
7 link-local: [ ipv4 ]
8 interfaces:
9 - eno1
10 - eno2
11 parameters:
12 mode: active-backup
13 primary: eno1
14 mii-monitor-interval: 1
15 gratuitious-arp: 5
Then, I have several files to create the VLAN aware virtual interfaces (0003-vlan200.yaml) with the rquired VLAN ID:
1network:
2 vlans:
3 vlan200-int:
4 id: 200
5 link-local: [ ipv4 ]
6 link: bond0
7 dhcp4: false
And finally, I have the bridge interface to allow attach to the VLAN interface and for the VMs to connect to (0004-vlan200-bridge.yaml):
1network:
2 bridges:
3 br-int:
4 interfaces:
5 - vlan200-int
6 mtu: 1500
7 parameters:
8 stp: true
9 forward-delay: 2
10 dhcp4: false
11 addresses:
12 - 10.0.200.2/24
13 routes:
14 - to: default
15 via: 10.0.200.1
16 nameservers:
17 addresses:
18 - 10.0.200.1
Please note that this bridge is a bit special because it acts as a default gateway globally, even though the other networks will not use it. This allows to update the host and other maintenance things (I know I could use the management interface for that but I didn’t want to for probably wrong reasons…).
So when I need/want a new VLAN, I simply create a new “VLAN interface” and bridge linked to that and here I go!
The switch
Given that my plan is to manage the network through the firewall, all the packets must be brought to the firewall with their VLAN ID. So this means that whenever I need to create a new VLAN on the host, not only should I add the necessary netplan files, but I also must update my switch to allow tagged packets on the ports which arrive to eno1 and eno2.
I am not a network specialist, I know enough to get going but not enough to have “advanced” networking concepts work from the first try just by imaging the architecture in my head. So there has been quite some trial and errors (again) due to the following aspect: tagged ports.
The receiving port in the switch, so the one where the cable comes from the switch to the eno1 and eno2 NICs, must be set to tagged. Otherwise, the VLAN IDs will be stripped and the firewall will not get what’s happening.
Another important aspect is that both switch receiving ports must be configured indentically given that behind, on the host, we’re using a bond interface (that one, I didn’t struggle with… yes, I’m proud of myself ;-)).
The firewall
Here, it will depend on your firewall. For me (OPNsense), it’s pretty simple:
- create new interface
- assign interface
- filter traffic
The virtualization
As I’m using a bridge on top of the VLAN interface for each VLAN, I can simply use the brdige mode of libvirt which simplifies a lot the libvirt integration in all this: libvirt and the VMs are unaware of the VLANs behind the scenes.
So when I create a VM, I will use the following:
1virt-install ... --network bridge=br-int,model=virtio ...
I could also imagine having VMs with multiple interfaces, with each of then connected to a different bridge interface.
Final thoughts
There may be better ways, more advanced, flexible, optimized… But this one works for me.
The concepts are simple enough for me to use, extent, debug. There’s only one place where the filtering happens (my global firewall), so this solution reduces redundant configuration…
On a final note, I wanted to point to an article I read which is a fantastic resource regarding VLANs, Linux bridges, vETH… I tested it, got it to work (again, proud) but I quickly found that given that all happens on the host (bridge), I cannot (at least, I didn’t manage to make it work) upstream the VLAN IDs to the firewall.