Multiple bridges for KVM virtualization

In a previous post, I went over how to create one brdige interface for the hosting of VMs.

Now I didn’t feel confortable having one VM exposed to the Internet in the same internal network than my other VMs.Therefore, I wanted to go for a more professionally approach: DMZ!

So, given that my host has 4 networking interfaces, I plugged another cable in it, added a DMZ VLAN with proper restrictions and searched the internet for a way to create another bridged interface for KVM, in case I needed to create more VMs in that DMZ.

And I couldn’t find any that applied to KVM hosting! Here is how I managed to pull it off.

Netplan

The first thing to do is to modify the netplan configuration file to include a new bridge called br-dmz:

network:
  renderer: networkd
  ethernets:
    eno1:
      dhcp4: false
      dhcp6: false
    eno2:
      dhcp4: false
      dhcp6: false
    eno3:
      dhcp4: true
    eno4:
      dhcp4: true
  version: 2

  bridges:
    br0:
      addresses: [10.0.20.2/24]
      routes:
        - to: default
          via: 10.0.20.1
          metric: 100
      mtu: 1500
      nameservers:
        addresses:
          - 10.0.20.1
      parameters:
        stp: true
        forward-delay: 2
      dhcp4: false
      dhcp6: false
      interfaces: [eno1]
    br-dmz:
      addresses: [10.0.21.2/24]
      mtu: 1500
      nameservers:
        addresses:
          - 10.0.25.1
      parameters:
        stp: true
        forward-delay: 2
      dhcp4: false
      dhcp6: false
      interfaces: [eno2]

The attentive reader has noticed that we don’t have a default route with br-dmz. And netplan will not allow you too! This actually makes sense because the default route is suppposed to be the complete default for all routes so another definition would just result in confusion.

The problem

So I tried this configuration, I was happy, I could reach my DMZ VM, ping the router, tried the NFS shares… but wait! Sometimes, the NFS shares would just hang, and sometimes it would work nicely. What is happening?

Well, the issue was that the packets from the DMZ were leaving via br-dmz but were sometimes, and sometimes only, come back via the br0 interface! This is probably because br0 contains the default route, which would mean that it’s the default for the out but also for the in (?).

To confirm this, I used tcpdump on both interfaces on the 2049 port and accessing an NFS share and could sometimes see packets destined to the DMZ network on the br0 interface.

The solution

The solution came from a post that I read, and basically said this: on Linux, it is possible to tag the packets when leaving so that when they come back, the kernel does force it back to the interface which is part of the tagging. Linux is so awesome, it’s so satisfying knowing you can go deeper and solve such a tricky problem!

So, first, we need to create a new routing table which will hold the tagging. Create a file /etc/iproute2/rt_tables.d/tagging_br-dmz_to_assure_return.conf and add the following:

200 dmz

So now, we can create the rules that will say: “when coming from this interface, use the 200 dmz routing table (tag) and when coming back on this interface, use the same routing table”. Also, now that we define a new routing, we can add a default route for that table only!

1user $ sudo ip rule add from 10.0.21.0/24 table dmz prio 1
2user $ sudo ip rule add to 10.0.21.0/24 table dmz prio 1
3user $ sudo ip route add default via 10.0.21.1 dev br-dmz table dmz

Making this persistent

I have not yet tested this given that I rarely restart my host (I have kernel live patching), but I have implemented it and if you see this message, this probably means that it’s working, otherwise, I would correct the following.

Create a shell script with the ip commands /usr/local/sbin/dmz_ip_commands.sh (don’t forget to chmod +x the script):

1#!/bin/bash
2ip rule add from 10.0.21.0/24 table dmz prio 1
3ip rule add to 10.0.21.0/24 table dmz prio 1
4ip route add default via 10.0.21.1 dev br-dmz table dmz

Create a systemd unit file /etc/systemd/system/dmz-multiple-bridges-ip-commands.service (don’t forget to reload the systemd daemon):

 1[Unit]
 2Description=DMZ bridge ip commands
 3After=network.target
 4
 5[Service]
 6User=root
 7ExecStart=/usr/local/sbin/dmz_ip_commands.sh
 8Type=oneshot
 9
10[Install]
11WantedBy=multi-user.target