Bridges and VLANs on Talos 1.10.4

[MichaelTrip]

Hi all,

I am trying to create 2 bridges on 2 vlan interfaces to use Multus in combination with KubeVirt. But somehow it doesn't work.

I have 2 vlans on my ethernet port: vlan20 and vlan1102. VLAN 1102 is also a PVID.

This is my configuration:

    network:
      nameservers:
        - <nameserver1>
        - <nameserver2>
      hostname: <hostname>
      interfaces:
      - interface: eth0
        dhcp: false
        vlans:
        - vlanId: 20
          dhcp: false
        - vlanId: 1102
          dhcp: false

      - interface: br0
        addresses:
          -<ip>/24
        bridge:
          stp:
            enabled: true
          interfaces:
            - eth0.1102
        routes:
            - network: 0.0.0.0/0 # The route's network (destination).
              gateway: <gateway> # The route's gateway (if empty, creates link scope route).
      - interface: br20
        dhcp: false
        bridge:
          stp:
            enabled: true
          interfaces:
            - eth0.20

br0 will be my management interface. That interface works. i can also assign a ip address to my interface br20. But after a minute or 2 it loses connection on the br20 interface. Also, when i connect a pod to my br20 it doesn't have any connectivity.

Does anyone have any experience with this type of setup ?

[ringods]

@MichaelTrip 👋🏻
Curious to hear if you got the VLANs working in combination with Talos & Kubernetes.

[MichaelTrip]

Hi @ringods,

It finally works. In my case i created bridge interfaces that use the vlan tagged interfaces. Then i use multus to get those interfaces to either a pod or a virtual machine.

    network:
      nameservers:
        - 10.99.102.254
        - 2a02:xxxx:xxxx:xxx::254
      hostname: virt1
      interfaces:
      - interface: br0
        bridge:
          stp:
            enabled: true
          interfaces:
            - eth0
        addresses:
          - 10.99.102.1/24
          - 2a02:xxxx:xxxx:xxxx::1/64
        routes:
            - network: 0.0.0.0/0 # The route's network (destination).
              gateway: 10.99.102.254 # The route's gateway (if empty, creates link scope route).
              # metric: 1024
            - network: ::0/0
              gateway: 2a02:xxxx:xxxx:xxx::254
              # metric: 1024
      - interface: eth0
        dhcp: false
        vlans:
          - vlanId: 20
          - vlanId: 99
          - vlanId: 1100
          - vlanId: 1105

      - interface: br20-mainlan
        bridge:
          stp:
            enabled: true
          interfaces: 
            - eth0.20
      - interface: br99-wifi
        bridge:
          stp:
            enabled: true
          interfaces: 
            - eth0.99
      - interface: br1100-testlan
        bridge:
          stp:
            enabled: true
          interfaces: 
            - eth0.1100
      - interface: br1105-dmz
        bridge:
          stp:
            enabled: true
          interfaces: 
            - eth0.1105

As you can see i have 1 untagged interface that is on my management interface. That goes directly to my eth0. The rest of the interfaces are tagged.

I also came across a specific Cilium problem. If you use Cilium in conjunction with this configuration, the vlan's didn't work. That is because Cilium filters vlan's by default. https://docs.cilium.io/en/stable/configuration/vlan-802.1q/. Only if a ebpf program is attached to that interface it will be allowed. Anyway, i just turned the filtering off.

bpf:
  vlanBypass: [ 0 ]

And for multus: this is a typical configuration i use:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: br0-vlan20-mainlan
  namespace: kube-system
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "name": "br0-vlan20-mainlan",
      "plugins": [
        {
          "type": "bridge",
          "bridge": "br20-mainlan",
          "ipam": {}
        }
      ]
    }
---

What this basically does is creating a l2 configuration for your pod / virtualmachine without any ipam mechanisms. So in this case you have to rely on external dhcp.