Trunking VLAN's towards Docker containers in VMware
This is what it will look like: a VM running Rocky Linux with one NIC in trunking mode, having presence on three VLAN’s. VLAN 50 is the main VLAN used for management access and VLAN 701 and VLAN 60 would be used for containerized applications - in my case cantainers will use maclan driver. Additional application VLAN’s can be added later, as needed.
Start by configuring trunking on vCenter port group: Depending on what you need you can trunk all VLAN’s towards the VM (not wise) or you can trunk just specific VLAN ranges:
You might need to set the policies as follows:
Next, reconfigure the NIC on the Rocky VM to work as a trunk:
[root@docker01 network-scripts]# more ifcfg-ens224 NAME=ens224 DEVICE=ens224 ONBOOT=yes NETBOOT="yes" TYPE=Ethernet
Then, configure VLAN 50 interface for management…
[root@docker01 network-scripts]# cat ifcfg-ens224.50 VLAN=yes TYPE=Vlan PHYSDEV=ens224 VLAN_ID=50 REORDER_HDR=yes GVRP=no MVRP=no HWADDR= IPADDR=192.168.50.10 NETMASK=255.255.255.0 GATEWAY=192.168.50.254 DNS1=192.168.50.100 PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=none DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens224.50 UUID=bdfdd998-7359-4b5f-8ae7-7e3b7786bb22 DEVICE=ens224.50 ONBOOT=yes PREFIX=24 RES_OPTIONS="rotate timeout:1 retries:1"
…and VALN701 interface:
[root@docker01 network-scripts]# cat ifcfg-ens224.701 VLAN=yes TYPE=Vlan PHYSDEV=ens224 VLAN_ID=701 REORDER_HDR=yes GVRP=no MVRP=no IPADDR=192.168.70.29 NETMASK=255.255.255.240 GATEWAY=192.168.70.30 DNS1=192.168.70.1 PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=none DEFROUTE=no IPV4_FAILURE_FATAL=no IPV6INIT=no IPV6_AUTOCONF=no IPV6_DEFROUTE=no IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens224.701 UUID=0809f5a6-1b2a-4a37-b664-fa9c46634c92 DEVICE=ens224.701 ONBOOT=yes PREFIX=28 RES_OPTIONS="rotate timeout:1 retries:1"
The same configuration should be performed for VLAN 60 interface adjusting settings as needed.
At this point there should be three functioning interfaces, ens224.50, ens224.60 and ens224.701. Next, you need to configure symmetric routing. Rather than taking a stab at explanation, here is a pretty good one instead.
So, in my case the following rule-* and route-* files are needed for VLAN 50 and VLAN 701. You need to perform similar configuration for any other VLAN’s you will need.
The following is routing for VLAN 50. Note the default entry; this is the management interface:
[root@docker01 network-scripts]# cat route-ens224.50 192.168.50.0/24 dev ens224.50 src 192.168.50.10 table rt50 default via 192.168.50.254 table rt50
Next, I specify rule under which the above routes will be utilized:
[root@docker01 network-scripts]# cat rule-ens224.50 from 192.168.50.10 prio 50 table rt50
Similarly the following files deal with VLAN701. Note the absence of default entry:
[root@docker01 network-scripts]# cat rule-ens224.701 from 192.168.70.29 prio 70 table rt701
[root@docker01 network-scripts]# cat route-ens224.701 192.168.70.16/28 dev ens224.701 src 192.168.70.29 table rt701
Now, I need to make sure the alternate routing tables are defined in rt_tables. This is simply a mapping file that say “a number maps to this friendly name”.
[root@docker01 network-scripts]# cat /etc/iproute2/rt_tables # # reserved values # 255 local 254 main 253 default 0 unspec # # local # #1 inr.ruhep 50 rt50 60 rt60 70 rt701
There are more details on the content of two files here.
Like me, you might need to install NetworkManager-dispatcher-routing-rules package that will allow NetworkManager process the route-* and rule-* files.
Finally, below is a snippet from docker-compose.yml. Note IP address assignment for interface VLAN 701 and its definition towards the bottom utilizing macvlan driver.
version: "3" services: nginx: container_name: nginx image: nginx:mainline-alpine restart: always ports: - 80:80 networks: db: vlan701: ipv4_address: 192.168.70.28 ... networks: vlan701: name: VLAN701 dmz-app to expose external apps driver: macvlan driver_opts: parent: ens224.701 ipam: config: - subnet: 192.168.70.16/28 gateway: 192.168.70.30
Depending on the environment you might need to fiddle with rp_filter, see more info below:
2 Nics with 2 different Gateway
When RHEL has multiple IPs configured, only one is reachable from a remote network. Or why does RHEL ignore packets when the route for outbound traffic differs from the route of incoming traffic?