Open vSwitch: Connecting VMs Using Tunnels

The goal of this configuration cookbook is to allows VMs on two different hosts to communicate over port-based GRE tunnels.

Environment

This configuration cookbook assumes an environment configured as described in the sections below.

Two Physical Networks

• Transport Network: This Ethernet network is for tunnel traffic between hosts running OVS. Depending on the tunneling protocol being used (this cookbook uses GRE), some configuration of the physical switches may be required (for example, it may be necessary to adjust the MTU). Configuration of the physical switching hardware is outside the scope of this cookbook entry.
• Management Network: Strictly speaking this network is not required, but it is a simple way to give the physical host an IP address for remote access since an IP address cannot be assigned directly to a physical interface that is part of an OVS bridge.

Two Physical Hosts

The environment assumes the use of two hosts, named Host1 and Host2. Both hosts are hypervisors running Open vSwitch. Each host has two NICs, eth0 and eth1, which are configured as follows:

• eth0 is connected to the Transport Network. eth0 has an IP address that is used to communicate with Host2 over the Transport Network.
• eth1 is connected to the Management Network. eth1 has an IP address that is used to reach the physical host for management.

Four Virtual Machines

Each host will run two virtual machines (VMs). VM1 and VM2 are running on Host1, while VM3 and VM4 are running on Host2.
Each VM has a single interface that appears as a Linux device (e.g., "tap0") on the physical host. (Note: for Xen/XenServer, VM interfaces appears as Linux devices with names like "vif1.0". Other Linux systems may present these interfaces as "vnet0", "vnet1", etc.) The IP addresses assigned to the network interfaces within these VMs are all assumed to be on the same subnet.
This diagram graphically illustrates the environment assumed by this configuration cookbook:

Configuration Steps

Before you begin, you'll want to ensure that you know the IP addresses assigned to eth0 on both Host1 and Host2, as they will be needed during the configuration.
Perform the following configuration on Host1:

1. Create an OVS bridge:
   ovs-vsctl add-br br0
   Note that you will not add eth0 to the OVS bridge.
2. Start (launch) VM1 and VM2 on Host1. If the VMs are not automatically attached to OVS, add them to the OVS bridge you just created (the commands below assume tap0 is for VM1 and tap1 is for VM2):
   ovs-vsctl add-port br0 tap0
ovs-vsctl add-port br0 tap1
3. Add a port for the GRE tunnel:
   ovs-vsctl add-port br0 gre0 -- set interface gre0 type=gre options:remote_ip=<IP address of eth0 on Host2>

Create a mirrored configuration on Host2 using the same basic steps:

1. Create an OVS bridge, but do not add any physical interfaces to the bridge:
   ovs-vsctl add-br br0
2. Launch VM3 and VM4 on Host2, adding them to the OVS bridge if needed (again, tap0 is assumed to be for VM3 and tap1 is assumed to be for VM4):
   ovs-vsctl add-port br0 tap0
ovs-vsctl add-port br0 tap1
3. Create the GRE tunnel on Host2, this time using the IP address for eth0 on Host1 when specifying the remote_ip option: ovs-vsctl add-port br0 gre0 -- set interface gre0 type=gre options:remote_ip=<IP address of eth0 on Host1>

Testing

Pings between any of the VMs should work, regardless of whether the VMs are running on the same host or different hosts.
Using ip route show (or equivalent command), the routing table of the operating system running inside the VM should show no knowledge of the IP subnets used by the hosts, only the IP subnet(s) configured within the VM's operating system. To help illustrate this point, it may be preferable to use very different IP subnet assignments within the guest VMs than what is used on the hosts.

Troubleshooting

If connectivity between VMs on different hosts isn't working, check the following items:

• Make sure that Host1 and Host2 have full network connectivity over eth0 (the NIC attached to the Transport Network). This may necessitate the use of additional IP routes or IP routing rules.
• Make sure that gre0 on Host1 points to eth0 on Host2, and that gre0 on Host2 points to eth0 on Host1.
• Ensure that all the VMs are assigned IP addresses on the same subnet; there is no IP routing functionality in this configuration.