Bridged Guest Configuration on Fedora 12 with Netcf

dale's picture

A revolutionary new feature of Fedora 12 is "Network Interface Management" via a library called netcf. With this library as the facilitator, it becomes possible for any tool, not just NetworkManager, to reliably and methodically configure network interfaces. The original impetus for this feature was to enable tools like oVirt and libvirt to manipulate the networks on the hosts which underlie virtual machines, and that's what I'll discuss below.

Netcf comes with a command called, ncftool, but for virtualization networks there is no need to use it directly. The virsh command line tool and the Virtual Machine Manager GUI are the primary tools which leverage libvirt, and by extension in this case, netcf. New libvirt features are quickly found in the closely bound virsh tool, but often take some time to show up in the python-based virt-manager. This is true with network interface configuration.

Before going any further, let's discuss libvirt networking.

libvirt Networking

Libvirt has long supported the ability to create a network configuration, beginning with release 0.2.0. Networks are typically virtual, in that they exist only on the host and guests. Their connection to physical networks is through NAT or routing performed by the host.

A feature of Fedora 13 will be "Shared Network Interface" which allows guests to exist on the same physical network as other physical (or virtual) machines. In truth, this has long been possible, but until the creation of netcf it wasn't trivial for a virtualization management tool to do so reliably. Such a configuration had to be created by hand. With Fedora 12, we are somewhere in the middle.


Default Virtual Networking uses NAT

The default mode of libvirt networking is network address translation or NAT. In this configuration, when libvirtd is started, a private network (192.168.122.0/24) is automatically created along with a copy of dnsmasq and dhcpd. Guests will automatically be assigned an IP address from this range and any traffic they send to the internet will appear to come from the host server's IP address.

The changes to the host machine are evident in the appearance of a virbr0 bridge interface and some changes to iptables. Stop libvirtd or 'virsh net-destroy default' and they go away until libvirtd is started again.

[root@tofu2 ~]# brctl show
bridge name	bridge id		STP enabled	interfaces
virbr0		8000.2211fa614c48	yes		vnet0

[root@tofu2 ~]# ifconfig virbr0
virbr0    Link encap:Ethernet  HWaddr 22:11:FA:61:4C:48  
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:10492 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9868 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1199572 (1.1 MiB)  TX bytes:14008168 (13.3 MiB)

[root@tofu2 ~]# iptables -t nat -n -L POSTROUTING
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  192.168.122.0/24    !192.168.122.0/24   

The details of this default network can be seen in the host defails view of virt-manager:

Virtual machines, or guests, are referred to by libvirt as domains (a hold over from Xen) and the commands such as 'define', 'destroy', etc. which operate on domains have a "net-" conterpart; 'net-define', 'net-destroy'. The same is true for other objects like storage pools, storage volumes, encryption secrets, and node devices. The term "node" comes from the oVirt project and refers to the physical host machine.

As with all things libvirt, this default network is defined in XML.


[root@tofu2 ~]# virsh net-list
Name                 State      Autostart
-----------------------------------------
default              active     yes       

[root@tofu2 ~]# virsh net-dumpxml default
<network>
  <name>default</name>
  <uuid>eca9f38f-4040-43e7-96ca-3077fca56325</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0' />
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254' />
    </dhcp>
  </ip>
</network>

Commands such as 'net-define' can be thought of as meaning, "create a configuration for", 'net-start' as "instantiate the configuration (or 'service virtual network start' perhaps)", and 'net-destroy' as "shutdown the vritual network, but leave the configuration intact".

[root@tofu2 ~]# virsh help |grep net-
    net-autostart   autostart a network
    net-create      create a network from an XML file
    net-define      define (but don't start) a network from an XML file
    net-destroy     destroy a network
    net-dumpxml     network information in XML
    net-edit        edit XML configuration for a network
    net-list        list networks
    net-name        convert a network UUID to network name
    net-start       start a (previously defined) inactive network
    net-undefine    undefine an inactive network
    net-uuid        convert a network name to network UUID

One way to change a network configuration would be through a series of commands such as:

  1. virsh net-dumpxml default > default.xml
  2. vi default.xml
  3. virsh net-destroy default
  4. virsh net-define default.xml
  5. virsh net-start default


Configuring Routed or Forwarded libvirt Guest Networking

A virtual network may also connect to the "real world" by using the host as a forwarding router without any network address translation. In this case, it is assumed the physical network knows to route all packets destined for 192.168.122.0/24 to the physical host where they will then be relayed to the guests.


Creating a Routed Network

Creating a routed network can begin by exporting the default NAT network.

[root@tofu2 ~]# virsh net-dumpxml default > net-routed.xml

Edit net-routed.xml and change it as follows. Note the new bridge is named 'virbr1', so the old network on virbr0 will remain available to guests.

<network>
  <name>routed</name>
  <forward mode="route" dev="eth0"/>
  <bridge name='virbr1' stp='on' delay='0' />
  <ip address='192.168.123.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.123.2' end='192.168.123.254' />
    </dhcp>
  </ip>
</network>

Now make it known to libvirt by defining it, and then list all networks both active (started) and inactive (destroyed or never started).

[root@tofu2 ~]# virsh net-define net-routed.xml
Network routed defined from net-routed.xml

[root@tofu2 ~]# virsh net-list --all
Name                 State      Autostart
-----------------------------------------
default              active     yes       
routed               inactive   no    

[root@tofu2 ~]# ifconfig virbr1
virbr1: error fetching interface information: Device not found

Notice there is no change made to the host until we start or make active the virtual network. We can also flag this new network to automatically start when libvritd starts.

[root@tofu2 ~]# virsh net-start routed
Network routed started

[root@tofu2 ~]# virsh net-autostart routed
Network routed marked as autostarted

[root@tofu2 ~]# virsh net-list --all
Name                 State      Autostart
-----------------------------------------
default              active     yes       
routed               active     yes     

[root@tofu2 ~]# ifconfig virbr1
virbr1    Link encap:Ethernet  HWaddr CE:39:48:A4:30:21  
          inet addr:192.168.123.1  Bcast:192.168.123.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:2452 (2.3 KiB)

The network is now visible within virt-manager, and available for use when creating new guests. As of version virt-manager-0.8.0-7.fc12 the network is erroneously described as "NAT to physical device eth0". Cole Robinson has already patched up a fix for that.


Moving a Guest to the Routed Network

Now that the networking is in place, moving an existing guest to it requires manual configuration. Begin by shutting down the guest, and then edit the XML configuration which defines the domain.


[root@tofu2 ~]# virsh shutdown f12-test
[root@tofu2 ~]# virsh edit f12-test

Change the "source network='default'" attribute to "source network='routed'". If you are feeling adventerous, you can create an interface for each network eth1 by duplicating this section.

...
    <interface type='network'>
      <mac address='52:54:00:55:f6:c4'/>
      <source network='routed'/>
      <model type='virtio'/>
    </interface>
...

After saving those changes, start the guest. Assuming it uses DHCP, it will automatically recieve an IP in our new 192.168.123.0/24 subnet. Remember, that unless the router upstream from your host knows you are responsible for 192.168.123.0/24, your guests will not recieve any responses to their packets. Perhaps you can perform a 'tcpdump -ni eth0 icmp' on a peer to the host and then ping it from the guest. You should see the icmp requests being forwarded by the host to your peer.


Configuring Bridged libvirt Guest Networking

Both NAT and routed types of networking give guests a different relationship to the physical network than that which the host has. What if you want the domain to "share" the physical network interface card with the host? In other words, what if you want the guest to be visible to other nearby physical machines on the same broadcast domain or IP subnet?

Such a configuration requires a bridge, but this bridge must now be "physical" rather than "virtual". That is, functionally known to the host and interactive with other physical hosts or devices. This is where netcf comes into play.


Creating a Bridge Interface on a Host using virsh

Note: Be sure to disable NetworkManager and enable the network service, or you will find only eth0 and no bridge after a reboot. It's probably a simple matter to inform NetworkManager of our changes so that it can remain enabled, but I don't typically need it on machines which host virtual guests anyway.

It is possible to create bridges using the ncftool, but it's best to stay consistent and let virsh do that through the netcf-libs for us.

In this example the host has 1 interface named eth0. A bridge named br0 will be created which will assume the IP previously used by eth0. To keep things simple, let's destroy any running networks and remove all bridges.

[root@tofu2 ~]# virsh net-destroy routed
Network routed destroyed
[root@tofu2 ~]# virsh net-destroy default
Network default destroyed
[root@tofu2 ~]# brctl show
bridge name	bridge id		STP enabled	interfaces

Take a look at the configs and a make backup of the way things are to begin with.

[root@tofu2 ~]# cd /etc/sysconfig/network-scripts/
[root@tofu2 network-scripts]# ls ifcfg-*
ifcfg-eth0  ifcfg-lo

[root@tofu2 network-scripts]# cat ifcfg-eth0
# Intel Corporation 82567LM-3 Gigabit Network Connection
DEVICE=eth0
HWADDR=00:24:E8:30:20:E7
ONBOOT=yes
IPADDR=10.10.10.158
BOOTPROTO=none
NETMASK=255.255.255.0
DNS2=10.10.215.161
TYPE=Ethernet
GATEWAY=10.10.10.254
DNS1=10.10.215.131
NM_CONTROLLED=yes
IPV6INIT=no
USERCTL=no
PREFIX=24
NAME="System eth0"
UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03

[root@tofu2 network-scripts]# mkdir bak
[root@tofu2 network-scripts]# cp -p ifcfg-* bak

Previously you would create an ifcfg-br0 configuration file and modify ifcfg-eth0, but let's do it using netcf by way of virsh and libvirt. The virsh network interfaces commands start with 'iface-'.

[root@tofu2 ~]# virsh help | grep iface
    iface-list      list physical host interfaces
    iface-name      convert an interface MAC address to interface name
    iface-mac       convert an interface name to interface MAC address
    iface-dumpxml   interface information in XML
    iface-define    define (but don't start) a physical host interface from an XML file
    iface-undefine  undefine a physical host interface (remove it from configuration)
    iface-edit      edit XML configuration for a physical host interface
    iface-start     start a physical host interface (enable it / "if-up")
    iface-destroy   destroy a physical host interface (disable it / "if-down")

The iface-dumpxml command will create an XML representation of the ifcfg-eth0 file. Take a look at it, and create a copy.

[root@tofu2 network-scripts]# virsh iface-dumpxml eth0
<interface type='ethernet' name='eth0'>
  <start mode='onboot'/>
  <mac address='00:24:E8:30:20:E7'/>
  <protocol family='ipv4'>
    <ip address='10.10.10.158' prefix='24'/>
    <route gateway='10.10.10.254'/>
  </protocol>
</interface>

[root@tofu2 network-scripts]# virsh iface-dumpxml eth0 > bak/eth0.xml

Notice that the virtual bridge interfaces virbr*, automatically created from libvirt network objects, do not show up in an interface listing.

[root@tofu2 network-scripts]# virsh iface-list
Name                 State      MAC Address
--------------------------------------------
eth0                 active     00:24:e8:30:20:e7
lo                   active     00:00:00:00:00:00

Time to create a bridge. Copy the XML from bak/eth0.xml to br0.xml, and make the following modifications.

<interface type='bridge' name='br0'>
  <start mode='onboot'/>

  <protocol family='ipv4'>
    <ip address='10.10.10.158' prefix='24'/>
    <route gateway='10.10.10.254'/>
  </protocol>

  <bridge>
    <interface type='ethernet' name='eth0'>
      <mac address='00:24:E8:30:20:E7'/>
    </interface>
  </bridge>

</interface>

Define the bridge interface to libvirt. At this point there is an error message, but it may be ignored.

[root@tofu2 network-scripts]# virsh iface-define br0.xml
error: Failed to define interface from br0.xml
error: invalid argument in virGetInterface

Notice the changes made to ifcfg-eth0, and the new ifcfg-br0 interface configuration file. At this point we have a "physical" bridge configuration which exists independently of libvirtd. It will start when the network service starts rather than only when libvirtd does.

[root@tofu2 network-scripts]# cat ifcfg-eth0
DEVICE=eth0
HWADDR=00:24:E8:30:20:E7
ONBOOT=yes
BRIDGE=br0
TYPE=Ethernet
BOOTPROTO=dhcp
PEERDNS=yes
PEERROUTES=yes
NAME="System eth0"
UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03

[root@tofu2 network-scripts]# cat ifcfg-br0
DEVICE=br0
ONBOOT=yes
TYPE=Bridge
BOOTPROTO=none
IPADDR=10.10.10.158
NETMASK=255.255.255.0
GATEWAY=10.10.10.254

At this point our eth0 is missing from the interface list, that's because from the libvirt / netcf perspective (which is just looking at the ifcfg files) we have replaced it with the br0 interface. However, br0 is not started, and it isn't listed either.

[root@tofu2 network-scripts]# virsh iface-list
Name                 State      MAC Address
--------------------------------------------
lo                   active     00:00:00:00:00:00

Restart the host network to bring up the new interface which will now own our IP address.

[root@tofu2 network-scripts]# service network restart
Shutting down interface eth0:  bridge br0 does not exist!
                                                           [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Disabling IPv4 packet forwarding:  net.ipv4.ip_forward = 0
                                                           [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface eth0:                                [  OK  ]
Bringing up interface br0:                                 [  OK  ]

[root@tofu2 network-scripts]# virsh iface-list
Name                 State      MAC Address
--------------------------------------------
br0                  active     00:24:e8:30:20:e7
lo                   active     00:00:00:00:00:00

[root@tofu2 network-scripts]# ifconfig
br0       Link encap:Ethernet  HWaddr 00:24:E8:30:20:E7  
          inet addr:10.10.10.158  Bcast:10.10.10.255  Mask:255.255.255.0
          inet6 addr: fe80::224:e8ff:fe30:20e7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:11089 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9420 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1328550 (1.2 MiB)  TX bytes:1891115 (1.8 MiB)

eth0      Link encap:Ethernet  HWaddr 00:24:E8:30:20:E7  
          inet6 addr: fe80::224:e8ff:fe30:20e7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1194296 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1045144 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:160987318 (153.5 MiB)  TX bytes:243170160 (231.9 MiB)
          Memory:fdfe0000-fe000000 
...


Moving a Guest to Bridged Network

Guest interfaces may be assigned to virtual networks or to host interfaces. We now have a permanent bridge that can be assigned to a guest. Shutdown the guest and use virsh edit to modify the interface element.

By default we have this:

[root@tofu2 network-scripts]# virsh dumpxml f12-test | grep -A 3 interface
    <interface type='network'>
      <mac address='52:54:00:55:f6:c4'/>
      <source network='routed'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
    </interface>

[root@tofu2 network-scripts]# virsh edit f12-test
Domain f12-test XML configuration edited.

[root@tofu2 network-scripts]# virsh dumpxml f12-test | grep -A 3 interface
    <interface type='bridge'>
      <mac address='52:54:00:55:f6:c4'/>
      <source bridge='br0'/>
      <model type='virtio'/>
    </interface>

The virt-manager doesn't yet allow you to modify a NIC and reassign it to a different network or interface, but it is possible to use virt-manager to add a new NIC, and remove the old NIC.

Start the guest and see that it now is on the same network as the host.


[dlbewley@f12-test ~]$ ifconfig
eth0      Link encap:Ethernet  HWaddr 52:54:00:55:F6:C4  
          inet addr:10.10.10.159  Bcast:10.10.10.255  Mask:255.255.255.0
          inet6 addr: fe80::5054:ff:fe55:f6c4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:32 errors:0 dropped:0 overruns:0 frame:0
          TX packets:25 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:4480 (4.3 KiB)  TX bytes:4332 (4.2 KiB)

Let's now re-enable our NAT and routed networks for simultaneous use.


Renabling Default and Routed Network

The previous default NAT network and our created routed networks can still be used with our new physical bridge, however they must be modified. Using 'virsh net-edit' change any occurance of 'eth0' to 'br0', and then start the networks using 'virsh net-start'. The routed network in the above example names eth0 and must be modified, the default network uses an implicite interface and does not require a change.

[root@tofu2 ~]# virsh net-dumpxml routed
<network>
  <name>routed</name>
  <uuid>7c349391-9637-fbaa-8b61-b9f82c993ae4</uuid>
  <forward dev='br0' mode='route'/>
  <bridge name='virbr1' stp='on' delay='0' />
  <ip address='192.168.123.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.123.2' end='192.168.123.254' />
    </dhcp>
  </ip>
</network>

[root@tofu2 ~]# virsh net-dumpxml default
<network>
  <name>default</name>
  <uuid>eca9f38f-4040-43e7-96ca-3077fca56325</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0' />
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254' />
    </dhcp>
  </ip>
</network>

With all three of these forms of networking in enabled there are one physical bridge, two virtual bridges (one NAT, one routed). Each of the three optionsare selectable for interface assignment in the new machine wizard.

[root@tofu2 ~]# brctl show
bridge name	bridge id		STP enabled	interfaces
br0		8000.0024e83020e7	no		eth0
virbr0		8000.3ea5c9e0d5a2	yes		vnet0
virbr1		8000.000000000000	yes	

Eventually the Shared Network Interface feature will help automate this process.


References

AttachmentSize
libvirt-network-0.png37.49 KB
libvirt-network-routed.png38.36 KB
libvirt-network-selection.png37.49 KB
libvirt-network-add-nic.png112.06 KB