Funky Routing
From OptionC
| Table of contents |
Introduction
This has been tested on Xen 2.0.7-1 with Shorewall 3.0.3-1. This Howto presupposes you already have a base Xen install. If not, and you are unfamiliar with Xen, then you probably want to start with the Xen_Debian_Quick_Start. If you do have such a system and you used the default, networking should be running relatively seemlessly. These configurations are only if you want to switch to routed networking (and have a bit more control). There is nothing particularly funky about them, but I haven't had a chance to change it, and that was the first thing that came off of my fingers when I created this page.
Things you need for all configurations covered on this page
The following configurations have the following in common. First VIF traffic is routed, not bridged, so /etc/xen/xend-config has "(network-script network)" instead of "(network-script network-route)." Second, you need to add a few packages.
Required Packages
- Xen and friends, including a xen kernel with the firewalling modules, such as kernel-image-2.6.11-msxen0 or kernel-image-2.6.11-ksxen0. If you followed our quickstart, you probably already have the following in your /etc/apt/source.list:
deb http://www.option-c.com/debian/ unstable main
However, that quickstart gives you the basic kernel, not the one you need for these configurations. As such...
# apt-get install kernel-image-2.6.11-msxen0
Check grub and all that, and boot into the xen kernel. If you were already running kernel-image-2.6.11-ocxen0, you shouldn't have any problem booting this one with the same grub configuration (except, of course, the name of the kernel image).
- Bridge-utils, iproute: If you installed xen through any of the normal ways (official Debian, our packages or many of the other packagers), bridge-utils and iproute should have been installed as dependencies. To make sure, do something like
# dpkg --status bridge-utils | grep Status Status: install ok installed # dpkg --status iproute | grep Status Status: install ok installed
If they don't show up as installed,
# apt-get install iproute bridge-utils
- Shorewall is required for the first two, and strongly recommended for the final one (also within the domUs). These directions refer to version 3.x.x, which is currently in etc/testing, not sarge/stable. Make sure you have something like the following in /etc/apt/sources.list
deb ftp://ftp.us.debian.org/debian/ etch main
# apt-get update # apt-get -t testing shorewall
(If you just added the line to /etc/apt/sources.list to get the package, you probably want to remove it now)
/etc/xen/xend-config.sxp
The critical change from the default is that "network-script" will be "network-route", not "network"
# /etc/xen/xend-config.sxp # Xend configuration file. # Port xend should use for the HTTP interface. (xend-port 8000) # Port xend should use for the event interface. (xend-event-port 8001) # Address xend should listen on for HTTP connections. # Specifying 'localhost' prevents remote connections. # Specifying the empty string '' allows all connections. (xend-address 'localhost') # The port xend should start from when allocating a port # for a domain console. (console-port-base 9600) # Address xend should listen on for console connections. # Specifying 'localhost' prevents remote connections. # Specifying the empty string '' allows all connections. (console-address 'localhost') ## Use the following if VIF traffic is routed. # The script used to start/stop networking for xend. (network-script network-route) # The default script used to control virtual interfaces. #(vif-script vif-route) ## Use the following if VIF traffic is bridged. # The script used to start/stop networking for xend. #(network-script network) # The default bridge that virtual interfaces should be connected to. (vif-bridge xen-br0) # The default script used to control virtual interfaces. (vif-script vif-bridge) # Whether iptables should be set up to prevent IP spoofing for # virtual interfaces. Specify 'yes' or 'no'. (vif-antispoof no) # Setup script for file-backed block devices (block-file block-file) # Setup script for enbd-backed block devices (block-enbd block-enbd)
Routing + Internal Bridge + Shorewall
Single external IP addresses, multiple domUs which need internet access
This is for a basic situation where you have one server with a single IP address, and you want a gaggle of Xen domUs to have (controlled) access to the internet. The basics of this configuration also apply if you have multiple IP addresses, need to do port forwarding, et cetera, the only difference is in how you configure Shorewall.
If your system is already running xend and some domains, it's probably best to stop them now.
# xendomains stop # xend stop
And remove the bridge as configured
# brctl delbr xen-br0
/etc/network/interfaces in dom0
Switch it so you add the bridge for the internal network "manually" (as opposed to through the xend scripts)
# This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo iface lo inet loopback # The primary network interface # (Will be a valid external IP address) auto eth0 iface eth0 inet static address 10.88.88.66 netmask 255.255.255.0 gateway 10.88.88.1 # The Xen bridged network # (An rfc1918 IP address) auto xen-br0 iface xen-br0 inet static pre-up brctl addbr xen-br0 address 10.66.66.66 netmask 255.255.255.0 network 10.66.66.0 broadcast 10.66.66.255 bridge_fd 0 bridge_hello 0 bridge_stp off
Bring up the bridge (this will be done automatically at boot time).
# ifup xen-br0
Check the bridge (the bridge won't have any interfaces on it until we bring up some virtual machines later)
# ifconfig xen-br0
xen-br0 Link encap:Ethernet HWaddr 00:00:00:00:00:00
inet addr:10.66.66.66 Bcast:10.66.66.255 Mask:255.255.255.0
inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:378 (378.0 b)
# brctl show
bridge name bridge id STP enabled interfaces
xen-br0 8000.000000000000 no can't get port info: Function not implemented
s:0 (0.0 b) TX bytes:378 (378.0 b)
/etc/network/interfaces in domUs
Whether you assign IP addresses through the configuration, within the domU, or through DHCP, they need to be on the same subnet as the bridge (xen-br0, 10.66.66.0/24) and have the bridge (10.66.66.66) as the gateway.
As sample /etc/network/intefaces would look like this.
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet static
address 10.66.66.67
netmask 255.255.255.0
gateway 10.66.66.66
broadcast 10.66.66.255
Restart xend
You should now be ready to restart xend.
# xend start
And if you have already created domains...
# xendomains start
When the domains come up, they should be able to talk to each other but won't be able to access the internet until you configure Shorewall in the next step.
Shorewall
If we want our domUs to have access to the internet, we need to treat them like a regular LAN. I use Shorewall to set up the masquerading, but if you want to set this up manually, by all means do. (As a reminder, these directions are for Shorewall version 3.x.x.)
We start with the basic shorewall two-interface example (http://www.shorewall.net/two-interface.htm), and simply modify it so that all references to eth1 become xen-br0. If you do not have direct physical access to this machine, then you need to add a rule to allow SSH traffic to $FW from net BEFORE starting shorewall.
# gunzip /usr/share/doc/shorewall/examples/two-interfaces/*.gz # cd /etc/shorewall # cp /usr/share/doc/shorewall/examples/two-interfaces/* .
Edit the config files.
/etc/shorewall/interfaces
# ############################################################################### #ZONE INTERFACE BROADCAST OPTIONS net eth0 detect dhcp,tcpflags,norfc1918,routefilter,nosmurfs,logmartians loc xen-br0 detect tcpflags,detectnets,nosmurfs #LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE
/etc/shorewall/masq
############################################################################### #INTERFACE SUBNET ADDRESS PROTO PORT(S) IPSEC eth0 xen-br0 #LAST LINE -- ADD YOUR ENTRIES ABOVE THIS LINE -- DO NOT REMOVE
/etc/shorewall/policy
(We want our firewall to have full access to the internet.)
############################################################################### #SOURCE DEST POLICY LOG LEVEL LIMIT:BURST loc net ACCEPT $FW net ACCEPT net all DROP info loc loc ACCEPT # THE FOLLOWING POLICY MUST BE LAST all all REJECT info #LAST LINE -- ADD YOUR ENTRIES ABOVE THIS LINE -- DO NOT REMOVE
/etc/shorewall/routestopped
(We want to be able to administer the firewall during start/stop/reload, and for the domUs to still talk to each other)
############################################################################## #INTERFACE HOST(S) OPTIONS xen-br0 - routeback eth0 sys.admin.ip.address source #LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE
/etc/shorewall/rules
(We want ssh access to our box; you may lock this down further. The line to accept Ping from bad places " Ping/ACCEPT net $FW" is for testing only and should quite likely be removed once the box is stable)
##################################################################### SSH/ACCEPT net $FW Ping/ACCEPT loc $FW Ping/ACCEPT net $FW Ping/REJECT net $FW ACCEPT $FW loc icmp ACCEPT $FW net icmp # #LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE
/etc/shorewall/shorewall.conf
(Make sure the following are set. Configure the rest as desired)
STARTUP_ENABLED=Yes IP_FORWARDING=On ADMINISABSENTMINDED=Yes
/etc/defaults/shorewall
(We want to be able to start shorewall)
startup=1
After all that
# shorewall start
At this point, if you are connected to the box remotely, try to ssh into it again. If you can't, you have configured something incorrectly. DO NOT CLOSE YOUR ORIGINAL CONNECTION UNTIL YOU FIGURE OUT WHAT.
Multiple external IP addresses, but not all mapped to domU
For this scenario, you follow the steps above (link to Funky_Routing#Single_external_IP_addresses.2C_multiple_domUs_which_need_internet_access), but you need the following additional configuration to map the IP address as needed to the domUs. For the following configurations, we have two "external" IP addresses in addition to the one assigned to eth0, 10.88.88.67 and 10.88.88.68, and they are mapped to the internal IP addresses of 10.66.66.67 and 10.66.66.68. Both are web servers.
/etc/shorewall/nat
10.88.88.67 eth0 10.66.66.67 no no 10.88.88.68 eth0 10.66.66.68 no no
/etc/shorewall/rules
(This file is as above, with the addition of the two "DNAT" lines.)
##################################################################### SSH/ACCEPT net $FW Ping/ACCEPT loc $FW Ping/ACCEPT net $FW Ping/REJECT net $FW DNAT net loc:10.66.66.67 tcp 80 - 10.88.88.67 DNAT net loc:10.66.66.68 tcp 80 - 10.88.88.68 ACCEPT $FW loc icmp ACCEPT $FW net icmp # #LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE
That's about it. Start as above. (shorewall start)
Multiple external IP addresses which are all mapped to domU
In this configuration, the valid external IP for dom0 is assigned to the bridge, and the valid external IPs for the domUs are assigned directly. Firewalling would be difficult (as the bridge is the external interface because eth0 has been added to it) and should be done at the machine level.
There would be no option to have a "LAN" in this scenario (as in, machines that communicate with each other but are not seen to the outside world); all domains must have an IP address within the subnet.
/etc/network/interfaces for dom0
/etc/network/interfaces
# The loopback network interface auto lo iface lo inet loopback # The primary network interface auto eth0 iface eth0 inet static address 0.0.0.0 auto xen-br0 iface xen-br0 inet static pre-up /sbin/ip link set eth0 up pre-up /usr/sbin/brctl addbr xen-br0 pre-up /usr/sbin/brctl addif xen-br0 eth0 address 10.88.88.66 netmask 255.255.255.0 network 10.88.88.0 gateway 10.88.88.1 bridge_fd 0 bridge_hello 0 bridge_stp off
/etc/network/interfaces for domU
Example; repeat for all domUs.
/etc/network/interfaces
# The loopback network interface auto lo iface lo inet loopback # The primary network interface auto eth0 iface eth0 inet static address 10.88.88.67 netmask 255.255.255.0 network 10.88.88.0 gateway 10.88.88.66
Bring up the bridge
At this point I'd reboot to see if everything comes up correctly. If you don't want to reboot, these are the steps. However, if you don't have physical access to the machine, you will lose connectivity doing this!!!
# xendomains stop # xend stop # ifdown eth0 # ifup xen-br0 # xend start # xendomains start
Also, keep in mind that there is a learning phase for the bridge, so you may not have connectivity directly after this; it could take up to 30 seconds. This is what things should look like when you are done.
# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 10.88.88.0 0.0.0.0 255.255.255.0 U 0 0 0 xen-br0 0.0.0.0 10.88.88.1 0.0.0.0 UG 0 0 0 xen-br0
# brctl show
bridge name bridge id STP enabled interfaces
xen-br0 8000.feffffffffff no eth0
vif1.0
vif2.0
# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:A0:C9:A3:A6:30
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:70 errors:0 dropped:0 overruns:0 frame:0
TX packets:63 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:7229 (7.0 KiB) TX bytes:5654 (5.5 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:3085 errors:0 dropped:0 overruns:0 frame:0
TX packets:3085 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:191615 (187.1 KiB) TX bytes:191615 (187.1 KiB)
vif1.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:54 errors:0 dropped:0 overruns:0 frame:0
TX packets:121 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4197 (4.0 KiB) TX bytes:8378 (8.1 KiB)
vif2.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:14 errors:0 dropped:0 overruns:0 frame:0
TX packets:67 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:588 (588.0 b) TX bytes:3372 (3.2 KiB)
xen-br0 Link encap:Ethernet HWaddr 00:A0:C9:A3:A6:30
inet addr:10.88.88.66 Bcast:10.88.88.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:76 errors:0 dropped:0 overruns:0 frame:0
TX packets:99 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4526 (4.4 KiB) TX bytes:6185 (6.0 KiB)
Recommended Reading
- Howto for XEN with Debian Sarge on a remote server (http://www.xmlvalidation.com/xen_howto_sarge.0.html)
- Ethernet Bridge + netfilter Howto (http://www.tldp.org/HOWTO/Ethernet-Bridge-netfilter-HOWTO.html#toc1)
- Shorewall (http://www.shorewall.net/)
- Xen and/or Bridging Documentation
- Xen and the Art of Consolidation (http://shorewall.net/XenMyWay.html)
- Shorewall and Bridged Firewalls (http://www.shorewall.net/bridge.html) Includes sample start-up scripts for the bridge for different distros
- /etc/shorewall/hosts usage (http://www.shorewall.net/Documentation.htm#Hosts)
- the Continue Policy (http://www.shorewall.net/Documentation.htm#CONTINUE)
- General Shorewall documentation
- Shorewall Setup Guide (http://www.shorewall.net/shorewall_setup_guide.htm)
- Shorewall and Aliased Interfaces (http://www.shorewall.net/Shorewall_and_Aliased_Interfaces.html)
- Routing on One Interface (http://www.shorewall.net/Multiple_Zones.html)
- Shorewall Error Messages (http://italy.shorewall.net/ErrorMessages.html)* ebtables/iptables interaction on a Linux-based bridge (http://ebtables.sourceforge.net/br_fw_ia/br_fw_ia.html)
- Xen and/or Bridging Documentation
- XenNetworking Wiki (http://wiki.xensource.com/xenwiki/XenNetworking)
