Custom Xen Kernels
|Table of contents|
Option 1: From the Xen Sources
2005-08-06: This has been reviewed and updated for 2.0.7
This is how I customized a new, Xen 2.0.6 kernel, in 8 easy steps.
My base system is a Debian testing system. Things might need to be slightly different for other distributions.
1. Download and Unpack
Download xen-2.0.7-src.tgz from http://www.cl.cam.ac.uk/Research/SRG/netos/xen/downloads.html (web page) or xen-2.0.7-src.tgz (http://www.cl.cam.ac.uk/Research/SRG/netos/xen/downloads/xen-2.0.7-src.tgz) (source file)
# wget http://www.cl.cam.ac.uk/Research/SRG/netos/xen/downloads/xen-2.0.7-src.tgz # tar -zxvf xen-2.0.7-src.tgz
2. Optional thing
(I often keep multiple versions around)
# mv xen-2.0 xen-2.0.7
3. Start work
Move into the correct directory to begin the magic.
# cd xen-2.0.7
- GCC (preferably v3.2.x or v3.3.x; older versions are unsupported)
- GNU Make
- GNU Binutils
- Development install of libcurl (e.g., libcurl3-dev)
- Development install of zlib (e.g., zlib1g-dev)
- Development install of Python v2.2 or later (e.g., python-dev)
If you are missing any of these, use apt-get in order to install them.
On a minimal system, this:
apt-get install gcc make binutils libcurl3-dev zlib1g-dev python-dev
libc6-dev libidn11-dev libssl-dev python2.3-dev
About 50 MB after install, and that's just the beginning...
4. Build Xen Hypervisor, Tools, and Base Kernel
First of all, make sure you're using the right version of GCC:
# gcc --version
Xen will not compile properly with gcc4, so if that's the default version you'll have to change it. If the system you're running this on is a dedicated Xen system, the easiest way to do this is to remove the gcc symlink pointing at version 4 and creating a new one pointing at 3:
# if [ -h /usr/bin/gcc ]; then cd /usr/bin; rm -f gcc; ln -s gcc-3.3 gcc ; fi
Make Xen (as root):
# make world
kdelete will delete all guest kernel build trees. This isn't important for a completely fresh install, but is important if you are trying to modify or upgrade your kernel. Wiping out all previous changes is often thought of as "bad".
If you are trying to upgrade your kernel (for example, you've already installed Xen 2.0.6 and you want to add or remove some kernel modules) then you don't want to do a make world. Instead, jump to step 6.
For other make options, type
# make help
Wait for a while (anywhere from 10 minutes to hours, depending upon the speed and memory of your machine).
Congratulations. Assuming no errors occurred while compiling, you now have a functional Xen/Linux kernel along with the needed Xen tools. These are all stored in the "dist" subdirectory. The default kernel is quite usable. If you don't wish to customize it, please skip to step 7.
If you have an existing configuration from which you wish to work then you should copy it to the xen-2.0.7/linux-2.6.11-xen0 directory:
# cp /PATH/existing_config /root/xen-2.0.y/linux-2.6.11-xen0/.config
To configure, choose one of the following:
# make ARCH=xen xconfig # make ARCH=xen menuconfig
When you've finished making changes, save, then exit the configuration utility. Then:
# cd .. # make kernels
Now it's time to get a snack and a drink. Again, this could take anywhere from a few minutes to an hour based on machine speed and memory. If you're using a really slow machine (for example, a Pentium II 400MHz machine) then you might consider going out to a movie or having a meal with friends and family.
Before installing, I always make sure I have some way to roll back to a previous (bootable) version. For example, in my Dom0 I don't remove the initial kernel from whatever installation I use (Debian, for the most part). If I'm updating a previous installation (adding/removing modules, etc.) I also like to ensure that I have a duplicate of my xen-#.#.#/dist directory.
# cp -dpR dist dist.installed
From the xen-2.0.7 directory:
# make install
Next we have to adjust the GRUB menu file to include our new kernel.
# vi /boot/grub/menu.lst
Add some lines similar to the following:
title Debian + Xen 2.0.7/2.6.11 (Stable) (hdd2) root (hd1,1) kernel /boot/xen.gz dom0_mem=393216 module /boot/vmlinuz-2.6.11-xen0 root=/dev/hdd2 ro console=tty0
Now you should be ready to reboot. Make sure you have a rescue disk, just in case. Good luck.
Addendum: Should you want the documentation, and who doesn't, then you can build that by typing "make docs". You will need additional packages in order for this to work, in particular, "latex", "transfig" and "ps2pdf". (Thanks to Jan de Haan)
Option 2: From the Debian Sources with the Xen Patches
If you're familiar with custom-compiling kernels in Debian (not a pre-requisite, the instructions should give you all the steps), then the biggest questions are:
- Is the Xen patch for the version you are compiling available as a Debian package?
- Is the Debian-patched source available for the version you are compiling?
- What configuration options can you select and still get a bootable system?
If the answer to the first two questions is "no" you should probably just follow the "Xen" instructions above. If the source is available but the patch isn't, you are _probably_ safe in creating the patch yourself and building the package from one of the previous versions. I'll mention how to do that later. For now we'll go with the simple situation - patches and kernel source are available.
Keep in mind that this assumes you've already installed the rest of the Xen tools, and just need to compile a new kernel. If this is not the case, you'll want to set that up before actually trying to boot your new kernel.
Prepare your system
If you haven't built a kernel before, you may need to add a few packages. This should be most of it.
apt-get install kernel-package grep-dctrl dpkg-dev patch
Since I configure using "menuconfig" I also need:
apt-get install libncurses5-dev
Get the Xen Patch
Install the xen patch
If you are using the unofficial Xen packages available on this site, then:
apt-get install kernel-patch-VERSION-xen
If you are using the Xen packages available in sid/unstable (as of 2005-07-21 the version you will get is 2.0.6/2.6.11):
apt-get install kernel-patch-xen
OR Make the patch
Sometimes the Debian patches are for Xen may not be the most recent available. If that is the case, and you can't track down somebody who has created the kernel-patch-xen deb for this verion, you may need to create the Xen patches and build the .deb yourself.
Download the tarball of the Xen source for the version you are planning to custom compile a kernel for, http://www.cl.cam.ac.uk/Research/SRG/netos/xen/downloads.html, and unpack it
tar -zxvf xen-VERSION-src.tgz cd xen-VERSION make mkpatches
After that runs you should end up with a few files named "linux-VERSION-xen.patch", usually one for 2.4 and one for 2.6. If you want to make a Debian package of it, this will probably work (no guarantees on this)
mkdir tmp_build_dir/DEBIAN dpkg -X CURRENT_XEN_KERNEL_PATCH.deb tmp_build_dir dpkg -e CURRENT_XEN_KERNEL_PATCH.deb tmp_build_dir/DEBIAN cd tmp_build_dir rm usr/src/kernel-patches/diffs/xen/linux-OLD_VERSION-xen.patch.gz cp XEN_SOURCE_DIR/linux-VERSION-xen.patch.gz usr/src/kernel-patches/diffs/xen/linux-VERSION-xen.patch.gz
You probably want to look at the control file (tmp_build_dir/DEBIAN/control) and perhaps make changes. To finish
cd .. dpkg --build tmp_build_dir MY_XEN_PATCH.deb dpkg install MY_XEN_PATCH.deb
Get and unpack the source
The most recent version of the debian patched kernel source is normally in sid/unstable (and the xen packages are also currently in unstable), so make sure you have a line in /etc/apt/sources.list for that looks someting like this:
deb ftp://ftp.us.debian.org/debian/ sid main
If you just added the line for sid/unstable, you'll also need to:
Install the source:
apt-get install kernel-source-VERSION
Unpack the source:
cd /usr/src tar xjf kernel-source-VERSION.tar.bz2
Choose a base config
Copy a .config file into the source directory. The copying part is easy - the question is, which config? Customary wisdom seems to be "start with your current debian config and prepend the necessary XEN lines."
cp /path/to/config/my_config /usr/src/kernel-source-VERSION/.config
This has never worked for me, but you might get lucky, so here's what you need to add to your current configuration to xen-ify it for a dom0.
# CONFIG_XEN=y CONFIG_ARCH_XEN=y CONFIG_NO_IDLE_HZ=y # # XEN # CONFIG_XEN_PRIVILEGED_GUEST=y CONFIG_XEN_PHYSDEV_ACCESS=y CONFIG_XEN_BLKDEV_BACKEND=y CONFIG_XEN_NETDEV_BACKEND=y # CONFIG_XEN_BLKDEV_FRONTEND is not set # CONFIG_XEN_NETDEV_FRONTEND is not set CONFIG_XEN_WRITABLE_PAGETABLES=y CONFIG_XEN_SCRUB_PAGES=y CONFIG_X86=y # CONFIG_X86_64 is not set CONFIG_HAVE_ARCH_DEV_ALLOC_SKB=y
Alternatively, you could start from one of the Sample Kernel Configurations linked to on this page.
After all that preparation, the command to build the kernel is straightforward. (--revision and --append-to-version are optional, but useful to keep your kernels straight.
cd /usr/src/kernel-source-VERSION/ PATCH_THE_KERNEL=YES make-kpkg --config=menuconfig --arch=xen --revision=1 \ --append-to-version=myxen0 kernel_image
You'll be able to add and delete options before compiling. If the compilation succeeds, you will end up with a *.deb for your kernel in the directory above the source directory, which you can install with "dpkg install my-kernel.deb". By default this package will not modify the grub entry or make an initrd.
The grub entry should look something like this:
title Debian + Xen (Stable) root (hd0,0) kernel /boot/xen.gz dom0_mem=65536 module /boot/xen-linux-VERSION-myxen0 root=/dev/hdd1 ro console=tty0
If you have problems with the kernel you've created, and get stuck in a reboot loop, you may want to change the xen line so that you can stop the reboot and get a chance to look at the screen:
kernel /boot/xen.gz dom0_mem=65536 noreboot
For initrd, you can look at the man page for make-kpkg to see the options to have the initrd image created as part of the package intallation. If you don't go that route, after you install your kernel:
mkinitrd -o /boot/initrd.img-VERSION-myxen0 VERSION-myxen0
And put this additional line in your grub entry:
Clean source tree
If you are doing something other than just adding one more module to an existing configuration, then you'll want to clean your source tree between builds. This may not work all the time.
cd /usr/src/kernel-source-VERSION/ PATCH_THE_KERNEL=YES make-kpkg --arch=xen clean
If you encounter error messages when you compile the kernel for the second or the third time, try deleting /usr/src/kernel-source-VERSION and untar the kernel again. You might want to back up your .config before deleting the source directory, though.
Compiling the domU is pretty much like compiling the domO, but with a different .config. Some people make sure they compile everything they need into the domU kernel. In Xen 2.0, the modules live in the domU file system, but the kernel is on the dom0 file system, so if you have no modules, then it is simpler to change kernel versions for a domU.
cp /path/to/config/my_config /usr/src/kernel-source-VERSION/.config cd /usr/src/kernel-source-VERSION/ PATCH_THE_KERNEL=YES make-kpkg --config=menuconfig --arch=xen --revision=1 --append-to-version=-myxenu kernel_image
Sample Kernel Configurations
- Base configuration as shipped from Xen for 2.0.6/2.6.11
- default dom0 config (http://www.option-c.com/downloads/xen_kernel_configs/xen0_defconfig_2.6.11_2.0.6) - from the xen source, /PATH/xen-2.0/linux-2.6.11-xen-sparse/arch/xen/configs/xen0_defconfig
- default domU config (http://www.option-c.com/downloads/xen_kernel_configs/xenU_defconfig_2.6.11_2.0.6) - from the xen source, /PATH/xen-2.0/linux-2.6.11-xen-sparse/arch/xen/configs/xenU_defconfig
- If you use the unofficial packages and kernel from this site for 2.0.6/2.6.11, these are the configurations for dom0 and domU
- Configuration for a firewall with direct physical access (http://www.option-c.com/downloads/xen_kernel_configs/config-2.6.11-fw0) - this works for dom0 or domU
- a rather unstable but fully loaded dom0 (http://www.option-c.com/downloads/xen_kernel_configs/config-2.6.11-ksxen0) - it does boot
- Xen/XML/Sarge Howto (http://www.xmlvalidation.com/xen_howto_sarge.0.html)
- make-kpkg man page (http://www.annodex.net/cgi-bin/man/man2html?1+make-kpkg)
- The Debian GNU/Linux FAQ Chapter 6 - Basics of the Debian package management system (http://www.debian.org/doc/FAQ/ch-pkg_basics.en.html)
- source for non-free modules (http://www.acm.rpi.edu/~dilinger/kernel-source-nonfree-2.6.11/)
- Xen-users "xen and Debian kernel package" (http://lists.xensource.com/archives/html/xen-users/2005-06/msg00223.html)
- Create Debian Linux packages (The IBM Way) (http://www-106.ibm.com/developerworks/linux/library/l-debpkg.html?ca=dgr-lnxw82DebianLinux)