Custom Xen Kernels

From OptionC

Table of contents
1 Option 1: From the Xen Sources
2 Option 2: From the Debian Sources with the Xen Patches

2.1 Prepare your system
2.2 Get the Xen Patch

2.2.1 Install the xen patch
2.2.2 OR Make the patch

2.3 Get and unpack the source
2.4 Choose a base config
2.5 Compile dom0
2.6 Clean source tree
2.7 Compile domU

3 Sample Kernel Configurations
4 Recommended Reading
[edit]

Option 1: From the Xen Sources

2005-08-06: This has been reviewed and updated for 2.0.7

This is how I customized a new, Xen 2.0.6 kernel, in 8 easy steps.

My base system is a Debian testing system. Things might need to be slightly different for other distributions.


1. Download and Unpack
Download xen-2.0.7-src.tgz from http://www.cl.cam.ac.uk/Research/SRG/netos/xen/downloads.html (web page) or xen-2.0.7-src.tgz (http://www.cl.cam.ac.uk/Research/SRG/netos/xen/downloads/xen-2.0.7-src.tgz) (source file) 

# wget http://www.cl.cam.ac.uk/Research/SRG/netos/xen/downloads/xen-2.0.7-src.tgz
# tar -zxvf xen-2.0.7-src.tgz

2. Optional thing
(I often keep multiple versions around) 

# mv xen-2.0 xen-2.0.7


3. Start work
Move into the correct directory to begin the magic. 

# cd xen-2.0.7
Note (from the README): Before building Xen, your machine must have
  • GCC (preferably v3.2.x or v3.3.x; older versions are unsupported)
  • GNU Make
  • GNU Binutils
  • Development install of libcurl (e.g., libcurl3-dev)
  • Development install of zlib (e.g., zlib1g-dev)
  • Development install of Python v2.2 or later (e.g., python-dev)

If you are missing any of these, use apt-get in order to install them.

On a minimal system, this: 

apt-get install gcc make binutils libcurl3-dev zlib1g-dev python-dev

Also added: 

libc6-dev libidn11-dev libssl-dev python2.3-dev

About 50 MB after install, and that's just the beginning...


4. Build Xen Hypervisor, Tools, and Base Kernel
First of all, make sure you're using the right version of GCC: 

# gcc --version

Xen will not compile properly with gcc4, so if that's the default version you'll have to change it. If the system you're running this on is a dedicated Xen system, the easiest way to do this is to remove the gcc symlink pointing at version 4 and creating a new one pointing at 3: 

# if [ -h /usr/bin/gcc ]; then cd /usr/bin; rm -f gcc; ln -s gcc-3.3 gcc ; fi

Make Xen (as root): 

# make world
Note: This does a combination of "make clean", "make kdelete", and "make dist".

kdelete will delete all guest kernel build trees. This isn't important for a completely fresh install, but is important if you are trying to modify or upgrade your kernel. Wiping out all previous changes is often thought of as "bad".

If you are trying to upgrade your kernel (for example, you've already installed Xen 2.0.6 and you want to add or remove some kernel modules) then you don't want to do a make world. Instead, jump to step 6.


For other make options, type 

# make help


5. Wait
Wait for a while (anywhere from 10 minutes to hours, depending upon the speed and memory of your machine).


6. Custom compiling the kernel

Congratulations. Assuming no errors occurred while compiling, you now have a functional Xen/Linux kernel along with the needed Xen tools. These are all stored in the "dist" subdirectory. The default kernel is quite usable. If you don't wish to customize it, please skip to step 7.

If you have an existing configuration from which you wish to work then you should copy it to the xen-2.0.7/linux-2.6.11-xen0 directory: 

# cp /PATH/existing_config /root/xen-2.0.y/linux-2.6.11-xen0/.config

To configure, choose one of the following: 

# make ARCH=xen xconfig
# make ARCH=xen menuconfig

When you've finished making changes, save, then exit the configuration utility. Then: 

# cd ..
# make kernels

Now it's time to get a snack and a drink. Again, this could take anywhere from a few minutes to an hour based on machine speed and memory. If you're using a really slow machine (for example, a Pentium II 400MHz machine) then you might consider going out to a movie or having a meal with friends and family.


7. Installation

Before installing, I always make sure I have some way to roll back to a previous (bootable) version. For example, in my Dom0 I don't remove the initial kernel from whatever installation I use (Debian, for the most part). If I'm updating a previous installation (adding/removing modules, etc.) I also like to ensure that I have a duplicate of my xen-#.#.#/dist directory. 

# cp -dpR dist dist.installed

From the xen-2.0.7 directory: 

# make install

Next we have to adjust the GRUB menu file to include our new kernel. 

# vi /boot/grub/menu.lst

Add some lines similar to the following: 

title           Debian + Xen 2.0.7/2.6.11 (Stable) (hdd2)
root            (hd1,1)
kernel          /boot/xen.gz dom0_mem=393216
module          /boot/vmlinuz-2.6.11-xen0 root=/dev/hdd2 ro console=tty0
Note: Make sure that the menu.lst file you edit is, in fact, the one that's connected with the GRUB root device. In the example above the root directory is on the second partition of /dev/hdd. If the GRUB root device is on the first partition of /dev/hdd (because you have a multiboot system and you just happen to be working with a different partition) then you must make sure that the above lines are in the menu.lst file for partition 1 of /dev/hdd. For more information, please refer to the GRUB manual.


8. Wrap-up
Now you should be ready to reboot. Make sure you have a rescue disk, just in case. Good luck.


Addendum: Should you want the documentation, and who doesn't, then you can build that by typing "make docs". You will need additional packages in order for this to work, in particular, "latex", "transfig" and "ps2pdf". (Thanks to Jan de Haan)

[edit]

Option 2: From the Debian Sources with the Xen Patches

If you're familiar with custom-compiling kernels in Debian (not a pre-requisite, the instructions should give you all the steps), then the biggest questions are:

  • Is the Xen patch for the version you are compiling available as a Debian package?
  • Is the Debian-patched source available for the version you are compiling?
  • What configuration options can you select and still get a bootable system?

If the answer to the first two questions is "no" you should probably just follow the "Xen" instructions above. If the source is available but the patch isn't, you are _probably_ safe in creating the patch yourself and building the package from one of the previous versions. I'll mention how to do that later. For now we'll go with the simple situation - patches and kernel source are available.

Keep in mind that this assumes you've already installed the rest of the Xen tools, and just need to compile a new kernel. If this is not the case, you'll want to set that up before actually trying to boot your new kernel.

[edit]

Prepare your system

If you haven't built a kernel before, you may need to add a few packages. This should be most of it. 

 apt-get install kernel-package grep-dctrl dpkg-dev patch

Since I configure using "menuconfig" I also need: 

apt-get install libncurses5-dev
[edit]

Get the Xen Patch

[edit]

Install the xen patch

If you are using the unofficial Xen packages available on this site, then: 

apt-get install kernel-patch-VERSION-xen

If you are using the Xen packages available in sid/unstable (as of 2005-07-21 the version you will get is 2.0.6/2.6.11): 

apt-get install kernel-patch-xen
[edit]

OR Make the patch

Sometimes the Debian patches are for Xen may not be the most recent available. If that is the case, and you can't track down somebody who has created the kernel-patch-xen deb for this verion, you may need to create the Xen patches and build the .deb yourself.

Download the tarball of the Xen source for the version you are planning to custom compile a kernel for, http://www.cl.cam.ac.uk/Research/SRG/netos/xen/downloads.html, and unpack it 

tar -zxvf xen-VERSION-src.tgz
cd xen-VERSION
make mkpatches

After that runs you should end up with a few files named "linux-VERSION-xen.patch", usually one for 2.4 and one for 2.6. If you want to make a Debian package of it, this will probably work (no guarantees on this) 

mkdir tmp_build_dir/DEBIAN
dpkg -X CURRENT_XEN_KERNEL_PATCH.deb tmp_build_dir
dpkg -e CURRENT_XEN_KERNEL_PATCH.deb tmp_build_dir/DEBIAN
cd tmp_build_dir
rm usr/src/kernel-patches/diffs/xen/linux-OLD_VERSION-xen.patch.gz
cp XEN_SOURCE_DIR/linux-VERSION-xen.patch.gz usr/src/kernel-patches/diffs/xen/linux-VERSION-xen.patch.gz

You probably want to look at the control file (tmp_build_dir/DEBIAN/control) and perhaps make changes. To finish 

cd .. 
dpkg --build tmp_build_dir MY_XEN_PATCH.deb
dpkg install MY_XEN_PATCH.deb
[edit]

Get and unpack the source

The most recent version of the debian patched kernel source is normally in sid/unstable (and the xen packages are also currently in unstable), so make sure you have a line in /etc/apt/sources.list for that looks someting like this: 

deb ftp://ftp.us.debian.org/debian/ sid main

If you just added the line for sid/unstable, you'll also need to: 

apt-get update

Install the source: 

apt-get install kernel-source-VERSION

Unpack the source: 

cd /usr/src
tar xjf kernel-source-VERSION.tar.bz2
[edit]

Choose a base config

Copy a .config file into the source directory. The copying part is easy - the question is, which config? Customary wisdom seems to be "start with your current debian config and prepend the necessary XEN lines." 

cp /path/to/config/my_config /usr/src/kernel-source-VERSION/.config

This has never worked for me, but you might get lucky, so here's what you need to add to your current configuration to xen-ify it for a dom0. 

#
CONFIG_XEN=y
CONFIG_ARCH_XEN=y
CONFIG_NO_IDLE_HZ=y
#
# XEN
# 
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PHYSDEV_ACCESS=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_XEN_NETDEV_BACKEND=y
# CONFIG_XEN_BLKDEV_FRONTEND is not set
# CONFIG_XEN_NETDEV_FRONTEND is not set
CONFIG_XEN_WRITABLE_PAGETABLES=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_X86=y
# CONFIG_X86_64 is not set 
CONFIG_HAVE_ARCH_DEV_ALLOC_SKB=y


Alternatively, you could start from one of the Sample Kernel Configurations linked to on this page.

[edit]

Compile dom0

After all that preparation, the command to build the kernel is straightforward. (--revision and --append-to-version are optional, but useful to keep your kernels straight. 

cd /usr/src/kernel-source-VERSION/
PATCH_THE_KERNEL=YES make-kpkg --config=menuconfig --arch=xen --revision=1 \
   --append-to-version=myxen0 kernel_image

You'll be able to add and delete options before compiling. If the compilation succeeds, you will end up with a *.deb for your kernel in the directory above the source directory, which you can install with "dpkg install my-kernel.deb". By default this package will not modify the grub entry or make an initrd.

The grub entry should look something like this: 

title           Debian + Xen (Stable) 
root            (hd0,0)
kernel          /boot/xen.gz dom0_mem=65536
module          /boot/xen-linux-VERSION-myxen0 root=/dev/hdd1 ro console=tty0

If you have problems with the kernel you've created, and get stuck in a reboot loop, you may want to change the xen line so that you can stop the reboot and get a chance to look at the screen: 

kernel          /boot/xen.gz dom0_mem=65536 noreboot

For initrd, you can look at the man page for make-kpkg to see the options to have the initrd image created as part of the package intallation. If you don't go that route, after you install your kernel: 

mkinitrd -o /boot/initrd.img-VERSION-myxen0 VERSION-myxen0

And put this additional line in your grub entry: 

module          /boot/initrd.img-VERSION-myxen0
[edit]

Clean source tree

If you are doing something other than just adding one more module to an existing configuration, then you'll want to clean your source tree between builds. This may not work all the time. 

cd /usr/src/kernel-source-VERSION/
PATCH_THE_KERNEL=YES make-kpkg --arch=xen clean

If you encounter error messages when you compile the kernel for the second or the third time, try deleting /usr/src/kernel-source-VERSION and untar the kernel again. You might want to back up your .config before deleting the source directory, though.

[edit]

Compile domU

Compiling the domU is pretty much like compiling the domO, but with a different .config. Some people make sure they compile everything they need into the domU kernel. In Xen 2.0, the modules live in the domU file system, but the kernel is on the dom0 file system, so if you have no modules, then it is simpler to change kernel versions for a domU. 

cp /path/to/config/my_config /usr/src/kernel-source-VERSION/.config
cd /usr/src/kernel-source-VERSION/
PATCH_THE_KERNEL=YES make-kpkg --config=menuconfig --arch=xen --revision=1 --append-to-version=-myxenu kernel_image
[edit]

Sample Kernel Configurations

  • Base configuration as shipped from Xen for 2.0.6/2.6.11
    • default dom0 config (http://www.option-c.com/downloads/xen_kernel_configs/xen0_defconfig_2.6.11_2.0.6) - from the xen source, /PATH/xen-2.0/linux-2.6.11-xen-sparse/arch/xen/configs/xen0_defconfig
    • default domU config (http://www.option-c.com/downloads/xen_kernel_configs/xenU_defconfig_2.6.11_2.0.6) - from the xen source, /PATH/xen-2.0/linux-2.6.11-xen-sparse/arch/xen/configs/xenU_defconfig
  • If you use the unofficial packages and kernel from this site for 2.0.6/2.6.11, these are the configurations for dom0 and domU
    • dom0/ocxen0 (http://www.option-c.com/downloads/xen_kernel_configs/config-2.6.11-ocxen0)
    • domU/ocxenu (http://www.option-c.com/downloads/xen_kernel_configs/config-2.6.11-ocxenu)
  • Configuration for a firewall with direct physical access (http://www.option-c.com/downloads/xen_kernel_configs/config-2.6.11-fw0) - this works for dom0 or domU
  • a rather unstable but fully loaded dom0 (http://www.option-c.com/downloads/xen_kernel_configs/config-2.6.11-ksxen0) - it does boot
[edit]

Recommended Reading

Retrieved from "http://www.option-c.com/xwiki/Custom_Xen_Kernels"