Table of Contents
PXE Image
Build Instructions for PXE Images
Build Setup
The current build package for making pxe images lives on:
console.sb5.orbit-lab.org:/root/pxe/omf/pxe/
In there is a make master make file with all the parameters to build the kernel / initramfs image pair. It will also build a pxelinux.bin from sources. This version of pxelinux.bin doesn't need to match with kernel / initramfs pair.
in the Makefiel the values of the following variables control what kernel directory the make process will consult to build a kernel:
KERNEL_VERSION = 3 KERNEL_PATCHLEVEL = 0 KERNEL_SUBLEVEL = 4
In the /root/pxe/omf/pxe/src directory should be a directory named linux-VERSIONNUMBERS. In this directory is the config file for the kernel being built.
Additionally ins the same source dir is a busbox-VERSION directory with a .config file that controls what features are built into busybox.
Building in NTP clock updates
- Even though the version of the kernel source tree is
specified in the make file, the build process does not
use the kernel config file that was downloaded from the
mirror. Instead it uses the file named
linux-omf-pxe-#.#.#.config
in the directory:/root/pxe/omf/pxe/config
- the same is true for the busy box, I think it uses the file:
busybox-winlab.config
- Finally the startup script rcS.actual will be copied to the init-ram-fs and run as part
of the pxe preOS sequence. We recently modified that file to have it poll NTP and
correct the system clock. We added the following lines:
/usr/sbin/ntpd -d -n -q -p consolec /sbin/hwclock --directisa -w
Bumping the Kernel Version
Made a few modification for the most current version based on kernel 3.11.1
- Copied a running ubuntu config from an ubuntu 13.10 install to serve as sane default. It does throw one question about vesa support which I said yes to, but mostly complies. Happily.
- After some testing we decided to remove R8169 (realtek) support and only enumerate the intel interfaces (this will cause some other boards to pxe/boot off eth0).
- Made specially modified pxelinux.cfg file that boots off of eth0 for the newest (Gen4 asus) nodes
Usage Notes
Makefile
PXE has a 'changelog' in the top directory. The following extracts the version number from the first line in the changelog - enjoy:
24 VER = $(shell grep -o -E '[0-9]+\.[0-9.-]+' -m 1 changelog)
The following line copies a directory from one place to the other, but without ".svn" directories and emacs temporary files (*~)
119 tar -C static --exclude=.svn --exclude='*~' -cf - . | (cd $(RDISK); tar -xf -)
The image target should now exist on its own (the incuded software is a sub target). I just added a few 'sudo' commands to allow it to compile from a user account.
The main addition is the apt target:
141 apt: dirs kernel image 142 @echo Building APT V$(VER) 143 rm -rf $(BUILD_APT_DIR) 144 install -d $(BUILD_APT_DIR) 145 cp -rp debian $(BUILD_APT_DIR) 146 rm -rf $(BUILD_APT_DIR)/debian/.svn 147 mv $(BUILD_APT_DIR)/debian/Makefile $(BUILD_APT_DIR) 148 cp changelog $(BUILD_APT_DIR)/debian 149 cp ../copyright $(BUILD_APT_DIR)/debian 150 if [ -r README.debian ] ; then \ 151 cp README.debian $(BUILD_APT_DIR)/debian; \ 152 fi 153 env O_VER=$(VER) O_KERNEL_V=$(KERNEL_VERSION) $(MAKE) -C $(BUILD_APT_DIR) deb 154 # $(MAKE) DESTDIR=$(BUILD_APT_DIR)/debian/$(APT_NAME) install 155 156 apt-install: apt 157 scp $(BUILD_APT_DIR)/../*.deb $(REPOSITORY):$(REPOSITORY_ROOT)/binary 158 ssh $(REPOSITORY) sudo $(REPOSITORY_ROOT)/rebuild.sh
The BUILD_APT_DIR variable was defined above, but it's in build. 143 and 144 create new apt directory. I added a new top level directory 'debian' to hold all the debian related files. This directory is copied into the apt build area (145, 146 - Should really be using something similar to 119). The 'debian' directory contains the Makefile which should really be a level up in the build area. Debian apt's also need a change log which we copy into the build area (148) as well. Same goes for the copyright file which we take from the nodehandler proper (149), and the README.debian, if it exists (150-152). 153 is a bit tricky. Basically we need to know the version of this apt, as well as the kernel version for the 'install' target in 'debian/Makefile'. However, we are calling the 'deb' target, which calls some dpkg util, which ultimately uses the 'debian/rules' file to call the 'install' target of 'debian/Makefile' - still following? In short, we put the two versions into environment variables "env O_VER=… make …' before calling the first make which the 'install' target can retrieve and use (remember a non-initialized Makefile variable takes it's value from an environment variable of the same name if it exists).
debian/Makefile
install: echo "INSTALL $(O_VER) $(O_KERNEL_V)" install -d $(TFTP_DIR) install -m666 ../initrd-orbit-pxe-$(O_VER).img $(TFTP_DIR) install -d $(CFG_DIR) sed -e s/@KERNEL_VER@/$(O_KERNEL_V)/ ../../config/default.orbit | sed -e s/@VER@/$(O_VER)/ > $(CFG_DIR)/default.orbit-$(O_VER) install -m755 ../../config/makeLink $(CFG_DIR)
The O_VER and O_KERNEL_V come form the environment variables defined in 153 above. This apt installs the image we built into /tftpboot, as well as 'default.orbit-XXX' and 'makeLink' in /tftpboot/pxelinux.cfg. The 'default.orbit-XXX' file is created from the template stored in the config directory.
DEFAULT boel LABEL boel KERNEL linux-@KERNEL_VER@ APPEND vga=extended root=/dev/ram0 rw load_ramdisk=1 prompt_ramdisk=1 ramdisk_size=32768 init=/vmlinuz initrd=initrd-orbit-pxe-@VER@.img console=ttyS0,9600 DISPLAY message.txt PROMPT 1 TIMEOUT 1
Default Pxe Issues
As part of the normal (non-pxe) boot process the nodes must pass though the pxe phase of the bios and "fail". The pxe client on all of the nodes should contact the tftp server and request a file named with the node ip adress in hex. It will then proceed to remove the trailing character until it runs out of characters. Once that process fails, the node will look for a file named default (no extention).
The current default used the localboot pxe directive (part of the standard) to instruct the nodes to fail over to disk. The file (now named default.localboot) has the following contents:
default harddisk label harddisk localboot 0
A recent test with some of the newest mother boards however failed. The pxeclient did not honor the locaboot 0 option (or rather they locked up when given this option). A work around documented here (stored as a pdf for future use) recommended trying the syslinux chainloader. This required a new default file with the contents:
default harddisk label harddisk KERNEL chain.c32 APPEND hd0 0
Service Requirements
To bootstrap the testbed nodes, in fact any server in the system, we use PXE to have the nodes load an image which contains a nodeagent and a frisbee client.
PXE requires the following support:
- A DHCP server pointing to an TFTP server
- A TFTP server proving the following:
- A network bootstrap program (NBP)
- A file containing instructions for the NBP
- A boot image
- A PXE grid service
Ideally the later referenced "orbit-pxe-server" apt package should include dependencies to all required packages and should setup everything. For instance, it should depend on gridservices and should then automatically instantiate the PXE grid service.
DHCP Server
To support PXE we need a DHCP server. This server does not necessarily need to be instantiated at the same machine as the TFTP server, but it's configuration needs to point to it.
Installing a DHCP server on Debian is as simple as:
% apt-get install dhcp
To support PXE, we need to add to /etc/dhcpd.conf something like:
option domain-name "example.com";
default-lease-time 600;
max-lease-time 7200;
allow booting;
# The next paragraph needs to be modified to fit your case
subnet 192.168.1.0 netmask 255.255.255.0 {
range 192.168.1.200 192.168.1.253;
option broadcast-address 192.168.1.255;
# the gateway address which can be different
# (access to the internet for instance)
option routers 192.168.1.1;
# indicate the dns you want to use
option domain-name-servers 192.168.1.3;
}
group {
# The address of the TFTP server that hosts the PXE image.
next-server 192.168.1.3;
host tftpclient {
# tftp client hardware address
hardware ethernet 00:10:DC:27:6C:15;
# The PXE boot loader that instructs the client to load a script corresponding to its IP address.
# The script defines the kernel and memory based image to download and boot. Think of it as a
# network aware grub/lilo type boot loader.
filename "/tftpboot/pxelinux.bin";
}
}
After you have edited the dhcpd configuration file, restart it with "/etc/init.d/dhcpd restart".
Installing a TFTP server
Debian makes that easy. We prefer atftpd since its the only tftp server that has verbose output. It also gives the ability to control the number of threads which handle transfers. With 400+ simultaneous clients, a good tftp server is required.
% apt-get install atftd
Installing NBP — pxelinux
'pxelinux.bin' is a file found in the syslinux package. Syslinux The nitty gritty of how it works, what to do to make a machine netboot, and all of the other specifics can be found at Syslinux.
Installing an PXE image
The PXE client gets the address of the TFTP server through DHCP. Taking it's IP address it attempts to download a specific file containing further instructions from the TFPT server. The name of the file is created from the PXE client's IP address. For Orbit, the instructions are identical for every node and are stored in "/tftpboot/pxelinux.cfg/default.orbit-version". This file instructs the node to fetch the image "/tftpboot/initrd-orbit-pxe-version.img" from the TFTP server and boot into it.
Both files will be installed through:
% apt-get install orbit-pxe-server
Installing the PXE Grid Service
Every service on Orbit which can be controlled by the experimenter requires a grid service component. The PXE grid service must be installed on the TFTP server.
% apt-get install gridservices % cd /etc/gridservices/enabled % ln -s ../available/pxe . % /etc/init.d/gridservices restart
