Kerrighed on NFSROOT (contrib)

From Kerrighed

[edit] Kerrighed 2.2.x on NFSROOT

by University of Piraeus, Software Libre Society

[edit] General setup

Server

• Runs nfsd, pxelinux, atftpd.
• Runs a normal Debian Linux with a normal Linux kernel.
• Pxelinux is used to PXE boot the nodes.
• atftpd is used to serve the kerrighed kernel to the nodes.
• nfsd is used to serve the root filesystem to the nodes (which is located in /NFSROOT/kerrighed in the server).

Nodes

• Boot from PXE.
• Are running Debian linux with a kerrighed kernel which is served by the server's atftpd.

Through the whole tutorial, 192.168.1.1 will be the IP of our headnode.

On our server, we setup Debian Linux and start creating the chrooted NFSROOT directory, which we will serve to the nodes.

# mkdir /NFSROOT

We debootstrap the chroot system, mount a proc filesystem in it and finally chroot inside!

# debootstrap sid /NFSROOT/kerrighed <your_favorite_mirror_here>
# mount -o bind /proc /NFSROOT/kerrighed/proc
# chroot /NFSROOT/kerrighed

Now we will start setting up our nodes.

Let's start with the root password.

# passwd

We install the stuff that our nodes need, while chrooted.

# apt-get install dhcp3-common nfs-common nfsbooted gcc autotools libncurses ssh

We add the following to our `/etc/fstab` in the chrooted system.

### /etc/fstab ###

none    /proc       proc    defaults    0 0
none    /sys        sysfs   defaults    0 0
none    /var/run    tmpfs   defaults    0 0
192.168.1.1:/NFSROOT/kerrighed/var     /var    nfs rw,hard,nolock 0 0
192.168.1.1:/NFSROOT/kerrighed/tmp     /tmp    nfs rw,hard,nolock 0 0
192.168.1.1:/NFSROOT/kerrighed/root    /root   nfs rw,hard,nolock 0 0
#we need this as rw to setup ssh (refer to our ramdisk.sh script)
192.168.1.1:/NFSROOT/kerrighed/etc     /etc    nfs rw,hard,nolock 0 0

###

We execute the following command, because the original install document says so:

# ln -sf /etc/network/if-up.d/mountnfs /etc/rcS.d/S35mountnfs

(Be careful! The script should not collide with other services in the /etc/rcS.d.)

We download the latest kerrighed source tarball and the appropriate kernel source and we put them in `/usr/src`. After we extract them we will have to configure the kerrighed sources.

# cd kerrighed-x.x.x/
# ./configure --with-kernel=/usr/src/linux-2.6.x
# make patch

We then move to the kernel sources and we set them right (install libncurses5-dev, first) Make sure that the filesystem driver and our disk's controller are compiled in the kernel and not as modules, since we will not use initrd.

# cd ../linux-2.6.x
# make menuconfig

We then move back to the kerrighed sources, and execute make in them.

# cd ../kerrighed-x.x.x
# make kernel
# make

If everything goes well, kerrighed will built the modules, the kernel and the userland tools correctly. Now it's the time to install them

# make kernel-install
# make install

To make sure that everything is set correctly, check that the following actually exist:

/boot/vmlinuz-2.6.x-krg
/boot/System.map
/lib/modules/2.6.x-krg
/etc/init.d/kerrighed
/etc/default/kerrighed
/usr/local/share/man
/usr/local/bin/krgadm
/usr/local/bin/krgcapset
/usr/local/bin/migrate
/usr/local/lib/libkerrighed-*
/usr/local/include/kerrighed

Now, you are supposed to be done with the nodes/chroot configuration. Time to start configuring our server.

Exit the chrooted directory so that you are ready to start configuring our headnode.

Install everything that we need in order to serve the kernel to our nodes.

# apt-get install dhcp3-server nfs-kernel-server atftpd syslinux

Check out `/etc/default/atftpd` and make sure that it doesn't run from inetd and that the root directory is actually `/srv/tftp`

Move to `/srv/tftp` and make sure that the serving of the kernel goes as planned

# cd /srv/tftp
# cp /boot/vmlinuz-2.6.-krg . # This is the kerrighed kernel that we created. And should be put there to be served.
# cp /usr/lib/syslinux/pxelinux.0 .
# mkdir pxelinux.cfg

Get into `pxelinux.cfg` and create a boot config (check pxelinux.org for more info)

For example check our pxelinux.cfg/default:

### /srv/tftp/pxelinux.cfg/default ###

default patates
label patates
        kernel /vmlinuz-2.6.20-krg
        append console=tty1 root=/dev/nfs nfsroot=192.168.1.1:/NFSROOT/kerrighed ro ip=dhcp pci=nommconf

######

Now it's time for us to config our DHCP server.

### /etc/dhcp3/dhcpd.conf ###
# General options
option dhcp-max-message-size 2048;
#this feeds the hostname that we have declared in our mac-> ip pool to the client
#so that he can use it as his hostname
use-host-decl-names on;
deny unknown-clients;
deny bootp;

### PART 2
option domain-name "example.domain.com";
option domain-name-servers <your_domain_name_server>;
option ntp-servers <your_ntp_server>;

### PART 3
subnet 192.168.1.0 netmask 255.255.255.0 {
  option routers <your_router>;
  option broadcast-address <broadcast_address>;
  # Define the first and last IP address to be authorized
  range <range start> <range end>;
  # This set up the node name to « krgnodeXX » with XX the id of the node (ip-address based).
  send host-name = concat("krgnode", binary-to-ascii(10, 8, ".", substring(leased-address, 3, 1)));
}

### PART 4
group {
  filename "pxelinux.0"; # path is relative as to tftpd's root
  option root-path "192.168.1.1:/NFSROOT/kerrighed";
  next-server 192.168.1.1; #192.168.1.1 = our tftp/nfs/dhcp server's ip
  # You can also specify which boxes can be used with kerrighed
  # host ssi1 { fixed-address 192.168.1.10; hardware ethernet xx:xx:xx:xx:xx:xx; }
}

######

Now it's time for us to config our NFS server. Here is ours:

### /etc/exports ###

/NFSROOT/kerrighed 192.168.1.0/255.255.255.0(ro,async,no_root_squash,no_subtree_check)
/NFSROOT/kerrighed/tmp 192.168.1.0/255.255.255.0(rw,sync,no_root_squash,no_subtree_check)
/NFSROOT/kerrighed/var 192.168.1.0/255.255.255.0(rw,sync,no_root_squash,no_subtree_check)
/NFSROOT/kerrighed/root 192.168.1.0/255.255.255.0(rw,sync,no_root_squash,no_subtree_check)
/NFSROOT/kerrighed/etc 192.168.1.0/255.255.255.0(rw,sync,no_root_squash,no_subtree_check)

######

We restart our servers' services to get their updated configurations.

# /etc/init.d/atftpd restart
# /etc/init.d/dhcp3-server restart
# /etc/init.d/nfs-kernel-server restart
# killall portmap && portmap    #don't ask me why but we HAVE to do this in every headnode reboot.

Now we/you SHOULD have a fully functional cluster

[edit] Extra Stuff

If we need ssh to all our nodes we have to create automatically keys for the ssh everytime we setup a new node and save it to the server

For this, we are gonna use a script called ramdisk.sh and throw it into /etc/init.d of the nfsroot chrooted directory

# cp /path/to/ramdisk.sh /NFSROOT/kerrighed/etc/init.d
# chroot /NFSROOT/kerrighed
# update-rc.d ramdisk.sh defaults NN

where NN is a number smaller than the starting sequence number of the sshd.

We also have to change the sshd_config so that the HostKey attributes point the first part, by default to `/var/ramdisk/etc/ssh/ssh_host_*_key`

We exit chroot and create the directory in which the keys are gonna get saved

# mkdir /etc/ssh/keys

the script is gonna be saving the keys in a "key-hostname" filename manner