Remote Rebuild, CentOS-style
Problem: you've got a remote server that's significantly hosed, either through a screwup somewhere or a power outage that did nasty things to your root filesystem or something. You have no available remote hands, and/or no boot media anyway.
Preconditions: You have another server you can access on the same network segment, and remote access to the broken server, either through a DRAC or iLO type card, or through some kind of serial console server (like a Cyclades/Avocent box).
Solution: in extremis, you can do a remote rebuild. Here's the simplest recipe I've come up with. I'm rebuilding using centos5-x86_64 version 5.5; adjust as necessary.
Note: dnsmasq, mrepo and syslinux are not core CentOS packages,
so you need to enable the rpmforge
repository to follow this recipe. This just involves:
wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm rpm -Uvh rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm
1. On your working box (which you're now going to press into service as a
build server), install and configure dnsmasq
to provide dhcp and tftp services:
# Install dnsmasq yum install dnsmasq # Add the following lines to the bottom of your /etc/dnsmasq.conf file # Note that we don't use the following ip address, but the directive # itself is required for dnsmasq to turn dhcp functionality on dhcp-range=ignore,192.168.1.99,192.168.1.99 # Here use the broken server's mac addr, hostname, and ip address dhcp-host=00:d0:68:09:19:80,broken.example.com,192.168.1.5,net:centos5x # Point the centos5x tag at the tftpboot environment you're going to setup dhcp-boot=net:centos5x,/centos5x-x86_64/pxelinux.0 # And enable tftp enable-tftp tftp-root = /tftpboot #log-dhcp # Then start up dnsmasq service dnsmasq start
2. Install and configure mrepo to provide your CentOS build environment:
# Install mrepo and syslinux yum install mrepo syslinux # Setup a minimal /etc/mrepo.conf e.g. cat > /etc/mrepo.conf [main] srcdir = /var/mrepo wwwdir = /var/www/mrepo confdir = /etc/mrepo.conf.d arch = x86_64 mailto = root@example.com smtp-server = localhost pxelinux = /usr/lib/syslinux/pxelinux.0 tftpdir = /tftpboot [centos5] release = 5 arch = x86_64 metadata = repomd repoview name = Centos-$release $arch #iso = CentOS-$release.5-$arch-bin-DVD-?of2.iso #iso = CentOS-$release.5-$arch-bin-?of8.iso ^D # (uncomment one of the iso lines above, either the DVD or the CD one) # Download the set of DVD or CD ISOs for the CentOS version you want # There are fewer DVD ISOs, but you need to use bittorrent to download mkdir -p /var/mrepo/iso cd /var/mrepo/iso elinks http://isoredirect.centos.org/centos/5.5/isos/x86_64/ # Once your ISOs are available in /var/mrepo/iso, and the 'iso' line # in /etc/mrepo.conf updated appropriately, run mrepo itself mrepo -gvv
3. Finally, finish setting up your tftp environment. mrepo should have copied
appropriate pxelinux.0, initrd.img, and vmlinuz files into your
/tftpboot/centos5-x86_64 directory, so all you need to supply is an
appropriate grub boot config:
cd /tftpboot/centos5-x86_64 ls mkdir -p pxelinux.cfg # Setup a default grub config (adjust the serial/console and repo params as needed) cat > pxelinux.cfg/default default linux serial 0,9600n8 label linux root (nd) kernel vmlinuz append initrd=initrd.img console=ttyS0,9600 repo=http://192.168.1.1/mrepo/centos5-x86_64 ^D
Now get your server to do a PXE boot (via a boot option or the bios or whatever), and hopefully your broken server will find your dhcp/tftp environment and boot up in install mode, and away you go.
If you have problems with the boot, try checking your /var/log/messages file on the
boot server for hints.