Grappling with IPMI, PXE and SOL

Installing one server isn’t difficult. It’s not really time consuming either. But it’s a whole lot different when you have to setup a huge number of servers. The ideal state of affairs, for me, is that once the physical install is done, which means rack mounting and plugging in of cables, I don’t want to be in the data centre anymore.

Today’s post is a bit of the server techie sort, so pardon me, this is probably not casual reading material. I thought I’ll post some of this info to share, and more importantly, to serve as a reminder to myself for future new cluster setups.

We keep learning something new. I haven’t really been doing much of server installation lately. Even when I did, in the past, it’s always a very modest number, like half dozen or so, in any one project.

This time, however, we have a few dozens to go at one time. In the past, I would tolerate booting each server up, go into BIOS, do a couple of required setup steps, before finishing off the operating system install either via a locally attached minimal net install image or PXE boot.

This time, too, our servers have become too modern. At the same time, I’ve found myself out-of-touch with UEFI. Yeah, to cut a long story short, lots of time was wasted because the server was trying to do UEFI while I was thinking in legacy BIOS terms. Plus, I didn’t like problems being magically solved, I have to know why and how they work.

It’s also that the servers were getting smart. By default they’ll use UEFI. But the moment you specifically instruct it to boot off a USB flash drive that does not do EFI, it falls back to legacy BIOS automatically. In fact, the simple matter of PXE boot also confused me, because the server was trying to do an UEFI PXE boot while my PXE setup was not prepared for it, but yet when you specifically instruct the server to perform a PXE boot, it falls back to legacy mode again. It was just so confusing.

I’ll get into details in a post another time. But here some some important things to know about first. It’s about configuring various boot and SOL (serial-over-LAN) settings via IPMI, so you do the work remotely over a network.

1. Get some sort of IPMI tools, such as OpenIPMI in Linux. OS X already has IPMI tools out-of-the-box. The general command format to run the IPMI too is:
   $ ipmitool -I lanplus -U <username> -H <ipaddress>
   where <username> is the IPMI admin account (on Supermicro, this is admin, with password admin), and <ipaddress> is the IP address of the IPMI card. In this post, subsequently, I’ll just list the remaining command portion when I prefix ipmitool>. That is, in fact, also the ipmitool’s prompt when you run it with the shell command.
2. Turn on serial-over-LAN with:
   ipmitool> sol set enabled 1
3. If the server is already gone past the step trying to PXE boot, you can reset it via IPMI:
   ipmitool> chassis power reset
4. Connect to the serial console like so:
   ipmitool> sol activate
   If console is already activate, deactivate with the next command and try again:
   ipmitool> sol deactivate
5. Monitor the console and follow through on your PXE/Kickstart installation.
6. In case your server continues to boot with PXE as higher priority, IPMI can set the boot device:
   ipmitool> chassis bootdev disk options=efiboot
   Oh yeah, that efiboot option was so important, because otherwise my server switches down to legacy BIOS mode if you try to override its boot default. Conversely, if your OS is already installed, and the hard disk has higher boot priority, you can use this to force PXE first:
   ipmitool> chassis bootdev pxe options=efiboot
7. If you need the change permanent (you probably do want the disk boot to be the permanent boot option), then:
   ipmitool> chassis bootdev disk options=efiboot,persistent
Ok, that didn’t work for me. The efiboot option doesn’t get set with persistent at the same time. It seems multiple options ought to be specified together, and that seems to work, except for efiboot. (If anyone knows how this can be fixed, please let me know, thanks.)

Most of these settings can be done in BIOS, of course. But the point is that I don’t want to sit in front of the actual server (or its KVM). Remote KVMs may be nice and convenient. However, apart from extra costs, remote KVMs involve lots of extra cabling, something I don’t fancy at all.