After my first run-in with kexec for the Jetfire, I ended up needing to use it as a bootloader for another device. I figured this was good time to learn how to set up Petitboot.

What is Petitboot

Petitboot is a set of programs designed to run on a minimal Linux environment (like an initramfs) that scans local disks for bootloader config files, parses them, and then presents a list of bootable operating systems to the user.

Now, on most "PC" adjacent systems, this is kind of pointless - GRUB, SYSLINUX, PXELINUX etc. can boot more things, and is often quicker and simpler to set up.

Petitboot has it's roots on the PS3 - on the early (NAND flash) models, the system's lv1 hypervisor would boot an otheros.bld file that had been copied onto the system's flash. The PS3 naturally isn't supported by those other bootloaders, so getting a "dumb" firmware to load a Linux image directly from flash is probably easier than dealing with porting an existing bootloader, or writing more complex code to handle partitions and filesystems.

As a result, there's no a lot of documentation on actually building or using petitboot since it's really only going to be embedded into a device's firmware by the manufacturer.

Why not

So, to get it out of the way, here's a dot point list of reasons you're probably better on sticking with the bootloader you already use on x86:

  • Linux can take a few seconds to initialise, and you could even "DOS" your system by plugging in badly behaving hardware (e.g. I have a wireless keyboard dongle that confuses Linux when the keyboard isn't turned on, and can hang things for 60 seconds).
  • Kexec can only load "linux" or "multiboot" files. You won't be able to run memtest or MS-DOS, not at least out of the box. You also can't "chainload" EFI files from it.
    • Similarly, while it probably could, it does not manage your EFI bootloader (e.g. via efibootmgr).
  • Petitboot (currently, at least) only has an ncurses UI. A graphical UI is probably possible, but I'm not aware of one that exists. You're stuck with serial console or text mode video+keyboard.
  • Petitboot (so far) doesn't seem to support saving settings. That appears to be somewhat hardware specific and the code isn't there for x86.
  • Petitboot relies on the Linux kernel to discover and mount filesystems. This means that you may run into situations where your Petitboot image may not have the drivers / modules, or new enough code, to read a filesystem. (This is noticeable with EXT4 - really old kernels don't like the feature flags on really new file systems).
  • Petitboot, on x86, still needs another bootloader to run.

When it makes sense

If you haven't read my Jetfire post, I'll summarise it - the bootloader in flash could only understand the Linux kernel format up to Linux 2.6.21. Any newer kernel would result in the system trapping in the BIOS debugger. I ended up using a 2.6.21 kernel to jump into a 6.x kernel built into it, and then look for bootable systems on disk.

Similarly, I've got an old Sophos XG-125 Rev 2 router that's "just" a normal x86-64 EFI PC. But, when using GRUB2, even the latest version, I can't boot any Linux kernel newer than 6.12 - they all just hang before earlyprintk can kick in. Weirdly, it /can/ boot a Linux kernel built as an EFI stub file, so I can build a Linux system that goes and finds the real system on disk.

So, from my experiments, petitboot as an "intermediate" shines when you find yourself with these kinds of situations:

  • Your firmware or existing bootloader /can/ boot Linux but not the Linux you want (Jetfire, Sophos XG-125R2).
  • Your firmware can't boot from your storage device (the HP Microserver G8 can't boot from an HP SAS HBA, and I couldn't get GRUB2 to work from a USB drive).
  • You use a "storage abstraction" system (e.g. LVM, md, BTRFS, ZFS et al), and don't want to rely on complex bootloader replication (e.g. writing the MBR to every disk, or scripts to update the ESP or /boot partition on every disk) or "disjointed" bootloader set ups (e.g. /boot on a USB drive)
  • You want to have some kind of recovery system (since Petitboot is just running on Linux, you can use whatever is available on the image) available before the OS is even loaded.

How it works

  1. Your Linux OS starts up
  2. udev discovers your disks, parititons, LVM LVs, filesystems and Network interfaces.
  3. Petitboot's "server" goes and runs a bunch of "discovery" agents on each filesystem / NIC.
  4. Each agent will mount the applicable filesystem (or bring up the NIC), scan for config files and report back to the "server" each bootable entry.
  5. Separately, udev starts up a client pb-console process on each configured TTY, to present the ncurses UI to the user.
  6. Each entry sent to the server is communicated to the console, and the user is shown a list of bootable entires.
  7. The user selects an entry (or the default one is selected automatically with a timeout), and then the server fires off kexec with the right arguments.
  8. The Petitboot server starts a shutdown of the system but instead of halt-ing the system, the running Linux kernel performs the jump.
  9. Your selected OS boots!

Additionally, if you selected a netboot option, the petitboot server will handle grabbing the kernel and initramfs over TFTP first.

As an embedded bootloader in the system firmware, petitboot is pretty hard to beat when it comes to hunting down a bootable Linux OS.

Configuring Buildroot

Thankfully, someone has gone through and added Petitboot to Buildroot, so we don't need to do too much to make it work nice.

Buildroot config

If you already have a Buildroot config / project for your device, I would recommend "forking" it. Since we're making a system that is /just/ the bootloader, we want to keep build times and image sizes small.

Our requirements are as follows:

  • One single file (i.e. a Linux kernel with embedded initramfs)
    • BR2_LINUX_KERNEL=y
    • BR2_TARGET_ROOTFS_INITRAMFS=y
    • # BR2_TARGET_ROOTFS_TAR is not set
  • Build for your system (in my case, x86-64 EFI)
    • BR2_x86_64=y
    • BR2_ROOTFS_OVERLAY="<< YOUR PATH >>/board/petitboot/rootfs_overlay/" - Place some files strategically on the filesystem
    • BR2_ROOTFS_POST_BUILD_SCRIPT="<< YOUR PATH >>/board/common/fix_setsid.sh << YOUR PATH >>/board/common/usr_blkid.sh" - Run a couple of fix ups before the image is completed.
    • BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
    • BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="<< YOUR PATH >>/board/<< BOARD NAME >>/linux_petitboot.config"
    • BR2_PACKAGE_ACPID=y - Optional - Handle ACPI signals from the system firmware (e.g. power button press)
  • (Recommended) Keep everything small
    • BR2_LINUX_KERNEL_LZMA=y
    • BR2_TARGET_ROOTFS_CPIO_LZMA=y
    • BR2_OPTIMIZE_S=y
  • With petitboot installed
    • BR2_ROOTFS_DEVICE_CREATION_DYNAMIC_EUDEV=y - This probably should be set automatically...
    • BR2_PACKAGE_PETITBOOT=y
    • BR2_PACKAGE_UTIL_LINUX_BINARIES=y - Required by udev for blkid
    • BR2_PACKAGE_KEXEC_ZLIB=y - Optional.
  • Display petitboot on the serial console and the primary display TTY:
    • BR2_PACKAGE_PETITBOOT_GETTY_PORT="tty0 ttyS0" - I would leave your "main" console out of this option to start with to ensure you can log into the system to debug the Buildroot config.
  • (Optional) Run getty on an alternative TTY:
    • BR2_TARGET_GENERIC_GETTY=y
    • BR2_TARGET_GENERIC_GETTY_PORT="tty2" - Set this to your "main" console until you're happy that Petitboot is working well enough, then disable the previous option or set this one to an alternative TTY.
  • (Optional) Housekeeping:
    • BR2_PER_PACKAGE_DIRECTORIES=y - Builds each package with only it's requirements. Allows building packages in parallel.
    • BR2_TARGET_GENERIC_HOSTNAME="x86-64-petitboot"
    • BR2_TARGET_GENERIC_ISSUE="x86-64 EFI Petitboot"

The above should be all you need in your buildroot config.

Linux Kernel config

Ensure you enable all the needed drivers / options (e.g. Block devices, SCSI, PCI / USB, AHCI, partitions, filesystems, MD) for the kernel to find and mount your boot data.

On top of that, we need:

  • kexec:
    • CONFIG_KEXEC=y
    • CONFIG_KEXEC_FILE=y - (If supported by your arch)
  • A quiet console:
    • CONFIG_CMDLINE="earlyprintk=efi console=tty2 panic=5" - I would recommend you keep console=ttyxxx set to your main console until you know your Buildroot config is working enough to log in. panic=5 will restart the system if the kernel panics.
  • File locking (for udev):
    • FILE_LOCKING=y - Should be enabled by default, and is hidden unless EXPERT=y
  • (Optionally) A TTY on the graphics output:
    • CONFIG_DRM=y
    • CONFIG_DRM_CLIENT_LOG=y
    • CONFIG_DRM_AST=y - To suit your hardware
    • CONFIG_FB=y
    • CONFIG_FB_EFI=y - To suit your hardware
    • CONFIG_FB_SIMPLE=y
    • CONFIG_USB_HIDDEV=y

File system config

So with your kernel and Buildroot config "done", we have a few more tiny tweaks to do:

setsid

For some reason util-linux's setsid fails when it's asked to take over our TTYs. This might be a missing kernel config option, but weirdly busybox's setsid works perfectly fine.

So, instead of fixing it properly, I just made this script that we reference in our BR2_ROOTFS_POST_BUILD_SCRIPT option:

#!/bin/bash

echo [fix_setsid] Ensuring setsid is a symlink to busybox...

rm -fv "$1"/usr/bin/setsid
ln -v -s /bin/busybox "$1"/usr/bin/setsid

blkid

For some reason udev expects blkid to live under /usr/sbin/, but Buildroot puts it under /sbin.

Using Buildroot's "Merged /usr" option seems to cause other failures during the build, so lets just use another hacky script:

#!/bin/bash

echo [usr_blkid] Creating symlink for /usr/sbin/blkid...

ln -v -s -f /sbin/blkid "$1"/usr/sbin/blkid

Serial tty setup

While refining my image, I had the kernel console set to my serial line. As part of that, I had the baud rate set.

When I was happy enough and decided to quieten down the boot by pointing the console elsewhere, the tty now defaulted to 9600 baud. The firmware defaults to 38400, so 9600 just causes garbled output.

So, let's make a little init script in our overlay (board/petitboot/rootfs_overlay/etc/init.d/S09serialbaud) to set our baud rate before udev starts our consoles up:

#!/bin/sh

case "$1" in
        start|restart)
                stty -F /dev/ttyS0 38400
                ;;
        stop)
                echo ''
                ;;
        *)
                echo "Usage: $0 {start|restart}"
                exit 1
esac

How to use Petitboot

Now that we've got a working Petitboot image, how do we actually use it? This, annoyingly, is not well documented.

Petitboot has a reasonably modular code - under the discover folder there are several key items:

  • grub2/
  • native/
  • kboot-parser.c
  • syslinux-parser.c
  • yaboot-parser.c

We can surmise that Petitboot should be able to understand the config files from these bootloaders. If you boot your Petitboot image on a system with GRUB 2 already set up, it should discover your config and present you with the same options you already get in GRUB. (As of writing, Petitboot doesn't understand BTRFS subvolumes, so it will only find config files in the "root" subvolume).

Similarly, if you insert any media with a SYSLINUX or GRUB config on it, you should see it pop up on the list too.

If you want to boot your "real" Buildroot project (so not this system, but your main OS), you can create either a GRUB config in /boot/grub/grub.cfg, or a SYSLINUX config in /syslinux.cfg.

For me, the SYSLINUX syntax was nice and simple:

$ cat /syslinux.cfg
default buildroot-2

label buildroot-2
kernel /boot/bzImage
append console=tty1 console=ttyS0,38400 iommu=pt intel_iommu=on panic=5 root=/dev/sda3

The normal caveats apply - if you're not using an initramfs (like the above example) you'll need to make sure your root=/dev/sdxn is updated when you update the image. Petitboot is smart enough to translate kernel /boot/bzImage to "load the kernel from /tmp/petitboot/.../sda3/boot/bzImage" but it doesn't seem to support templating or substituting in the command line.

Note the default buildroot-2 line. That tells Petitboot to boot the option buildroot-2 unless user input cancels it. From my observations, only the first default found is applied - if you have multiple files with defaults set, Petitboot may end up using a different default each boot due to differences in detection times.