Problems netbooting non-EFI Pi 4 into HookOS

[mmeier86]

Hello Tinkerbell team,

I'm currently trying to get HookOS netbooted on a Raspberry Pi 4 without EFI.

Setup

All of my attempts are working with a Dnsmasq setup, with Tinkerbell's DHCP functionality completely switched off. Dnsmasq is supplying the Pi boot files via TFTP. For all of the Raspberry Pi firmware files, I've downloaded the Armbian image from here, specifically the "Minimal/IOT images with Armbian Linux v6.12" Ubuntu Noble image. I then copied the entire content into the TFTP root directory.

The config.txt I used did not change much, besides sometimes changing the kernel/initramfs filenames:

# Automatically load overlays for detected cameras
camera_auto_detect=1

# Automatically load overlays for detected DSI displays
display_auto_detect=1

# Automatically load initramfs files, if found
# auto_initramfs=1

# Don't have the firmware create an initial video= setting in cmdline.txt.
# Use the kernel's default instead.
disable_fw_kms_setup=1

# Disable compensation for displays with overscan
disable_overscan=1

# Run as fast as firmware / board allows
arm_boost=1

[pi4]
max_framebuffer=2

[cm4]
# Enable host mode on the 2711 built-in XHCI USB controller.
# This line should be removed if the legacy DWC2 controller is required
# (e.g. for USB device mode) or if USB support is not required.
otg_mode=1

[cm5]
dtoverlay=dwc2,dr_mode=host

[all]
kernel=vmlinuz-armbian-bcm2711-legacy
initramfs initramfs-armbian-bcm2711-legacy followkernel
arm_64bit=1
cmdline=cmdline.txt
dtoverlay=dwc2
dtoverlay=vc4-kms-v3d
enable_uart=1

The cmdline.txt did change a bit for each test, but they all had the same result. I copied it from a successful EFI/iPXE boot on the same Pi to get all the correct Tinkerbell parameters. One example:

tink_worker_image=ghcr.io/tinkerbell/tink-agent:v0.18.3-b817f7f2 syslog_host=192.0.2.111 grpc_authority=192.0.2.111:42113 tinkerbell_tls=false tinkerbell_insecure_tls=false worker_id=e4:5f:01:bc:f4:ce hw_addr=e4:5f:01:bc:f4:ce modules=loop,squashfs,sd-mod,usb-storage console=serial0,115200

Attempt with vanilla HookOS 0.10

My first boot attempt was with the initramfs and kernel from the HookOS 0.10.0 release, specifically the hook_armbian-bcm2711-current.tar.gz file. This attempt did successfully boot the kernel and the initramfs, as evidenced by the fact that containerd got started. There were a couple of errors in the logs for that attempt:

Failed to read service spec error "open /containers/services/getty/config.json: No such file or directory"

That was the last message. Past that, no further errors appeared and I wasn't getting a console (understandable, because getty failed to start).
Looking a bit further up, I also saw this message:

failed to run hwclock: exit status 1

That appears right after the kernel is done booting and right before the LinuxKit splash appears with "Welcome to LinuxKit". This error was confirmed by the next log lines from the containerd startup, which all had 1970-01-01 timestamps. This error makes sense of course, because the Pi 4 doesn't have a HW clock.

Scrolling through the kernel logs, the only problematic message I was able to see was this one:

rootfs image is not initramfs (read error): looks like an initrd

I'm not sure whether that one is really relevant though, because the initramfs is definitely getting mounted. Just something seems to be wrong.

Going back to the initial message, I thought something was wrong with the initramfs and I checked the following:

• The /containers/services/getty/config.json file is present in the initramfs
• The initramfs itself seems to be fine, unpacks without issue running gunzip -c initramfs | cpio -i
• Repacked the initramfs, this then lead to a kernel panic - I don't know what I might have done wrong, I ran find . | cpio -H newc -o | gzip -9 > /somepath/initramfs

I tried a lot of other things, especially changing the cmdline.txt, but with no effect at all. Amongst other things, I tried an Ubuntu 24.04 kernel I'm using for my production Pi 4, it showed exactly the same behaviour.

Attempt with the most current armbian-bcm2711-current

Next I wanted to check whether a newer kernel would work. I cloned the Hook repo at commit 0182cbc, the current HEAD of main.

I then ran then following commands:

./build.sh kernel armbian-bcm2711-current
./build.sh build armbian-bcm2711-current

When I executed that command, it provided me with 6.12.35-S8292-Dbdda-P0000-Ce6dbH2313-HK01ba-Vc222-Ba566-R448a.

I copied the kernel and initramfs as well as the overlays/dtbs produced into the TFTP root and booted. Just to be greeted by a similar issue. The only difference was that I wasn't getting the previous getty startup error. Instead, there was just this message as the only error I could find:

WARN[1970-01-01T00:00:17.287409971Z] failed to load plugin                         error="failed to create CRI service: failed to create cni conf monitor for default: failed to create the parent of the cni conf dir=/etc/cni: mkdir /etc/cn
i: read-only file system" id=io.containerd.grpc.v1.cri type=io.containerd.grpc.v1

The hwclock issue also persisted. At the end of the log output, I got this message:

INFO[1970-01-01T00:00:17.289087879Z] containerd successfully booted in 1.073162s

And then nothing more. As before, I did not get a console prompt.

Attempt with a 5.15 Armbian kernel

Next, I tried an older 5.15 Armbian kernel, just on the odd chance that it had something to do with the newer kernel. I introduced the following change in to the bash/inventory.sh file:

      #
      ## Armbian bcm2711 (Broadcom) legacy, from RaspberryPi Foundation with many CNCF-landscape fixes and patches; for the RaspberryPi 3b+/4b
      define_id "armbian-bcm2711-legacy" METHOD='armbian' ARCH='aarch64' TAG='armbian-sbc' ARMBIAN_KERNEL_ARTIFACT='kernel-bcm2711-legacy'
      add_bootable_id "rpi" HANDLER='rpi_firmware' TAG='armbian-sbc'

Then I rebuild HookOS:

export ARMBIAN_KERNEL_VERSION="5.15.92-S14b3-Dfae2-P3c78-C7e7bHfe66-HK01ba-Vc222-B6a41-R448a"
./build.sh kernel armbian-bcm2711-legacy
./build.sh build armbian-bcm2711-legacy

That kernel version was the newest non-6.0 kernel I could find.

Booting this kernel/initramfs showed exactly the same issue as above. Containerd shows a happy "containerd booted successfully", and the nothing at all further happens.

Final

During all of these boots, the Pi never acquired an IP. As far as I could see that's because DHCPC is launched in a container as well. I also saw absolutely nothing in the Tinkerbell logs, which makes sense considering that the Pi never got an IP.

If there's anything more I can provide, please ping me.

[mmeier86]

A short update: It looks like some sort of truncation problem. When looking at the unpacked image for my armbian legacy kernel tests, I get about 600 MB. Looking at an unpacked initramfs on the Pi, the entirety of / is only 400 MB in size, and there's a weird 90 MB initrd.image file in / where I don't have any idea where it might be coming from.

I will continue investigation and see what happens when I slim down the HookOS image a bit.

[mmeier86]

Alright, it definitely looks like a Pi firmware restriction, possibly related to TFTP. See the end of this bug: raspberrypi/rpi-eeprom#375

Ignore the beginning, there they discuss something different.

When looking at the serial console of the 2nd stage of the Pi boot, where the initramfs is loaded, I'm seeing this line:

MESS:00:00:55.768976:0: initramfs loaded to 0x29440000 (size 0x5bbfa44)

The given size is about 96 MB. But the initramfs used here is about 122 MB. At the same time, the pcap of the TFTP transfer looks like all of the file is getting transferred, no problem. But I might also be reading that wrong?

Anyway, I will now look into doing a setup with two stages. Initial initramfs is just a small image with busybox. Than then looks at the cmdline for something like a "hookosinitrd=http://10.0.0.1:80/hooks-initramfs-real" option and downloads and mounts that file. Then it does something like a pivot_root to that new image and runs its /init, which will be the init from the real HookOS image. A bit complicated, but it avoids the size limit.