OpenBMC Remote OS Deployment: A Simplified Approach
OpenBMC Remote OS Deployment: A Simplified Approach
Many BMC implementations can accept a disk image and present it as a read-only USB mass storage device inserted into the host machine, allowing the host machine to boot from this “disk” for remote installation of the operating system or maintenance tasks.
How it work
BMC hardware that supports this functionality must integrate a USB Peripheral Device Controller (UDC) connected to the host machine’s USB Host Controller. The Linux kernel’s USB Gadget subsystem supports presenting local block devices as USB mass storage devices through the UDC, and it can specify any existing block device at runtime and start or stop it at will via the configfs interface. This way, any block device on the BMC can be presented as a USB mass storage device on the host machine.
The NBD subsystem is responsible for presenting external disk images as block devices on the BMC.
Some BMCs also support providing disk images through a web browser. Such BMCs need to implement a protocol using JavaScript that allows random read-only access to files provided by the user through the browser via WebSockets, and then translate it into the NBD protocol to listen as an NBD server locally on the BMC. The OpenBMC jsnbd project provides such an implementation, and a complete overview of the OpenBMC virtual media subsystem can be found in this document.
Implementation for Supermicro x11ssh-f
Enable Features
The BMC on x11ssh-f cannot run a complete OpenBMC due to insufficient memory, and many user-space components need to be trimmed. However, as long as an NBD server can be forwarded to the BMC via SSH, and commands can be used to operate the underlying parts of the virtual media subsystem, remote installation and maintenance can still be achieved.
First, support for UDC and NBD in the Linux kernel must be enabled. To do this, a set of Linux kernel configuration file fragments to enable the required Gadget drivers needs to be added to meta-aspeed/recipes-kernel/linux/linux-aspeed/, along with its description file placed in the meta-aspeed/recipes-kernel/linux/linux-aspeed/udc/ directory, and declared as an additional configuration in meta-aspeed/recipes-kernel/linux/linux-aspeed.inc under the “usbgadget” feature. Additionally, for some reason, the general configuration for aspeed-g4 does not include the NBD driver, while the general configurations for aspeed-g5 and aspeed-g6 do include it, so the NBD driver also needs to be added to meta-aspeed/recipes-kernel/linux/linux-aspeed/aspeed-g4/defconfig.
Next, enable the vhub device for the Aspeed G4 chip in the device tree of x11ssh-f, and add MACHINE_FEATURES += “usbgadget” in meta-supermicro/meta-x11ssh/conf/machine/x11ssh.conf to declare that x11ssh-f requires this feature.
Recompiling will yield an x11ssh-f OpenBMC firmware image with UDC and NBD support added.
Log in to the BMC with root privileges and run # nbd-client
Writing MAC Address
The BMC on x11ssh-f includes two Ethernet interfaces, eth0 and eth1, where eth0 shares physical layer components with the host’s eth0 via NC-SI, and eth1 is an independent network card. Their MAC addresses are set through u-boot environment variables. The read-only environment variables in the u-boot code do not contain MAC addresses, only the following default values:
bootcmd=bootm 20080000
bootdelay=2
baudrate=115200
verify=yes
spi_dma=yes
To pre-set the MAC addresses in the firmware image, you can append the two MAC addresses after the default values:
ethaddr=<mac of eth0>
eth1addr=<mac of eth1>
Use mkenvimage to convert it into a multi-copy little-endian U-Boot environment variable image, and then overwrite it to the firmware image at address 0x60000.
$ mkenvimage -rs 65536 -o uboot.env <variable list file>
$ dd if=uboot.env bs=64k count=1 seek=6 conv=notrunc of=obmc-phosphor-image-x11ssh.static.mtd
However, during the testing phase, you can also write the MAC addresses to the U-Boot environment variables at runtime, and they will take effect after rebooting the BMC.
# fw_setenv ethaddr <mac of eth0>
# fw_setenv eth1addr <mac of eth1>
Configuring Dropbear
The OpenBMC SSH server, by default, does not allow root login and only permits members of the priv-admin group to log in. However, associating NBD devices and configuring the USB gadget subsystem requires root privileges, so we need to remove this restriction.
This restriction is implemented by configuring DROPBEAR_EXTRA_ARGS="-G priv-admin -B" in /etc/default/dropbear. Therefore, log in to OpenBMC as root via the serial console and change /etc/default/dropbear to DROPBEAR_EXTRA_ARGS="", which will allow root login via SSH.
SeaBIOS Serial Console (Sercon)
SeaBIOS, by default, interacts with users through a keyboard and monitor, but it also supports simulating a VGA graphics card via the serial port, allowing interaction with users through the serial console. To enable this, when compiling the coreboot image with SeaBIOS as the payload, you can enable Add SeaBIOS sercon-port file to CBFS (CONFIG_SEABIOS_ADD_SERCON_PORT_FILE=y) and set the SeaBIOS sercon-port base address (CONFIG_SEABIOS_SERCON_PORT_ADDR) to the address of the first serial port, which is 0x3f8 by default.
After logging into the BMC, run obmc-console-client to connect to the host’s serial port. To return to the shell, we need to specify an escape character, such as ‘!’, which can be done with # obmc-console-client -e !. After that, you can exit obmc-console-client and return to the BMC shell by continuously entering “~!” and pressing Enter.
Installation Media
Most GNU/Linux distribution installation media do not support interaction via the serial console, so we need to create an installation medium that can interact through the serial port. We can refer to https://wiki.debian.org/DebianInstaller/Remote to create an installation medium that supports network console, installing GNU GRUB as the bootloader under BIOS, and refer to https://www.gnu.org/software/grub/manual/grub/html_node/serial.html to add the following to its GRUB configuration file:
serial --unit=0 --speed=115200
terminal_input console serial
terminal_output gfxterm serial
Activate the ability to interact via the serial console.
Running
On the local machine, use nbdkit to present the prepared installation image as an NBD server. Then, log in to the BMC as root via SSH while also forwarding the NBD server to the BMC. Run the command to associate the NBD server with the NBD device and present it to the host as a USB mass storage device. Next, run obmc-console-client to prepare for serial interaction with the host.
After the host starts, pay attention to the coreboot logs returned via the serial console. As soon as SeaBIOS begins to run, repeatedly press the ESC key to enter the SeaBIOS boot menu. From the boot options, select one similar to USB MSC Drive Linux File-Stor Gadget 060 and choose Automated install to start the network console (see the terminal playback below).
Then, in another terminal, use SSH to log in to the network console with the preset password from the installation media, allowing you to perform remote installation or maintenance tasks (see the terminal playback below).
Conclusion
The initial port of Supermicro x11ssh-f, shared publicly in March 2024, was improved using the previous codebase for our PoC (Proof of Concept) available at https://github.com/hardenedvault/openbmc/tree/x11ssh-f. Our goal is to manage multiple OpenBMC nodes with ease, utilizing tools such as chiba. The night is dark, cherish the self-soverignty while you still can. Be hold!