Supermicro x11ssh-F OpenBMC Porting (WIP)

Introduction to BMC and OpenBMC

BMC, short for Baseboard Management Controller, is a specialized microcontroller embedded in certain computers, typically servers. It has its own firmware, memory, and network connection (through its dedicated NIC or Network Controller Sideband Interface, NC-SI, sharing the host’s NIC). It serves as the hardware for the Intelligent Platform Management Interface (IPMI) and is connected to satellite controllers distributed across different system modules via an Intelligent Platform Management Bus/Bridge (IPMB), which extends from I²C. The BMC can independently monitor and manage the operational status of these system modules, such as power control, temperature sensing, fan speed adjustment, and even act as a remote console to display screen output and interact with the operating system using keyboard and mouse. It can also simulate external storage to facilitate remote OS installation.

The extensive management capabilities of BMC are made possible by its deep integration into the host system’s buses. BMC often replaces the SuperIO chip found in general x86 computer systems and is connected to the host via LPC and SMBus buses, providing corresponding functions to the host (such as serial ports and fan management). In order to enable remote console and external storage simulation, BMC is also connected to the host’s PCI(e) bus and behaves as a graphics card for video output, as well as connected to the host’s USB bus to act as an input device and external storage.

While most BMC original firmware uses the Linux kernel, it usually relies on outdated versions, and its user-space programs are generally proprietary software. As a result, proprietary BMC firmware is gradually being recognized as a security risk. In collaboration with the Linux Foundation, the OpenBMC project was launched. It is a GNU/Linux distribution designed for BMC and provides a freely available BMC software stack that can be compiled from source code. OpenBMC utilizes the Yocto Project as its build and distribution generation system, allowing it to generate firmware images that can be directly flashed onto the BMC’s ROM chip. Internally, OpenBMC employs D-Bus for inter-process communication and provides a web application for users to interact with the BMC software stack.

x11ssh-f and AST2400 Introduction

For hardware specifications, please refer to the link: https://doc.coreboot.org/mainboard/supermicro/x11-lga1151-series/x11ssh-f/x11ssh-f.html, where the firmware of its host system can be replaced with coreboot.

The x11ssh-f’s BMC (Baseboard Management Controller) uses ASPEED AST2400 and is equipped with 32MiB SPI NOR flash, 128MiB memory, and a dedicated network card. It replaces the host’s SuperIO and is connected to the host’s PCI-E and USB, also serving as the host’s graphics card.

ASPEED AST2400 is a system-on-chip specifically designed for BMC, which integrates an armv5 processor, MAC sections for two Gigabit Ethernet cards, a PCI-E device for graphics, USB 1 and 2 devices (HUB), a USB 2 host controller, an LPC controller, five UART serial ports, and 216 GPIOs. For more details, please refer to https://www.datasheetcafe.com/ast2400-datasheet-server-management-processor/.

The original firmware of the BMC can be programmed using an SPI flash programmer during power-off. For example, it can be directly read using a programmer made with an STM32 development board from https://github.com/dword1511/stm32-vserprog. Through analysis with binwalk, it is known that the BMC firmware uses an outdated Linux 2.6.28 kernel, even in the latest version obtained from https://www.supermicro.com/en/support/resources/downloadcenter/firmware/MBD-X11SSH-F/BMC.

Porting

First, use the method in https://github.com/mine260309/openbmc-intro/blob/master/Porting_Guide.md to create configuration files that describes the target for x11ssh.

OpenBMC uses a very new version of the Linux kernel, so unlike OEM firmware using the 2.6.28 kernel, hardware information needs to be described in the Flattened Device Tree (FDT) format instead of being compiled as part of the kernel in the form of a mach file. Although OpenBMC already supports many boards using ASPEED AST2400, the support for them in the Linux kernel has been accepted by the upstream repository. Therefore, the device tree source (dts) files for these boards already exist in the arch/arm/boot/dts/aspeed/ directory of the Linux source code. However, what we need to do now is to add support for x11ssh to OpenBMC, so the Linux source code does not include the dts file for x11ssh.

Fortunately, in a downstream branch of the u-bmc project, another free software implementation of BMC, we found the dts file for supporting x11ssh. Based on this, we can refer to the dts files for other ASPEED AST2400 boards that are already supported and modify them to adapt them to OpenBMC.

However, this file does not exist in the source tree of the Linux kernel. According to the usual process, the dts file for adapting x11ssh should be added to the kernel source tree, and the necessary Makefile should be modified to generate the corresponding FDT file during the compilation process. Then, the OpenBMC build system should use this modified kernel. However, for the OpenBMC build system, the Linux kernel is one of the third-party projects that OpenBMC needs to use. In this case, the build system generally does not allow direct use of modified third-party projects, but it often supports automatically patching third-party projects during the build process. Therefore, we need to represent the above modifications as patches to the Linux kernel. We add this dts file to a local git repository of the Linux kernel (if it doesn’t exist, clone one), modify the Makefile, commit the changes, and then export these modifications as a diff file (using git show or git format-patch) and trim it. Place the file in the meta-aspeed/recipes-kernel/linux/files/0001-add-aspeed-bmc-supermicro-x11ssh-dts.patch directory of the OpenBMC repository. Also, add a meta-aspeed/recipes-kernel/linux/linux-aspeed_git.bbappend file as a supplement to linux-aspeed_git.bb, indicating to the build system to apply this patch before compiling the kernel:

FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
SRC_URI += "file://0001-add-aspeed-bmc-supermicro-x11ssh-dts.patch"

To enable debugging, at least one serial port is required for the console to output logs and log in to the OpenBMC system being developed. Both OpenBMC and u-bmc default to using uart5 as the console, but on x11ssh, uart5 is not exposed in a convenient form for connection. Fortunately, ASPEED AST2400 provides a register for remapping the order of each serial port. This feature can be used to remap uart5 to an exposed serial port.

The code in https://github.com/osresearch/u-bmc/commit/8b7d1ae02a33bbfda7487b6a2162a3317a355f00 contains the code that writes to this register in u-bmc’s own bootloader to remap uart5 to COM2. We need to port this code to the u-boot used by OpenBMC and represent it as a patch, named 0002-u-boot-console-com1.patch, located in meta-aspeed/recipes-bsp/u-boot/files/. Additionally, we need to add u-boot-aspeed_2016.07.bbappend in meta-aspeed/recipes-bsp/u-boot/ to automatically apply the patch during the compilation process.

+FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
+SRC_URI += "file://0002-u-boot-console-com1.patch"

A part of GPIO pin definitions for power management can be found at https://github.com/osresearch/u-bmc/blob/kf/x11/platform/supermicro-x11ssh-f/pkg/gpio/platform.go. These need to be rewritten in the required format for OpenBMC and placed in meta-supermicro/meta-x11ssh/recipes-phosphor/skeleton/obmc-libobmc-intf/gpio_defs.json.

At this point, you can run “source setup x11ssh” in the root directory of the OpenBMC repository, and then run “bitbake obmc-phosphor-image” to build the OpenBMC image. If the source code of some third-party components is not readily available for download, you can use the ${https_proxy} and ${HTTPS_PROXY} variables to provide an HTTP proxy server when running the second command. It is recommended to set both environment variables, for example: “https_proxy=http://192.168.3.3:8080 HTTPS_PROXY=$https_proxy bitbake obmc-phosphor-image”. The image will be generated at build/x11ssh/tmp/deploy/images/x11ssh/. The file obmc-phosphor-image-x11ssh.static.mtd in this location can be flashed to the 32MiB SOIC-16 package SPI NOR flash chip used to store the OpenBMC firmware on the x11ssh-f board. For the connection method, please refer to https://trmm.net/SPI_flash/. Before starting debugging, it is essential to back up the original firmware.

After flashing, when the system is powered on, you can receive logs from COM2:

but the logs indicate that OpenBMC stops running shortly after startup. Upon investigation, it has been found to be a known issue mentioned in https://github.com/openbmc/openbmc/issues/3969, where two conflicting services are occupying the same D-bus interface. To resolve this, one of the services needs to be disabled. Therefore, add the necessary configuration to the target configuration file meta-supermicro/meta-x11ssh/conf/machine/x11ssh.conf:

VIRTUAL-RUNTIME_obmc-host-state-manager = "x86-power-control"
VIRTUAL-RUNTIME_obmc-chassis-state-manager = "x86-power-control"
VIRTUAL-RUNTIME_obmc-discover-system-state = "x86-power-control"

to resolve the conflict.

After recompiling the image and flashing it, you can log in via COM2 using the username ‘root’ and password ‘0penBmc’ (note that the first character of the password is the digit zero).

Current Progress and future TODO

At this stage, OpenBMC ported to x11ssh-f can be successfully booted, and interact through the serial port is possible. However, several important daemons, such as systemd-journald and systemd-udevd, are currently being killed due to segmentation faults.

[   40.519097] systemd[1]: systemd-journald.service: Main process exited, code=killed, status=11/SEGV
[   40.549488] systemd[1]: systemd-journald.service: Failed with result 'signal'.
...
[   44.773066] systemd[1]: systemd-udevd.service: Main process exited, code=killed, status=11/SEGV
[   44.790901] systemd[1]: systemd-udevd.service: Failed with result 'signal'.

Furthermore, the network interface card is not initialized due to the killing of systemd-udevd, resulting in the inability to access the web interface. Although the NIC is recognized by the kernel, it cannot read the correct MAC address from the flash and is assigned a random temporary MAC address.

Future work will primarily focus on two aspects: first, extracting complete GPIO pin definitions from the original firmware and the provided 2.6.28 version of the Supermicro kernel source code, especially from the mach files. Secondly, collaborating with the upstream OpenBMC community to address the issue of critical daemon crashes, ultimately achieving a functional BMC firmware implementation.

Update (March 21 2024): Thanks for the feedback from Zev Weiss. We have confirmed (see logs) that the segfaults issues has been resolved via https://lore.kernel.org/linux-arm-kernel/[email protected]/T/