NBD: A Powerful Tool for data center

NBD, the abbreviation for Network Block Device (protocol), is a protocol that was initially implemented in the Linux kernel version 2.1.55 in 1997, used to map a file or block device on a remote host to a local block device.

In Unix-like operating systems, in terms of supported operations, block devices are similar to regular files in many aspects, with the main difference being that block devices have a fixed size and do not support “appending,” meaning that data cannot be added to the end of a “file” to increase its size. Therefore, the key to the NBD protocol lies in serializing random access operations on block devices and transmitting them in a streaming manner.

In its most basic usage, the NBD server presents a file or block device on the computer, and once the NBD client connects to the server, it associates the socket for NBD protocol interaction with the local NBD block device (usually located at “/dev/nbdX”). Subsequent accesses to the NBD device are represented as NBD requests sent to the remote server for processing. Note: an NBD server can accept multiple incoming connections simultaneously, but if write access is permitted, it is best for the NBD server to be accessed exclusively by the same client (allowing multiple connections from the same client), otherwise data consistency cannot be guaranteed.

nbdfuse(1)

NBD and loop block devices have similarities, with the latter redirecting random access operations on block devices to local files. However, with the presence of a user-space file system (FUSE), it is generally preferred to represent various remote resources as local files first, and then either map them to block devices using loop or handle them directly in user space.

nbdfuse(1) meets this requirement. It can mount an NBD server onto a local directory, appearing as a file within it (default name “nbd”). Through this method, we effectively map a file from a remote computer to local storage for random access (except for append operations), without needing to download it locally.

nbdkit(1)

Since the essence of the NBD protocol is to serialize random access operations on files, in theory, any protocol supporting random file access can be translated into the NBD protocol. If the original protocol only supports reading and not writing, it can be translated to a subset of read operations of NBD protocol.

nbdkit(1) is a feature-rich NBD server implementation provided by the libguestfs project, capable of such translation through its extensible system.

For instance, to support resumable downloads, HTTP optionally allows downloading arbitrary segments of a file, effectively enabling the HTTP protocol for read-only random access to files. nbdkit(1) can utilize its curl plugin to translate an HTTP URL supporting resumable downloads into an NBD server allowing only read access, which, when combined with nbdfuse(1), enables the direct locally mounting of files provided by the HTTP URL for random access without the need to download the entire file:

$ mkdir fuse.mnt
$ nbdfuse -r fuse.mnt --command nbdkit -rs curl --filter=cache cache-on-read=true cache-min-block-size=1M --filter=retry https://example.com/path/archive.zip &
$ unzip -l fuse.mnt/nbd 
$ unzip -j fuse.mnt/nbd path/to/file
$ fusermount -u fuse.mnt

The –command option of nbdfuse(1) can run an NBD server as a subprocess, but it requires the NBD server to provide services via standard input and output. The -s option of nbdkit(1) supports this working mode.

For read-only access, you can enable read caching through the cache filter, so that segments that have already been downloaded do not need to be downloaded again. You can also adjust the cache granularity (corresponding to the size of each segment downloaded during an HTTP request) to improve the efficiency of HTTP. For example, based on the settings mentioned above, regardless of how many bytes are read at once from fuse.mnt/nbd, nbdkit(1) always aligns data downloads from the HTTP server to 1 MB boundaries and caches them. Downloading larger segments at once can more effectively utilize bandwidth. Additionally, enabling a retry filter can enhance the robustness of access by automatically retrying when a segment fails to download.

This working mode is particularly effective when extracting a few files from a zip archive or filesystem image provided by an HTTP server.

(Extracting files from a filesystem image can be implemented in user space using guestfish(1) without the need for kernel space implementation via loop block devices, which is omitted here.)

nbdkit(1) extensions are divided into two categories: plugins and filters. Plugins are used to adapt data sources (such as files, HTTP URLs, another NBD server, etc.) to the NBD server, while filters are used to transform data.

Some specific plugins and filters include:

• nbdkit-curl-plugin(1): Translates HTTP with support for resumable downloads into NBD, as introduced earlier.
• nbdkit-split-plugin(1): Concatenates multiple files in sequence similar to cat(1) and presents them through NBD.
• nbdkit-partitioning-plugin(1): Represents multiple files as partitions in the same disk image and presents them through NBD.
• nbdkit-xz-filter(1): Transparently decompresses xz files. If the xz file is made in a block-wise manner (refer to the man page of nbdkit-xz-filter(1)), it enables precise random access to that xz file down to the block level.
• nbdkit-partition-filter(1): Identifies and presents specified partitions in a disk image.
• nbdkit-luks-filter(1): Provides transparent access to LUKS1 in user space.
• nbdkit-iso-plugin(1), nbdkit-floppy-plugin(1), nbdkit-linuxdisk-plugin(1): Use directories as data sources and present them as filesystem images of iso9660, fat, ext2/3/4 respectively.
• nbdkit-ext2-filter(1): Presents an individual file within an ext2/3/4 filesystem image.