Entering the ARM Server Arena
With the announcement of the 64-bit extension of the ARM architecture (AArch64) in 2011, ARM set the stage for not only consolidating its leadership in the mobile microprocessor market, but also entering the edge and server markets. Several silicon manufacturers started developing their AArch64 implementation on Systems-on-Chip (SoCs) whose performance characteristics were getting more and more similar to those of personal computers, and it was only a matter of time before ARM-based servers appeared. ARM microprocessors had the potential to deliver better efficiency in terms of computing resources versus power consumption, and their hardware virtualization capabilities made them an ideal candidate for entering the cloud computing arena. With the AWS Graviton platform, Amazon was the first major cloud provider to make AArch64 cloud instances available to the general public.
Nanos on AWS Graviton
AArch64 was the first architecture alternative to x86-64 that Nanos gained support for, and Nanos has been running on AArch64 for a while now. The initial porting of the Nanos kernel to AArch64 focused on getting the kernel to run under a QEMU hypervisor, either on ARM hardware with KVM virtualization (such as a Raspberry Pi 4), or on a fully-emulated environment such as a generic non-ARM computer. More specifically, the mechanism being used to boot the system, i.e. load the kernel in memory and start executing it, makes use of the `-kernel` QEMU command line option, which means that QEMU takes care of the initial booting steps, thus the disk image containing the kernel and user application does not have to be a bootable image; in addition, the initial porting targeted a specific virtual machine, i.e. the virt QEMU machine type, and this allowed the kernel to make some assumptions about what peripherals are available in the system and use hardcoded memory addresses to interface with various peripherals such as the PCI host bridge or the interrupt controller.
If we want a unikernel application (or any operating system, for that matter) to be deployed to a cloud instance, the primary storage device attached to the instance must contain a bootable disk image; this means that the system firmware installed on the instance (which is the software that runs during the early stages of the boot process, before the user-installed software takes over) must be able to "jump" to a given location in the disk image and let the machine instructions contained at that location take it from there and do what is needed to bring up the operating system. This is where the bootloader comes in: the job of a bootloader, in simple terms, is to do the initial steps required to load the operating system. Writing a bootloader was thus the first thing we needed to do in order to make Nanos run on AWS Graviton. Another important part that was missing from the initial porting is the ability of the kernel to probe at run time information about what system peripherals are available and where they can be accessed: while this was not necessary to be able to run on a single platform such as the virt QEMU machine, moving to a different platform required making the kernel code more generic so that it can adapt to the particular machine it is running on. Finally, we needed to add support for more peripherals (such as the serial port, disk and network devices), as well as enhance some of the existing drivers (such as the interrupt controller driver), in order to account for the differences between the peripherals present in a virt QEMU machine and those in a Graviton instance.
Deploying a Nanos Unikernel Application to a Graviton Instance
In this section we show how to use Ops, the orchestration tool for Nanos, to create an Amazon Machine Image (AMI) containing a Nanos unikernel application and deploy it to an AWS Graviton instance. We start with a statically linked program, which can be deployed from any Linux or MacOs computer, and then we discuss how to deploy a generic application, which currently requires working from a 64-bit ARM Linux machine.
Deploying a Golang Web Server
In order to make the deployment process as seamless as possible, when generating a cloud image from a given program, Ops looks for and automatically includes in the image the dynamic libraries needed by the program; but this does not currently work when the program is an ARM executable file and Ops is running on something other than an ARM Linux machine. Since the ARM architecture is not (yet) widespread in the personal computer market, chances are that you are not reading this from an ARM-based machine. Luckily, if the program in question is statically linked (i.e. it's not linked to external dynamic libraries) Ops does not need to find any libraries, so we can use it to deploy the application from any Linux or MacOS computer.
How to create a statically linked program depends on the programming language the program is written in and on the build tools being used; furthermore, creating an executable file that is meant to run on a different architecture than the one on which the file is being created usually requires cross-compilation tools, and this also varies depending on the language the program is written in. If we have a Golang program, the Go build tools make it very easy to create an executable file for an arbitrary architecture and operating system. In this section, we show how to create an AArch64 Linux executable file (which is what the AArch64 Nanos kernel is able to run) from a Go program that implements a simple web server, and how to deploy the web server to an AWS Graviton instance.
First of all, we need to install the Go build tools. If you are on a Mac, open a shell terminal and type the following:
brew install gosudo apt install golang
package main
import (
"fmt"
"net/http"
)
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Welcome to my website!")
})
http.ListenAndServe("0.0.0.0:8080", nil)
}
GOOS=linux GOARCH=arm64 go build webg.goNow we are going to use Ops to deploy our web server to the cloud. If you haven't installed Ops yet, install it with the command below:
curl https://ops.city/get.sh -sSfL | shOps supports several cloud providers, including AWS; in order to use Ops with AWS, make sure the pre-requisites outlined in the documentation are satisfied. Then, before creating an image we need to use a little trick in order for Ops to pick the ARM kernel and bootloader files to be inserted in the image instead of the default x86 files (this is only a temporary workaround that will no longer be needed in the future when Ops support for ARM improves):
- Build a dummy image using the latest Nanos nightly release for x86:
This is only needed in order for Ops to update its internal cache with the timestamp of the latest nightly build, so that the x86 nightly release will not be downloaded again when executing the next commands.
ops build -n webg - Delete from the Ops cache the build artifacts downloaded above:
rm -rf ~/.ops/nightly/bootx64.efi ~/.ops/nightly/k* ~/.ops/nightly/mkfs ~/.ops/nightly/dump - Download the latest Nanos nightly release for ARM, extract its contents and copy them in the Ops cache directory ~/.ops/nightly: this effectively replaces the x86 nightly release with the ARM release
{
"RunConfig": {
"Ports": ["8080"]
},
"CloudConfig" :{
"ProjectID": "prod-1000",
"Zone": "us-west-1c",
"BucketName": "my-s3-bucket",
"Flavor": "t4g.nano"
}
}
ops image create -t aws -c config.json -n webgops instance create -t aws -c config.json webgDeploying a generic application
In the previous section we saw how to create a simple web server application and deploy it to a Graviton instance from any Linux or MacOS computer. The application executable file we created could be easily turned into a working unikernel image because it was statically linked, i.e. it had no dependencies on external libraries to be loaded at run time. Most applications, however, do have dependencies on external libraries, so they won't start correctly unless all libraries they depend on are found in the image filesystem. As mentioned above, Ops automates the process of looking up and adding to the image the set of needed libraries, but in order for this to work for ARM images Ops needs to be run on an ARM Linux machine. If you don't have an ARM Linux machine at your disposal, you can create an AWS Graviton instance with a Linux operating system (AWS offers a number of Linux AMIs for Graviton) and connect to it via SSH. Then, you can take any AArch application that runs on Linux and turn it into a Nanos unikernel image: just follow the same steps as in the previous section to install Ops, retrieve Nanos nightly release artifacts for ARM, create an image, and start a Graviton instance (after making any adaptations to the config.json file that may be needed for your application, and replacing the "webg" executable name in the Ops commands with the name of your application).
As of now, we cannot create working ARM images using the packages that are available via the Ops "pkg" set of commands, because these packages are built for the x86 architecture. But as Ops support for ARM improves, in the future there will be ready-to-use AArch64 packages just like there are x86 packages now.
Technical Details
ARM SoCs for the embedded and mobile markets have historically been characterized by large differences between devices from different vendors, with each manufacturer implementing its own set of system peripherals, resulting in operating system kernels having to be configured and built specifically for a given SoC in order for them to be able to run on that SoC. With time, as more and more mutually incompatible SoCs were released, it became apparent that this approach was not sustainable, and there was a need for some standardization that allowed a single kernel binary to be able to run on many different SoCs. This standardization took the form of the Device Tree (DT) specification: a set of rules to build an interface between hardware and software, which allows an SoC vendor to provide a formal description of the hardware components available in a given platform so that the software knows what drivers to load and how to configure them to properly operate the hardware.
Device Trees are widely used in the embedded market, but for the server market a different approach is being taken, which involves the use of the Advanced Configuration and Power Interface (ACPI) standard. The motivation behind ACPI is the same as what prompted the adoption of DTs, i.e. to provide a separation between software and hardware that allows operating systems to more easily handle the variability between different incompatible hardware platforms. But while a DT provides a mere description of hardware components, ACPI tables (which constitute the data that the system firmware makes available to the operating system) contain also executable code that can be invoked by the OS to operate on the hardware. The rationale of this approach is that the job of OS developers is made easier if they don't need to know the internal details of the hardware peripherals, and can delegate some of the work to the system firmware. This obviates the need for operating systems to implement drivers that are aware of the internal details of each peripheral that can be found on any server platform, and makes it easier for operating system vendors to provide a single image that can run on many different platforms.
To facilitate adoption of the 64-bit ARM architecture in the server market, ARM published a set of requirements (in terms of functionalities to be made available to operating systems and hypervisors) that all AArch64 server platform vendors are encouraged to follow: the Server Base System Architecture (SBSA). The interface between platform vendor-provided firmware and OS software is described in a separate document, the ARM Base Boot Requirements. Here, requirements for different platform types are specified in "recipes"; more specifically, the recipe for servers is referred to as SBBR (Server Base Boot Requirements). Among other things, the SBBR prescribes that ARM servers must boot the operating system via UEFI (more on this later), and must implement the ACPI specification.
AWS Graviton instances are SBSA- and SBBR-compliant.
The rest of this section delves more deeply into the details involved in the development activity that has been done in order to port Nanos to the Graviton platform. There were three main areas of work:
- writing a bootloader
- implementing run-time probing of hardware characteristics
- adding new peripheral drivers and enhancing existing drivers
The Bootloader
As mentioned, the booting process chosen by ARM in the SBBR is based on the Unified Extensible Firmware Interface (UEFI) specification. This means that the system firmware in an SBBR-compliant server during boot looks for a FAT32-formatted partition (called the EFI partition) containing an EFI/Boot/ folder with a bootaa64.efi file in it. This file must be compliant with the PE/COFF format (a file format used for executables, object code, and libraries) and contains the bootloader code: thus, the system firmware will jump to the entry point of the bootloader.
A while ago, when we added support for Hyper-V and Azure Generation-2 instance types we implemented an UEFI bootloader for Nanos; while that runs exclusively on x86 machines, its modular design made it fairly easy to port it to AArch64. Its job is to locate the primary disk device that contains the boot partition (i.e. the partition where the Nanos kernel file is located), load the kernel in memory, retrieve from the system firmware basic information on the resources available in the machine (i.e. the map of physical memory regions and the location of the ACPI root table), set up a data structure to be read by the kernel containing the above information, and finally jump to the kernel entry point. From that, the kernel takes over: it reads the contents of the data structure set up by the bootloader, and uses them to:
- calculate the total amount of available memory
- initialize the various memory heaps that are used throughout the kernel
- parse the ACPI tables
ACPI
Memory addresses at which various system peripherals are accessed are not architecturally defined as fixed values, and thus may differ between machine types; likewise, the presence of specific peripherals and their characteristics may vary. Thus, the kernel needs a way to retrieve this information at run time, and the way it does this is by parsing the ACPI tables.
The ACPI standard is widely used in the personal computer and server markets, and ARM chose it as part of its requirements for SBBR-compliant AArch64 servers. The system firmware in a machine that implements ACPI populates a set of standardized data structures (called ACPI tables) and puts them in memory for the operating system to read. One of these tables, called Root System Description Table (RSDT), contains references to the other tables, and is thus the main entry point for the kernel to inspect all the tables. Therefore, the kernel only needs to know the location of the RSDT (referred to as the Root System Description Pointer, or RSDP) in order to be able to retrieve the information it needs to operate the system peripherals. As mentioned above in the bootloader section, the RSDP value is included by the bootloader in the data structure passed to the kernel.
Nanos already had an ACPI implementation, based on the acpica library, which we added when we implemented support for attaching and detaching volumes to/from running instances; so we could leverage this library to allow the kernel to retrieve the information it needs to operate a Graviton instance, such as the type and memory address of the serial port, and the memory address of other peripherals such as the PCI host bridge and the interrupt controller.
Device Drivers
The existing Nanos kernel for AArch64 was only equipped with the device drivers needed to run on the virt QEMU machine; this means that the only storage and network devices that were supported are those implementing the virtIO specification. However, AWS Graviton instances use different peripheral types; more specifically, they use the same peripherals as those present in x86 instances based on the Nitro hypervisor, i.e. NVM Express (a.k.a. NVMe) disk controllers and Elastic Network Adapters (ENA). Since Nanos already supported AWS Nitro-based instances, we already had NVMe and ENA drivers for the x86 version of the kernel, so adding them to the Aarch64 kernel was trivial.
During the kernel porting activity, after enabling verbose debugging messages in the kernel code we discovered that the serial port driver was not working: not only there were no messages printed to the serial console, but trying to print anything resulted in the kernel being stuck in an infinite loop. A working serial port is often an invaluable debugging tool when porting low-level software such as an operating system kernel to a new platform. It turned out that Graviton instances are equipped with a different kind of serial port than the one used in the virt QEMU machine: more specifically, Graviton instances have a 16550-compatible serial port; thus, we wrote a simple 16550 UART driver (similar to the one we already use for the x86 PC platform) to get those useful debug messages out to the AWS serial console. In addition, in order to make the kernel able to run in both virt and Graviton machines, we implemented run-time probing of the serial port peripheral, which parses the Serial Port Console Redirection (SPCR) ACPI table to retrieve the information of interest.
What probably took most time in the porting activity was the driver for the interrupt controller. Most ARM devices, from simple 32-bit embedded microcontrollers to high-end 64-bit server CPUs, incorporate a GIC (Generic Interrupt Controller). The specifications on how the GIC operates have undergone many revisions, to accommodate the needs of different types of CPUs and efficiently handle modern virtualization technologies; the ARM SBSA, i.e. the reference architecture to which AWS Graviton instances adhere, prescribes the presence of a GICv3 in AArch64 servers.The existing port of the Nanos kernel to AArch64 had a driver for the GICv2 interrupt controller, which is needed in order for hardware virtualization to work when running on a Raspberry Pi 4, so we needed to enhance the existing GIC driver to be able to support the GICv3. More specifically, we already had basic support for the GICv3, but what was missing is support for Message-Signaled Interrupts (MSIs), which is the mechanism used by PCI devices (such as the disk controller and the network adapter) to assert interrupts to the processor; the existing driver supported MSIs for the GICv2, but not for the GICv3. MSIs in the GICv3 are handled as Locality-based Peripheral Interrupts (LPIs), and are configured via the Interrupt Translation Service (ITS). The ITS is a fairly complex part of GICv3 that requires the kernel to allocate a number of tables in memory and configure the ITS to use these tables for its internal operations; then, various commands need to be sent to the ITS to enable the routing of specific interrupts coming from PCI peripherals to a CPU. Once we implemented all of this, we finally started seeing interrupts from the disk and network devices being properly serviced, and this was the final missing piece that allowed a fully functional unikernel application to run on a Graviton instance.
Conclusions
Implementing and piecing together all the parts needed to bring up Nanos on AWS Graviton ended up being a substantial amount of work, but now that it's done we have an AArch64 kernel that is no longer limited to running on the single platform for which the initial ARM port was done. The code now implements the basic standards prescribed by the ARM SBSA and SBBR documents: we have a UEFI bootloader and an ACPI-aware kernel, so we expect to be able to support fairly easily other ARM server platforms that may come up in the future, from AWS or other cloud vendors.
