Running Unikernels on the Bhyve Hypervisor on FreeBSD

Introduction

Bhyve is the type-2 hypervisor for FreeBSD. You might think that Linux has wiped out everything else in the server space but you'd be wrong. Some very famous companies still use FreeBSD such as Netflix and NetApp and yes, I just checked again - even HackerNews uses FreeBSD. Why? Some of the many reasons often cited are better latency and higher I/O.

The Nanos unikernel has great support for all of the major hypervisors, with KVM probably being the most supported as it is the most prevalent being the first choice on Linux. However, on FreeBSD we don't have access to KVM. The option there is bhyve. You can run qemu but won't receive access to hardware virtualization, which if you are hosting it yourself you really kind of want/need. So let's see how this might work on bhyve.

This was tested on FreeBSD 13.0. First things first - let's install a few dependencies:

pkg install qemu-devel bhyve-firmware

Note: qemu is only used if using 'ops run' right now. We're looking at integrating OPS with bhyve directly.

Let's try this small little go webserver:

package main

import (
  "fmt"
  "net/http"
)

func main() {
  http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Welcome to my website!")
  })

  fs := http.FileServer(http.Dir("static/"))
  http.Handle("/static/", http.StripPrefix("/static/", fs))

  http.ListenAndServe(":8080", nil)
}

Next, we'll cross-compile:

GO111MODULE=off GOOS=linux go build

Next we'll specify that we want the UEFI loader with this config.json and build with the nightly release branch:

{
  "Uefi": true
}

ops build -n -c config.json g
Bootable image file:/home/me/.ops/images/g.img

Now let's create a tap/bridge pair so we can take advantage of the networking super-powers of FreeBSD:

ifconfig tap0 create
sysctl net.link.tap.up_on_open=1
ifconfig bridge0 create
ifconfig bridge0 addm em0 addm tap0
ifconfig bridge0 up

Finally, we can run our application by pointing it at the image we built and including the firmware:

bhyve -AHP -s 0:0,hostbridge -s 1:0,virtio-blk,/home/me/.ops/images/gg.img \
-s 2:0,virtio-net,tap0 -s 3:0,virtio-rnd -s 31:0,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-l com1,stdio nanos

If it goes well you should see the following:

BdsDxe: loading Boot0001 "UEFI Misc Device" from
PciRoot(0x0)/Pci(0x1,0x0)
BdsDxe: starting Boot0001 "UEFI Misc Device" from
PciRoot(0x0)/Pci(0x1,0x0)
en1: assigned 172.16.41.136
en1: assigned FE80::2A0:98FF:FE61:5E76

curl -XGET http://172.16.41.136:8080/
Welcome to my website![~]$ curl -XGET http://172.16.41.136:8080/
Welcome to my website![~]$ curl -XGET http://172.16.41.136:8080/
Welcome to my website![~]$

To cleanup:

bhyvectl --destroy --vm=nanos

Pretty easy huh? Are you a bhyve/freebsd user? Want more native OPS integration? Let us know.