Confidential Computing with AMD SEV Enabled Unikernels

We brought AMD SEV into the Nanos unikernel.

SEV (secure encrypted virtualization) is an extension of the existing AMD SME (secure memory encryption) extension. Essentially, memory pages (including code and data) are encrypted outside the purview of the hypervisor using dedicated hardware. This hardware encrypts data as it is written to DRAM and decrypted upon read. In practice it is designed to prevent your cloud from snooping in on your application.

Confidential computing is a term to that describes protecting data in use and is a part of the end to end encryption framework.

When someone tells you "it is encrypted" it is helpful to understand what exactly they mean. Does it mean that the website serves over a TLS connection? Does it mean that the disk is encrypted? What actually does it mean when someone says "we use encryption"? End to end encryption is a framework for answering this question and describing three states of encryption:

• Encryption At Rest protects your data while it is being stored. Example: password hashing.
• Encryption In Transit protects your data when it is moving between two points. Example: HTTPS/TLS
• Encryption In Use protects your data while it is being processed. Example: Confidential Computing

Confidential Computing provides the last piece of end-to-end encryption: encryption-in-use. You can generalize this as "disk vs memory encryption". Think about our encryption at rest example where we might store a secret encrypted. What happens when we decrypt it? Normally it would show up as cleartext in memory and an attacker that has access to the host could potentially dump it. This isn't a hypothetical - nation state threat actors were already caught in high profiles CSPs... a decade ago. This is where confidential computing steps in.

On AWS and GCP you might use something like KMS to encrypt your AMI or disk images, however that only protects them when they are sitting in storage. When your application is running all that super sensitive memory is in the clear. GCP uses AMD SEV to give you the opportunity to apply confidential computing.

What happens is that when a vm boots an encryption key is made for the lifetime of the instance and it's something that Google themselves don't have access too. The keys are not accessible by the hypervisor itself. Furthermore, a process called attestation occurs. Attestation uses a TPM (trusted platform module) to verify the machines integrity at boot.

You can't really attest a guest from the hypervisor itself because the whole point is to ensure the hypervisor, or those running it, are not malicious. This is a key selling point of SEV. All of this is provided by hardware-based capabilities just as virtualization itself uses hardware based capabilities.

There are currently a few limitations on GCP when using confidential computing namely:

• No Live Migration
• No GPU/TPU (no pci passthrough)
• Requires NVME - can't use scsi

Another interesting limitation that you probably won't be exposed to is that each machine that supports SEV has a limited number of slots in the memory controller for storing encryption keys. For instance Naples has 16 and Rome has 255. Unless you are building your own cloud you probably won't really run into this.

We'd expect many of these limitations to go away in the future as SEV is an extension that has been updated frequently and continues to get updated. For instance SEV-SNP was a newer extension that was released just last year. SNP is short for secure nested pages and extends SEV to protect against data replay and memory remapping attacks.

It was kinda funny that the day of, or the day after, of us pushing initial SEV support the CacheWarp attack landed. This wasn't the first time SEV has been attacked and definitely won't be the last. A few years ago a voltage fault injection was demonstrated via a paper called One Glitch to Rule Them All: Fault Injection Attacks Against AMD’s Secure Encrypted Virtualization. It's quite normal for hackers to explore new technologies like this but I wouldn't let this deter you from adopting it. After all SSL/TLS are continuously attacked but we don't go around telling everyone to just use plain old http.

The most significant changes we needed to make to enable this feature was that we had to revisit how we mapped memory. Before we used a linear mapping for both kernel and DMA transfers.

Now all non-DMA is allocated from a page backed heap and all DMA memory is allocated from a linear backed heap. Why? This is because normal SEV kernel/userspace is encrypted but DMA transfers can not be encrypted (today). So what's an example of the difference? For instance your network traffic coming across the NIC will be using DMA and that will be sent and received unencrypted so you'll want to be using something like TLS to cover your bases here. Remember SEV only covers one third of the end to end encryption framework. This is why it's important to always ask "what is encrypted" when someone tells you "it is encrypted".

We also had existing support for the gVNIC (google virtual ethernet) network driver but only on ARM and we needed to added support for it for x86. SEV on GCP also requires UEFI which, again we had support for but needed to ensure the images were using it.

SEV is still very much a moving target. Just last year, AMD introduced a new extension - AMD SEV-TIO which does enable high performance DMA without bounce buffering. Wait - what is bounce buffering? Bounce buffering is simply a technique that the kernel employs to allow devices that might only be able to address lower memory for DMA operations even though they might need a larger chunk that is not available. So a 'bounce buffer' is used to transit the data from low memory to high memory. While the original problem that these were made for might eventually go away confidential computing throws a monkey wrench in because the system now has encrypted memory that a device can't access. This is something that is being addressed though. Consider AMD just released a technical preview last year it's probably going to be a while before we can use this.

I realize this is a lot of words so let's look at an actual example of this in action.

While we've implemented SEV for Google Cloud vms, to really understand the SEV functionality you'll want to run it on a real machine with AMD EPYC as you need to run the code in a hypervisor and see that the hypervisor can't snoop.

I believe this is possible on some AWS metal instances but haven't tested. I did boot a bare metal vultr instance though as it was only ~$1/hr (and I don't have EPYC hardware laying around).

First thing after you boot your machine you're going to want to enable SEV in the bios - it wasn't enabled in mine. Once you reboot you should see dmesg confirming that you have access to it now:

root@sevtest:~# dmesg | grep SEV
[    5.548125] ccp 0000:43:00.1: SEV API:1.52 build:4
[    5.584731] kvm_amd: SEV enabled (ASIDs 1 - 509)
[    5.584732] kvm_amd: SEV-ES disabled (ASIDs 0 - 0)

Next we want to build our unikernel. We need to enable UEFI here as that is a requirement.

{
    "Uefi": true,
    "Args": ["12345"]
}

Now we need to obtain the c-bit. The c-bit is a bit to mark if a memory page is encrypted or not. To get our cbitpos we'll extract it from cpuid:

modprobe cpuid
EBX=$(dd if=/dev/cpu/0/cpuid ibs=16 count=32 skip=134217728 | tail -c 16 | od -An -t u4 -j 4 -N 4 | sed -re 's|^ *||')
CBITPOS=$((EBX & 0x3f))

Mine turned out to be the number 51. Now let's take a look at this little program:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(int argc, char *argv[]) {
    char str[32];
    int luggagePassword = -1;

    if (argc > 1) {
      luggagePassword = atoi(argv[1]);
      sprintf(str, "My luggage password is %d\n", luggagePassword);
    }

    sleep(10000);
}

Now we can run our instance (after building the unikernel, eg: 'ops run -c config.json main'):

qemu-system-x86_64 -enable-kvm -cpu EPYC -machine q35 -smp 1 -m 1G \
-drive if=pflash,format=raw,unit=0,file=/usr/share/OVMF/OVMF_CODE.fd,readonly \
-drive if=pflash,format=raw,unit=1,file=/usr/share/OVMF/OVMF_VARS.fd \
-drive file=/root/.ops/images/main,if=none,id=disk0 \
-device  virtio-scsi-pci,id=scsi,disable-legacy=on,iommu_platform=true \
-device scsi-hd,drive=disk0 -nographic -s \
-device virtio-rng-pci,disable-legacy=on,iommu_platform=true \
-object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=1 -machine memory-encryption=sev0

Once it boots we exit to the qemu monitor and dump the memory:

(qemu) dump-guest-memory /tmp/sev 

Then we do the same thing but we leave off all the sev options:

qemu-system-x86_64 -enable-kvm -cpu EPYC -machine q35 -smp 1 -m 1G \
-drive if=pflash,format=raw,unit=0,file=/usr/share/OVMF/OVMF_CODE.fd,readonly \
-drive if=pflash,format=raw,unit=1,file=/usr/share/OVMF/OVMF_VARS.fd \
-drive file=/root/.ops/images/main,if=none,id=disk0 \
-device  virtio-scsi-pci,id=scsi,disable-legacy=on,iommu_platform=true \
-device scsi-hd,drive=disk0 -nographic -s \
-device virtio-rng-pci,disable-legacy=on,iommu_platform=true

Then we dump its memory:

(qemu) dump-guest-memory /tmp/nosev 

Now we can extract strings from our dump and see if we can find the luaggage password:

root@sevtest:~/xx/yy# strings /tmp/nosev | grep luggage
My luggage password is 12345
My luggage password is %d

As you can see without sev it is in the clear but with sev we can't find it.

root@sevtest:~/xx/yy# strings /tmp/sev | grep luggage

You might be wondering why the attacker can't just do what we just did. That's why we have the process called attestation. This allows it to prove its state and identity.

Now that you know the basics of SEV let's enable it. There is a small performance hit you'll get enabling this - some people have measured it up to 6%. Like the early days of ssl adoption I'd imagine this taking the same path where people might complain about the perf hit at first but then eventually down the road it's just accepted as best practice. This is, however, one reason why it's not on by default but it's also not available for every instance and every region anyways. For instance you need at least a Rome or Milan AMD processor. For this tutorial you'll want to enable the ConfidentialVM bool and set your flavor to n2d-standard-2:

{
  "CloudConfig": {
    "BucketName": "nanos-test",
    "ConfidentialVM": true,
    "Flavor": n2d-standard-2"
  }
}

Now once you boot your instance you can click on the logging link in the GCP console to verify that your attestation report is valid. You'll see something like this:

sevLaunchAttestationReportEvent: {
  integrityEvaluationPassed: true
  sevPolicy: {8}
}

Congrats! You've now enabled encryption in use for your unikernels in AMD SEV. Attacks such as Cache Warp will continue to be discovered but I don't think this is going to stop SEV adoption. Perhaps we'll wind up with a different standard but I think the general idea is something that will only continue to be embraced down the road and your Nanos unikernels now have support.