Enabling KASLR for the Nanos Unikernel

We recently enabled KASLR on by default. We currently don't even have a way of turning it off via manifest since it is turned on before we even detect the disk from which the manifest option might be located -- in other words you have to disable it via a custom build.

To answer what KASLR is first you might want to answer what ASLR is. ASLR or address space layout randomization is a process to randomize the location in memory of a program's stack, heap, library load location and program binary location. This is a measure to help prevent an attacker from being able to easily map the locations of a program in memory. Many memory based attacks rely not only on a vulnerability that an attacker is exploiting but also the location in memory of one or (most probably more than a handful) of important addresses. Thus changing the memory layout every time you run a program vs having the same memory layout everytime makes it more difficult to figure this out. ASLR randomizes these locations every single time a program is ran so even if an attacker finds a memory address if the program restarts (eg: after a deploy) they now need to figure out the address again.

KASLR adds similar support but for the kernel itself. This has the affect of changing memory locations for the kernel and the klibs.

Nanos can map these addresses over a range of 2GB and has a 4kb alignment so the number of possible addresses are 2**19 or 524,288. This doesn't make things bulletproof but it makes things a lot more difficult.

With some debugging macros enabled you can this in action:

kernel load offset 0xffffffffcb80d000
kernel load offset 0xffffffffd6242000

Now, not everyone is fond of KASLR, including, notably, Brad Spengler (grsecurity), but some of this is somewhat historical.

However, let's put some of this in a unikernel context.

For instance one of the complaints is that you can just look at /proc/kallsyms to get the locations, albeit non-root users have zeros. ... but unikernels are not linux. Linux runs many different programs and linux isn't normally rebooted all the time. There are still plenty of people that pride themselves on systems that haven't rebooted for months or years at a time throwing the kaslr magic out the door.

That's not the case with unikernels. If you reboot a unikernel the entire system is rebooting - not just the app. If you do a deploy you have a brand new image regardless. So right off the bat unikernels benefit from KASLR a lot more than linux does.

To be clear kaslr is what is called a statistical defense. It relies on the fact an attacker needs to guess (or more realistically be able to compute) an address across a large range of potential addresses. In many cases if you guess wrong you might crash the program and you have to start over. (This is also another vote for unikernels w/the lack of a traditional userland that would enable enumerating this.)

Other weaknesses that Brad and others bring up include:

• Low entropy in the randomness that can be applied to the kernel as a whole. This used to be a much larger deal on 32 bit systems.
• The leak of a single address can reveal the random offset applied to the kernel, thus revealing the rest of the addresses.
• The kinds of information leaks needed to reveal the offset are abundant and always cropping up.

This last point in particular is an ongoing process - in fact just the other day LWN had reported on another KASLR leak and there are projects such as KASLD that present a variety of techniques of determining the base virtual address as a local unprivileged user.

KASLR also shines again with unikernels in that it is vastly easier to compromise kaslr locally via pointer infoleaks than remotely. When you have a full blown userspace it is so much easier to attack and enumerate, especially on a linux system that throws everything and the kitchen sink into /proc.

Attackers go to great lengths to find infoleaks. There's a whole art/science to obtaining function pointer leaks because they are so valuable for an attacker to have. We're going to touch on this later and why this is highly relevant to some unikernel projects.

KASLR also has other more modern issues, such as integration with firecracker.

However we feel that 'perfect is the enemy of good' here. There is also 'finer-grained' kaslr that operates at the function level but that is for a different article on a different day. :)

Mapping memory layout is a critical component for ROP.

This is an issue that also crops up in rookit development. For instance, let's say we've already popped the box in question but now we want to persist in the enemy system. We need to design a rootkit that can evade attempts to purge it and to weaponize this properly it clearly needs to be autonomous.

Keep in mind that even running as root there is a lot of things you can't actually do unless you are in the kernel. So in order for us to map things out we want to insert a kernel module.

There is a function called 'kallsyms_lookup_name' which returns the address of any symbol in the kernel's symbol table, however we'll need to use a kprobe in our module to get it:

#include <linux/module.h>
#include <linux/init.h>
#include <linux/kernel.h>
#include <linux/kprobes.h>

static struct kprobe kp = {
    .symbol_name = "kallsyms_lookup_name"
};

int __init init_test(void) {
    register_kprobe(&kp);

    pr_alert("Found at 0x%px \n", kp.addr);

    return 0;
}

void __exit cleanup_test(void) {
    unregister_kprobe(&kp);
}

MODULE_LICENSE("GPL");

module_init(init_test);
module_exit(cleanup_test);

obj-m += hello.o

all:
  make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

clean:
  make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

make && sudo insmod hello.ko

Then we can look at dmesg:

[7728052.678422] Found at 0xffffffffa81fd400

Now we can verify that this is the correct address:

eyberg@venus:~/k$ cat /proc/kallsyms | grep ffffffffa81fd400

No? Sudo make a sandwich:

eyberg@venus:~/k$ sudo cat /proc/kallsyms | grep ffffffffa81fd400
ffffffffa81fd400 T kallsyms_lookup_name

Infoleaks are not a joke. I note this as other projects in this space seem to be unaware of how big of a problem they can be.

One of the things that a lot of people might not realize is just because a unikernel architecture dictates having one process - it doesn't mean that you get to change how the underlying chip/architecture works. At the end of the day they are machines with very explicit designs. I think if you truly want some exotic os architecture it needs to be designed at the chip level. Project such as firesim and CHERI are projects to look at with this view in mind.

This is why other projects like ours, but ignore the kernel/user separation in an attempt to gain superior performance, fail to understand the severe security issues at play. I'm not speaking in the sense of "this developer made a mistake and their system might crash". I'm speaking in the sense that a malevolent person is wishing to inflict harm upon your system.

You see there is this concept of privileged vs unprivileged instructions which is a different concept from root vs. non-root. Reading the time might not be privileged but halting the process, clearing memory, changing page permissions, turning off interrupts, flushing data, etc. -- that's privileged.

asm volatile ("sti");

Being more specific, reading and writing to the control register CR0 ... that is privileged. What might we do there? We could disable write protection - that is - pages that are marked read-only could be enabled for write and at that point all your page permissions be damned - it's game over.

I don't see any actual real safe way right now to have this style of system and so we just don't do it at all. That is not too diminish the many other immensely beneficial qualities that unikernels have though.

word cr0;
mov_from_cr("cr0", cr0);
cr0 = enable ? (cr0 | C0_WP) : (cr0 & ~C0_WP);
mov_to_cr("cr0", cr0);

After we disable write protection we can now write to read-only:

#include <stdio.h>
#include <stdlib.h>

int main() {

  char *s = "world";

  s[0] = 'o';
  s[1] = 'w';
  s[2] = 'n';
  s[3] = 'e';
  s[4] = 'd';

  printf("Hello, %s\n", s);
}

This is funnily enough, also an issue with wasm. Note: this will segfault in Nanos and Linux... as it should.

So now you know why KASLR is on by default in Nanos and other systems and why it's even better in unikernel based systems.