Unix is Dead

12/07/18

Ian Eyberg

What a crazy few weeks it has been. Marriott started it off with the announcement that over 500M records had been stolen from its database and worse - they know the attackers were in their systems for up to 4 years which was during an acquisition and even after the prior year when they found a RAT (remote access trojan).

As of if that wasn't enough Quora announced over 100M records had been breached. That's arguably not as bad as say your passport and credit card getting stolen (and all these starwood points!! - Marriott *does* have nice hotels) but in this day and age when we have artificial intelligence driven reporters combined with the latest and greatest in semantic context mining driven applications it's concerning. We already have a few firms out there that mine the social media landscape - however mining context for anonymous users is not the same thing as mining context for users with full emails and other 'personal'/'private' information. I'm sure this corpus will eventually wind it's way up into the public and give data scientists lots of juicy tidibits to gnaw on for a while.

Lastly our attention draws to kubernetes CVE-2018-1002105- which at first blush seemed like everyone was making a big deal out of nothing. After all if you are going to expose your infrastructure to the general public not sure what to tell you. Having said that, after having a few conversations it really starts to drill in on the point that we have not just not been improving our security (yes double negative, problem?) we've been making it worse.

I for one am dissapointed that there wasn't a matching logo/website/full branding for this one - isn't that the hip new thing to do for security reporting or is that only for people presenting at conferences like Enigma?

This k8s vulnerabilty is worse than a remote root. Why? At least the remote root only gives you access to one server. From there you need to sniff some traffic, discover some ssh key laying around, whatever. But this? Might as well spin up some cryptominers and max out the compute resources. Hopefully that company has auto-scaling turned on and then you can do some serious damage. There have already been reports of a few companies on the public clouds with exposed servers being proactively warned by AWS/GCE cause when an angry customer comes back and complains about the hundreds of thousands of dollars of a bill they got at the end of the month it's easier just to turn off the spigot beforehand.

It blows my mind but as of this writing *this* is still up and running - http://192.99.142.232:8220/222.json . OVH!? What the hell.

However, enough trash talking k8s. Let's talk about the real issue that we are starting to see here which is the real meat and potatoes of this article. The real issue is:

Unix is Dead

That's right. Unix is dead. We all collectively killed it the moment we started hyperscaling and blitzscaling and more importantly moved to the cloud.

You see back in the 90s we still had to vertically scale our servers. We had Mars the database and Jupiter the webserver and such. Then something interesting happened. VMWare brought back an ancient concept known as virtualization and commercialized the hell out of it. They did such a good job that a small bookstore up in Seattle started launching a few services based on the concept and forever changed the way developers interact with servers. That small bookstore is now massively propped up by said services massive profit.

At the same time the markets had just reached the point of irrational exuberance and everything came to a screeching halt around 2008. This drove a ton of money into private companies in the tech sector and that created an intense demand for hyperscaling the crap out of everything. Does it run on the cloud? Have a check.

Engineers at well known tech companies don't deal with one database - they deal with thousands of databases. Likewise many of them don't have fancy scheduling systems to use and it's a well known fact that the linux scheduling algorithm a) sucks and b) is not built for workloads that we are putting on there now. You have to keep in mind that Linux was 'already obsolete' when it came out in 1991 in the words of Andrew Tanenbaum. Oh, btw - small little factoid - Minix is more popular now.

So what's a devops to do? Separate the workloads. You wouldn't want that database with it's intense write workload to harm your nginx pool of web app servers right? It's not even that to be honest - it's the fact that juggling that many servers is just a pain in the ass if you don't segregate your programs into their own distinct vms.

The cloud in general is finally starting to realize this with gvisor from Google and firecracker from AWS. I don't even know if I would qualify those as first generation responses though. A duck is still a duck.

The First Nail

So we have engineers already treating their virtual machines as processes. That's the first nail in the coffin.

The second nail in the coffin was networking. It used to be back in the 90s that many programs would perform IPC and have one process talk to another and maybe they share memory and maybe they do other things. Now we utilize things like grpc and protobufs and all the other different frameworks out there. At the most basic level even your lowly 5 person SaaS app is going to have some json/rest stuff going on - eg: everything is going across the network now regardless. That's the second nail.

There are a few other nails like memory ballooning but let's get to the other one I like to talk about.

The Final Nail

The entire concept of the unix user has gone completely out of the window. No, I'm not talking about all the kubernetes/docker people running their daemons as root - I'm talking about the fact that the classic unix user model doesn't make sense in a world where you have a hyper-layer above the linux vm contained within something like AWS IAM or GCE IAM. GCE even goes to the extent of pre-populating in your vms users that are attached to those profiles. (and then of course for whatever batshit reason doesn't take care of it when you delete the user).

This was probably the final nail in the coffin.

So you say - we get these points. They're fair arguments but what are we supposed to do? What comes next? Do we revert to a new and improved unix? How do we move forward?

Listen - I will be the first to admit that I don't have a clear set of suggestions in place. This is more of me getting everyone to agree to a reckoning that things are broken because of the cloud and we all need to figure out what to do about it.

We clearly believe unikernels have a very strong proposition to fix some of these problems with their 4 point security model but we aren't going to sit there and bullshit you saying it's the grand unified theory of everything. It's just one piece. We have other ideas that we'll be talking about in the future as well but my ask of you all reading this article is what should we do?

What are your proposals to fix the cloud? Cause this is not working.