While compiling this blog post last couple of weeks this weekend all hell broke loose! A large-scale ransomware attack was launched against users of any Windows system worldwide. Showing us that security is all about a pre-care balance between protection and usability.
Running your infrastructure in a secure configuration is a daunting task even for security professionals. This blog provides practical advice to help administrators to harden their infrastructure following security best practices so that they can confidently deploy their services and lower their chances of being compromised.
Protecting your infrastructure successfully is all about understanding what and whom you are protecting, your infrastructure, against! If you know what and whom you are protecting against, makes taking the correct measures easier. One major measure is hardening your infrastructure.
Hardening is about securing the infrastructure against attacks, by reducing its attack surface and thus eliminating as many risks as possible. One of the main measures in hardening is removing all non-essential software programs and utilities from the deployed components. While these components may offer useful features to the administrator, if they provide ‘back-door’ access to the system, they must be removed during the hardening process.
But also, creating visibility in what goes on in the infrastructure is part of hardening your infrastructure. Making sure you will notice when an attack is/or has taken place and then making sure logs and traces are saved for law-enforcement and security specialists when needed.
Tip 1 – Less is More
Overly complex designs become harder for the IT team to manage and overlook and it makes it easier for an attacker to exploit and stay in the shadows. Simpler designs that can be easily overviewed are in basis more secure. I have seen deployments in the field where all default protocol ports had been switched for another random port, while they thought this was more secure, a simple port scan showed them that they were exposed on 17 points. Choosing a secure port to the outside, through port 443, and removing all unnecessary and obscure ports and protocols made it easier and more secure. Simplicity is security!
Tip 2 – Join the Dark Side
Look at your infrastructure and step into the darkness. At first it will feel strange to look at your own infrastructure and see how you would hack it. By doing this you will gain valuable insights which you can use when hardening. Try to answer the following questions:
- How would you get access to the infrastructure when hacking the system?
- Where do you start?
- What are the weak points?
- What is valuable in the infrastructure?
- What tools would you deploy/use?
One of the highest sought-after attack vector will be gaining access to management accounts and components. This will allow you, as an attacker, to gain access to most parts of the infrastructure.
Tip 3 – Get those Flare Packs out
So now you know what’s important in the infrastructure, like data and critical services, you can plan to protect them against attacks and take appropriate countermeasures. Within the hardening process of your infrastructure there are a few steps everyone should always consider and act upon, namely:
- Remove all non-essential software programs and utilities from the deployed components. While these programs may offer useful features to the administrator, if they provide ‘back-door’ access to the system, they must be removed during the hardening process. Think about additional software like web browsers, java, adobe reader and such. All which is not operating system or main software that has been deployed on that server, remove it! It will make maintaining an up-to-date patch level much easier.
- Patch operating systems, software, and firmware on infrastructure components. Most hacks succeed because there is already vulnerable software in use which is not up-to-date with current patch levels. So not just updated the VMware cluster, but all other software running on top of it. See also this blog post from Alex.
- Educate your staff, by deploying an employee awareness training you make sure that your employees are aware of strange behaviour and of their critical roles in protecting the organisation’s services and data. This is not only for the IT department, but for everyone within the organisation, because every organisation is becoming an IT company rapidly.
- Deploy an Access Control policy, managing access to management components is crucial for a good protection. Use the principle of least privilege. Provide the minimal privilege needed for some operation to occur. If a process or system is exploited, you don’t want to allow an attacker to gain any more access than is minimally required. Containment to keep the attackers from moving around too easily. Some standard measures and policies are:
- Do not use user accounts for admin access, reducing incidents and accidents
- Give every admin his own admin account, so it gets traceable
- Only give out access to what is needed for the job
- Limit users who can log in using Remote Desktop
- Try not to obscure hide access points, by use wrong measures. Like for instance changing protocol ports in use by random other ports and other tricks to try and hide ports and protocols in use. While this may look like a good choice at first, in practice this often makes the infrastructure harder to manage which opens other possibilities for attackers. Obscurity is not security!
- You can check which ports are in use by which service on a Windows system by using: #netstat -bona > portlist.txt | You can open the text file with for instance #notepad portlist.txt
- Use a clever Password management policy, which works for your organisation. Enforcing the use of strong passwords across your infrastructure is a valuable control. It’s more challenging for attackers to guess passwords/crack hashes to gain unauthorised access to critical systems. Selecting passwords of 10 characters with a mixture of upper and lowercase letters, numbers and special characters is a good start. Adding 2-factor authentication for Admin accounts is wise to look at, depending on what you need to protect of-course.
- Lockout policy that complements a strong password policy. Accounts will be locked after a small number of incorrect attempts. This can stop password guessing attacks dead in the water. But be careful that this can also lock everyone out of the system for a period! For service accounts, sometimes it is better just to raise alarms fast. Instead of locking the accounts. This way you gain visibility into suspicious behaviour towards your data/infrastructure.
- Add local protection mechanics, in addition to the border firewalls, intrusion detection, patching and such. You can make use of local mechanisms, like up-to-date anti-malware, firewalls and network segmentation. This way you create different rings-of-defence slowing an attacker down! For more information around adding micro-segmentation by using VMware NSX, please read these blogs on VMGuru.
- Have a recovery strategy in place, before you find out your infrastructure is breached you should know what to do when being compromised through attacks. Backup, backup, backup your data and make sure the backups cannot be accessed by an attacker to wipe them out. An offsite copy (air-gap) on any media is highly recommended to survive any attack.
Tip 4 – Secure by Design
Adding security to an already existing infrastructure is much harder and costly than thinking about it while designing a new or refreshing an existing infrastructure. In a virtual infrastructure, it is good use to build up a Master image which has been hardened from the start. Removing all known attack vectors and only opening access when functional software is added and needs specific openings or extra software to function properly. This way all builds are consistent and kept up-to-date which makes it secure in the basis.
For help by hardening those master images you can make use of the free benchmarks CIS provides. CIS Benchmarks help you safeguard systems, software, and networks against today’s evolving cyber threats. Developed by an international community of cybersecurity experts, the CIS Benchmarks are configuration guidelines for over 100 technologies and platforms.
Tip 5 – Visibility is Key
To know when you are under attack or have been breached it is vital to have visibility in the whole data flow path. You should be able to know what is ‘normal behaviour’ and what is NOT. Monitor your accounts and infrastructure for suspicious activity. Place virtual trip-wires, like e.g. creating a non-used admin account with alarms tied to it. When any activity on that account is observed, it will trigger a red alert instantly. There are several systems out there that can help you by alerting suspicious behaviour so you get aware that someone is snooping around and is trying to gain access to your infrastructure.
It is important to get alerts as soon as possible while defending against other attacks like viruses, malware and ransomware. The biggest fear of these attacks is that they may propagate to other systems fast. Having visibility into for e.g. potential ransomware activity is a big deal.
Example Systems that could help you create visibility are:
- A system that detects possible ransomware activity is for example Veeam ONE 9.5. There is a pre-defined alarm called “Possible ransomware activity.” This alarm will trigger if there is a high CPU utilisation combined with lots of writes to disk.
- VMware vRealize Network Insight can take VMs, objects, groupings and their physical elements and easily fingerprint the application and determine the internal and external flows, the client connections, etc. this way you get an analysis of what is ‘normal’ behaviour and what is not.
- VMware vCenter with alerts that are triggered on virtual trip-wires.
You can also deploy VMware NSX as a counter measure with micro segmentation to make sure the attack surface is as narrow as possible without blocking everyone to use the services. Visibility into the network and all data flows is crucial to help you protect all different rings/cells within your infrastructure. Bruno Germain did a great blog posts series around VMware NSX , visibility and containment here.
Summary
All software can be exploited. And remember software is everywhere, that hardware in your infrastructure is not running without any software. Can be as firmware or embedded in the hardware on an EPROM. All software has flaws that allow an attacker with enough motivation to exploit it. By hardening you will make it much harder for an attacker to get far fast within your infrastructure and he might skip your infrastructure and tries someone else’s.
Hardening resources
- 7 Practical tips to prevent ransomware attacks on backup storage, by Rick Vanover
- Context, Visibility and Containment – NSX Securing “Anywhere” Part V, by Bruno Germain