If you manage an IT organization, you have likely had several meetings about the latest malware, Petya (or NotPetya as some are referring to it), and your meetings are focusing on the escalation of malicious intent that these exploits are having. You are also discussing legacy operating systems on your network that you haven’t been able to upgrade due to reasons such as proprietary applications, user training hurdles, license and upgrade costs, and your IT team’s time and availability.
Personally, I have seen some relief that the exploits have patches available, and that even older operating systems had patches put out by Microsoft (leaving IT departments scrambling to ensure they apply the patch, and that the patch didn’t cause any issues.) I truly don’t want to spread FUD (fear, uncertainty and doubt) but I do wonder if an attack will come out before the appropriate patches are readily available. Vendors like Microsoft will be quick to respond no doubt, but there would certainly be a scramble to ensure systems are quickly updated and tested.
Reflecting on last week, I have been thinking a lot about Broadleaf’s systems and the systems we manage for our customers. I have been putting myself in the position that an exploit is already out, imagining a worst- case scenario that the Vendors are playing catch-up, and my focus is what knowledge and tools will help me avoid catastrophic issues. I wanted to take a moment to share my thoughts. Below is a list of items I consider critical in managing an IT infrastructure.
- I must have a reliable report of systems including operating systems and current patch status.
- I require a system in place that can deploy patches from a singular source and report success / failure of the patch.
- My user base must be trained in the technical environment of today. I can no longer afford to have naive users.
- I must be able to control, or at least have some visibility into the web and email traffic in my company.
- I need to know that my end points have the amount of protection that their level of access requires.
- My backups are my last defense. I need to know that they are running consistently and are secure from being compromised.
- I must have a plan of action in the event that data is compromised.
- I must have a “Post-Mortem Process” that will allow me to evaluate what happened and help prevent it happening again.
With that list above, I now have a skeleton for a strategy to ensure data protection. None of the points above are overly sophisticated, but all of them require some form of administration and a toolset specific to its need. I will now break out my thoughts on each item.
As your IT footprint grows, it is amazing how easy it is to lose track of systems. Was that laptop turned in after the employee quit? Was the dev server off-lined and destroyed? I often find myself performing site surveys and discover a system that was long shutdown found its way to being powered on. Sometimes these systems cause trouble and are identified. Other times, they sit unattended, unpatched and as a result unsecured. Having processes in place and a toolset that continuously monitors the network for changes is more than helpful. In today’s IT landscape, it is a critical function. The tool I utilize sweeps via ICMP and if a device is found then WMI, SNMP and SSH can be called on to discover information such as uptime, the OS, and the last user login. This is very helpful for us in keeping an eye on hundreds of systems.
I am fortunate that my monitoring and patching servers are tied together and I encourage you to have a similar tie in. This helps my team ensure that systems are not only patched on a regular basis, but also that if we have to implement a patch rapidly, the infrastructure is already built and ready. Should any systems have issues installing a specific update or patch, it is critical for the patching system to inform you of this.
An Informed User base:
If you are reading this, you are likely in IT. In your career you likely did, or do participate in user support. Few users understand how their actions relate to the network as a whole. Fewer still understand the criticality of the IT policies that enforce security measures. Users are, however, often the first to know and sense an issue on the network. Almost every call regarding a virus or malware I have received includes a statement from a user that they felt something was wrong- a slow system, popups and application crashes often clue the user in that something is amiss. A user base that is aware that these can be symptoms of a much larger problem is critical. Enabling the users to reach out directly for IT assistance is key! Our users can contact our helpdesk 24 x 7 and have their systems reviewed, antivirus checked and scans initiated. This line to support enables a much-needed dialog between the IT department and the user community.
I know my company and most companies we work with have a system to in place that works to stop malicious emails before reaching the user. This however doesn’t take into account Gmail, Yahoo mail, Microsoft Hotmail, etc. Furthermore, peer to peer applications that can be run in your environment download and upload across an array of systems. It may be tempting for users to enjoy your faster, and likely more consistent bandwidth as well as storage capacity for downloads. There are a lot of ways to black list IP ports, and websites. You just need to be aware that the behavior is occurring. I am a fan of Meraki firewalls for their ability to present me with clear usage data by system. There are other firewalls such as Cisco ASA with Firepower that offer even more advanced filtering and monitoring options. These advanced features are great for companies that have engineers that are able to monitor and act on these alerts. The difficulty in managing these systems can be daunting and this challenge was a driving force in our decision to offer a SIEM (Security and Event Management as a service).
End Point Protection:
All of our systems are critical of course and all of the end points should have some form of protection. However, there are times additional protection may be warranted. A system that maintains all the department data stores for example would likely benefit from snapshot technology and enhanced malware protection that detects encryption or a high ratio of writes to reads occurring (such as Sophos). Perhaps an environment that sees laptops leaving the environment should have enhanced protection features such as disk drive encryption and enhanced GPO policies such as disabling USB storage. I have also seen issues with devices that are off network for long periods of time being missed in patch schedules. Finally, when I think of end points, I have to think about phones (small computers that I don’t have visibility into) that are often not patched and have applications installed from potentially untrusted sources! Pro tip – Meraki makes it easy to create a separate network that is restricted to prevent phones from connecting to resources.
The Humble Backup:
No one wants to think about restoring from a backup. Unless the timing is fortunate, you will be looking at some time lapse between the backup completing and the need for the backups being realized (RPO). However, this time delta can be minimal with a strong strategy. Veeam, for example, can get you up and running quite painlessly (especially if the low bar is tape restores!) Whatever your strategy, remember that the solution must be able to inform you of failures, be able to restore in an acceptable amount of time, and be protected from the corruption that we are working to failsafe. A USB drive attached to the server can be just as vulnerable to malware as the local disk drive!
The Best Laid Plans of Action:
It is unlikely that you will be able to have a full grasp on every critical aspect of the company’s systems should the time come to remediate a system wide issue such as a ransomware outbreak. Variables such as month end reporting, payroll processing, and development cycles are just of few of the factors that determine what is required immediately in the event of a mass remediation. Ensure you have a clear documented path to the systems restoration. You will want key contacts’ names and phone numbers from department heads, key decision makers, and to your disaster recovery or backup provider if applicable. Your documentation should also list key and critical devices, as well as systems that are dependent upon each other. This plan should have some abilities to stage tests or mock events should your environment allow for it.
Understanding the past, or doomed to repeat it:
A post-mortem review is usually shown from the lens of a failure, and to be clear, failures in our protections and processes should be reviewed in detail. Understanding how a breach occurred is the best and most obvious way to prevent the lapses from occurring again, and without remediation they likely will. Sophos, for example, will help trace back to the ingress point of malware on your environment and your firewall may already have tools awaiting you to enable them. In my time in IT, I have found that a consistent review through post-mortem analysis answers questions and provides information you will need moving forward. There are several examples online, or perhaps you already have one. My advice is that your review should include:
- Defining the issue. I can’t stress this enough. Complicated issues are too often never fully understood.
- Note the actions taken and the results (I always try to get times for this as well.) This will let you know if the actions you took were needed, helped the issue, made it worse, or simply had no affect at all. This can help you greatly in the future!
- The resolution. I come from a troubleshooting background. All solutions seem obvious when found. However, in context of a critical outage things are a bit more complex. Knowing the resolution or steps for remediation is a straight forward requirement that I have seen overlooked. If the root of the issue was never identified, then perhaps the issue was just mitigated.
- Action Items moving forward. With the issue defined, an understanding of the steps taken to resolve, and a resolution documented, make sure to see this as an exercise for the future, not a review of the past. Determining how the issue could have been prevented and setting those action items as a process will serve you well. So will understanding what steps assisted in remediation.
I hope this has been insightful. I always welcome feedback or a discussion. Feel free to contact me at firstname.lastname@example.org with input or to schedule some time.