In the war to defend against a rising tide of vulnerabilities that continually expose our critical assets to attack in new and creative ways, we’re being backed into a corner.
The current prevailing strategy is to identify vulnerabilities, prioritize them, then wait for the perfect 3-day weekend to make the IT department patch them. It’s an antiquated approach that forward-thinking organizations are leaving behind in favor of a more scalable methodology: continuous patching.
Scott Savenelli, Manager of Vulnerability Management and Patching Services at Novacoast, wants patch windows to fade into the history of failed IT practices, though he’s not naive about the reasons why they are necessary.
The Looming Threat of Scale
Software vulnerabilities represent the top attack vector against endpoints, and endpoints represent 90% of targets from threat actors. It’s clearly a chink in any organization’s armor and requires a strategy to manage the risk associated.
It seems obvious that the answer is to just patch with a fixed version and move on, but…it’s complicated.
The shear quantity of vulnerabilities that are discovered and tracked daily multiplied by the vast number of assets under the average enterprise umbrella make for a nearly insurmountable task, and that’s best case scenario. The intensity and volume of new vulnerabilities, along with the staggering speed with which bad actors and nation states can weaponize them today is unprecedented. And there is no reason to believe the rate of new vulnerabilities or the forces that exploit them are going to pull back any time soon.
The challenge gets tougher while the solutions to deal with it are experiencing a rift in philosophy among vendors. To make matters worse, there is nothing institutional in place to guide organizations. While the popular frameworks like CIS and NIST provide some guidance for best practices in IT, and do cover vulnerability management, there are no separate defined standards for patch management.
And vulnerability management is pointless without patching.
The Patching Problem
Why is patching so hard? If you’ve ever run an update only to discover that changes to software have broken some critical service or rendered a previous configuration invalid then you can appreciate the risk that patching represents to an enterprise IT department presiding over large infrastructure and potentially thousands of endpoints. A broken update can bring a company to its knees.
Because uptime is often valued over security maintenance, IT groups have been forced to make updates in a designated window of time so as to minimize interruption to employee productivity. This is called the “patching window” and tends to be characteristic of Microsoft-based environments. It’s an effort to perform a bunch of postponed updates all at once.
It is very common for organizations to struggle with this cross-section of security and IT management. Success in vulnerability management seems to elude the groups that focus heavily on identifying vulnerabilities versus developing a method to effectively apply the patches. Some vendors aren’t helping as they continue to promote solutions that completely omit the remedial portion of the problem.
Continuing to tolerate long vulnerability windows is a game of roulette; it’s not a matter of if but when a standing vulnerability will be exploited in an attack.
Scott Savenelli – Novacoast
Increased Exposure
Security-wise, there are a few problems with sticking to the patching window methodology. First, it extends the window of vulnerability by delaying the fix. That period between when a vulnerability is published and when the patch becomes available is a particularly dangerous time when the vulnerability is called a “zero-day,” meaning the bug is public and the developers have zero days to fix it. It’s the only time an attacker can attempt to exploit the vulnerability with high confidence that it exists on target systems.
Second, by having multiple updates stacking up, that means there could be multiple vulnerabilities to chain in a given attack. Imagine a graph of what overall cumulative vulnerability over time looks like: An asymmetric ramp getting higher and higher until the updates are applied, then it plummets to a nominal value and begins climbing again with every new CVE published. It looks like the teeth of a saw blade; how tall and how far apart are your saw teeth?
To quote Novacoast’s Savenelli: “Continuing to tolerate long vulnerability windows is a game of roulette; it’s not a matter of if but when a standing vulnerability will be exploited in an attack.”
Solution: Continuous Patching
To shorten the vulnerability window and keep outstanding vulnerabilities at a manageable level, some organizations have adopted a patching methodology that’s akin to the rolling distribution model used by many modern Linux distros. Those distros continuously update without any single large shift to a new or radically different architecture or compatibility. Even in the Linux world, a distro upgrade threatens the most chance of upset to the system.
The idea with continuous security patching is simple: Apply the individual patch immediately upon release. The benefits are two-fold: It shortens the vulnerability window to its absolute minimum, and it isolates any breakage to a single patch versus a suite of patches and products.
Will things break? Maybe. But as a vulnerability management strategy it reduces exposure the most effectively. Bite-size updates reduce chance of breakage.
In the example of endpoint patching, the updates take less time because they’re only patching a single application at a time. Users can defer their updates up to 8 hours but no more depending on company policy. Eventually the culture and business practices that struggled with patching windows before adapt to a more frequent but less painful experience.
Automation To Meet Volume
Ask a CISO who’s tasked with reviewing vulnerability reports every month whether progress is being made, they’ll likely plant their face firmly in palm. Finding new vulnerabilities present in their environment is the easy part—new ones are reported every day. Identifying the cavalcade of vulnerabilities in itself accomplishes very little to achieve security. But the workload associated with actually patching is a very real bottleneck constrained by manpower and time. How to clear the blockage?
The recent piece on the shrinking lifespan of certificates makes the case for scriptable infrastructure renewing TLS certificates en mass due to a predicted post-quantum era when super powerful computers can break cryptography in a matter of days or hours. It’s conceivable that we’ll see a future where patch windows are useless against the volume of new vulnerabilities, and the only way to effectively make progress is by automating the process, either by integrating patching into VM or by orchestrating asset management tools.
It won’t happen tomorrow but it’s likely to be necessary in a not-too-distant future.
There Are Caveats
There is of course an argument for why patch windows are necessary. While some organizations might be free to play fast and loose with the unknown side effects of applying updates, others may have zero tolerance for downtime due to the nature of their business. These environments require a thorough and aggressive testing process prior to actually patching.
A business should exact their own calculus for security risk versus downtime risk. A low profile entity remaining vulnerable with a medium-severity bug for a few weeks may not represent the same risk as a high profile entity immediately seeing attacks against a fresh zero-day. But should systems supporting a critical ERP integration cause errors in global point-of-sale terminals from an update gone sideways? It could cost millions. Timely patching may not work for everyone.
Do VM Solution Vendors Do Continuous?
Some do. A number of vendors currently in the vulnerability management space are starting to incorporate automated patching. This is in contrast to legacy players who remain dedicated to a discovery-type solution which only accomplishes inventory of vulnerabilities, leaving the patching to be sorted out however the customer can manage.
It’s still a more common goal with vendors to focus on maximizing coverage—finding vulnerabilities in every single device in the organization. That’s a good thing, because bugs can’t be fixed unless we know what’s got them. But at some point visibility is useless without remediation.
How Do I Move to Continuous Patching?
There is no overstating this: shifting paradigms to a new methodology for remediating vulnerabilities is a huge undertaking. The act of applying a patch to a single vulnerable endpoint is not really the difficult part. Any environment is easily manageable at smaller scale.
What is difficult is managing execution and balancing the business considerations that can be threatened by any shift in IT asset management at larger scales of thousands of endpoints. Adopting continuous patching is tantamount to a sea change, one that will require motivation from a singular leader. This is very much an institutional challenge to overcome.
Many larger organizations have parallel teams that are each tasked with a distinct purview, whether it’s scanning/tracking vulnerabilities, or making direct endpoint changes/imaging assets, or presiding over asset configuration with tools like SCCM. Management that can get these teams working in more collaborative, less-siloed manner will be successful at coordinating a shift to a new, more agile patching methodology.
The simple answer is to patch as soon as possible after every patch is made available. The not-so-simple answer is to work out the institutional challenges that prevent that from easily happening. Change the equation to tolerate more potential hiccups from patch-introduced issues, to homogenizing teams tasked with vulnerability remediation, and start valuing metrics of remediation over those for visibility.
Conclusions
- Continuous patching should replace patch windows, if feasible.
Similar to shortening certificate lifespans, some level of automation will be the only way to realistically keep up with today’s rate of new vulnerabilities. Continuous patching shortens the vulnerability window for minimum exposure. - Consolidate features in tooling.
Using a vendor with both vulnerability discovery and patching in a single tool shrinks overhead and eliminates unknowns in the process. - Use a managed vulnerability and patch service.
Experts who can track vulnerabilities and perform the patching reduces risk and shortens the process as they tend to be less tentative with applying updates. - Prepare to shift institutionally.
Continuous patching will require collaboration and an integration of teams that’s not common in today’s heavily siloed enterprise organization. An involved singular leadership will be required to achieve a seamless process, from scanning to patching, in the shortest interval.