Patching Problems: Why Vendor Security Updates Are Letting Down Customers

Phil Muncaster investigates what organisations can do in the face of failed updates and poor communication

Vulnerability and patch management is one of the cornerstones of cybersecurity best practice. Despite an explosion in CVE volumes and evolving threat actor activity, continuous, risk-based and automated patching programmes offer organisations the best chance of minimising risk across the attack surface. Or at least they would if it wasn’t for the deteriorating quality of patches and vulnerability information released by the vendor community.

Over 18 months ago, Trend Micro’s Zero Day Initiative (ZDI) sounded the alarm about this. Unfortunately, its warnings have not been heeded. So, where does the industry go from here?

On the back foot

System administrators have never had an easy time. But between the growing complexity of their distributed IT environments, the record number of vulnerabilities being published, and the initial access brokers (IABs) scrutinising buggy code to exploit, life at the coal face has rarely seemed quite as intense.

“Many [zero-days] are well-understood classes of web vulnerability and are trivial to find and exploit” National Cyber Security Centre

The stakes are rising. Vulnerability exploitation is a top-three vector for ransomware attacks today. And zero-day exploits are appearing increasingly frequently in cyberspace – particularly ones aimed at perimeter-focused products like file transfer applications, firewalls and VPNs. In fact, last month, the National Cyber Security Centre (NCSC) was forced to issue new guidance on mitigating these risks. It warned: “Finding zero-day vulnerabilities might sound highly advanced, but many of these are well-understood classes of web vulnerability and are trivial to find and exploit.” One of these zero-days, in the MOVEit file transfer software, impacted tens of millions of end customers.

Then there are the commercial spyware makers that continue researching new vulnerabilities and exploits to sell to autocratic governments. Apple, a common target for these zero-click exploits, was forced to fix 20 zero-days in its products last year.

Once they have gained a foothold into corporate networks via vulnerability exploitation, threat actors are getting much better at moving laterally to get to the good stuff. A CrowdStrike study claims breakout time on average fell to just an hour last year – and in some cases, attackers managed to move from initial access to lateral movement in just two minutes. In short, network defenders are already on the back foot against an agile and resourceful adversary. They could do without patches also failing.

Making things worse

The ZDI’s beef is twofold. It is dismayed at the quality of patches, which it says can expose organisations to unnecessary extra risk and costs. And it is angry at the increasingly confusing and sometimes obfuscating language used in security advisories, which it claims leaves network defenders unable to gauge their risk exposure accurately.

Trend Micro’s ZDI head of threat awareness, Dustin Childs, tells Assured Intelligence there are two main issues of concern for CISOs regarding patching quality.

“The first is what it breaks. There are many examples of patches that break other software loaded on the system to be patched. We saw that in February where Microsoft patches wouldn’t even install on some Windows systems. Patches that break things are why I have been advised always to say ‘test and deploy’ rather than just ‘deploy’,” he explains.

“The other big problem comes from patches that don’t fully address the vulnerability. We see this a lot from many vendors. Changes introduce entropy to code bases, so vendors try to change as little as possible. However, in attempting to change as little as possible, they often make their patches too narrow.”

By narrowing their focus, vendors alert threat actors to the existence of a bug and give them a virtual head start on researching adjacent zero-days. This happened with a recent SmartScreen bypass zero-day that the ZDI found, which was a bypass of a previously patched bug. Other examples include a critical Outlook patch published in May 2023, which was bypassed almost immediately. The same happened to a Citrix NetScaler update from October last year.

HackerOne CISO, Chris Evans, argues that a lack of “variant testing” is often the cause of these patching snafus.

“In the world of operating systems, failure to perform variant analysis is all too common,” he tells Assured Intelligence. “For instance, certain zero-day exploits in iPhones and Android devices use variants of known patched security issues. The operating system vendors need to do good variant analyses so that this route is closed off to attackers.”

What CISOs can do

Evans claims that around 10% of vulnerabilities are fixed incorrectly.

“As an industry, we must do better,” he argues. “In the bug bounty world, we use a feature called ‘retesting’ to get hackers to perform independent validation of a proposed fix. This reduces the chance of an incorrect fix.”

“The existence of a patch raises the risk since threat actors reverse engineer and weaponise patches before most enterprises can deploy the fixes” Dustin Childs

However, that won’t help CISOs and their teams when faced with poorly engineered security updates. The ZDI’s Childs says security leaders must understand that just because a patch exists doesn’t mean the risk to their organisation is lowered.

“In many cases, the existence of a patch raises the risk since threat actors reverse engineer and weaponise patches before most enterprises can deploy the fixes,” he adds.

“They also need to employ multiple types of defence systems in cases where patching isn’t practical or has to be delayed. This includes things like disabling old protocols, implementing the principles of least privilege, segmenting networks, intrusion prevention, and extended detection and response (XDR).”

Ultimately, CISOs must view patch quality as a risk factor like any other, says HackerOne’s Evans. This will require them to understand which vendors to trust and to what level and the relative sensitivity of different corporate data types and systems.

“The most sensitive data should only be stored with the most trusted vendors, which will typically have strong controls to defend against inadequate security patches,” he argues. “This will include a bug bounty programme. Scaling out to the amazing ethical hacking community adds an extra defence against incomplete security patches.”

Time for change

For ZDI’s Childs, vendor inaction on this critically important topic over the past two years is due to a lack of coordinated customer backlash.

“Wherever they possibly can, customers should vote with their wallets. That’s not always possible. I don’t expect many to ditch Microsoft, for example. But wherever they can, they should let vendors know this is an important issue that should be improved,” he argues.

“Until there are real consequences, vendors won’t be motivated to spend the resources or the effort to improve. It will likely either take legislative action or other liability to force the change.”

One possible way to drive this change is through the cyber insurance sector. It’s already helping to promote industry best practices by demanding prospective customers improve baseline security before they are allowed to take out policies. Or, at least, by reducing the cost of coverage according to how mature policyholders’ security posture is. A similarly proactive approach from insurers could theoretically force vendors to improve patching processes, says Childs.

“If a cyber insurance company realises their client was compromised due to a patch quality issue, they could go after the vendor for remuneration,” he concludes. “I have no idea if it would work, but I think it’s just a matter of time before someone tries.”