A faulty configuration update for MS Windows was the reason behind one of the world’s most extensive technology disruptions ever. The outage on July 19, 2024 was triggered by an error during a CrowdStrike software update. This caused a system collapse, which resulted in the ‘blue screen of death’ (BSOD) screens on the affected machines. For now, cybersecurity company CrowdStrike has addressed the error and they say that systems are being restored to their operational state. ETCISO spoke to a number of cybersecurity leaders and top voices for their opinion on the fiasco.

“As per the CrowdStrike report the update was designed to target newly observed, malicious named pipes (inter application communications) being used by common C2 frameworks in cyberattacks. It is unthinkable that a company like CrowdStrike did not test before releasing it globally. There are untold truths in this saga which need to be brought out into the open and CrowdStrike should publish a clear RC with respect to this incident. In the future, we should have a contractual clause in dealing with critical endpoint protection solution providers,” says Dr. Durga Prasad Dube, Global CISO, Reliance Industries and a Ph.D. in cybersecurity.

“I am surprised to see that there are many companies and critical systems which were allowed to have real-time sensor updates without a change management approval process. There is a potential to have more such supply chain threats in the near future,” adds Ravi Burlagadda, Sr Vice President – Information Security at Jio Platforms Limited.

“The pervasive IT outage on July 19, 2024, throwing a ‘Blue Screen (BSOD) and hanging systems around the globe, disrupted installations and almost grounding businesses globally. This widespread disruption impacted critical services across multiple industries including airlines, banks, and hospitals. This outage highlights how digital infrastructure is vulnerable to disruption and so too is our dependency on it for critical services, especially as this was not the result of a cyberattack or malicious activity. This underscores the crucial importance of robust and reliable backup and recovery solutions for ensuring business resilience. Preparedness is key to data resilience and secure backup and recovery solutions are not just an IT concern, they are a strategic imperative for any organization aiming to safeguard its future. By investing in comprehensive data resilience strategies, businesses can ensure that they are well-equipped to navigate the uncertainties of the digital age and maintain continuity and trust in the face of adversity,” says Sandeep Bhambure, Vice President and Managing Director, India & SAARC, Veeam Software.

Many questions left unanswered

The fiasco also leads us to think that the basics of cybersecurity and hygiene should never be set aside. Had the basics been followed rigorously, maybe such a huge catastrophe could have been avoided.

Moreover, it is a common belief that the risk of software bugs leading to misconfigurations is quite substantial. Therefore, shouldn’t there be a central policy whenever an enterprise-wide software agent is being implemented? Shouldn’t upgrades be tested vigorously before they are sent for installed and implemented?

“Whenever we forget the basics, fiascos are bound to happen. Risk and impact analysis is the key. When we roll out enterprise-wide software agents being managed via central policy, the risk of a software bug in these agents and misconfiguration of central policy is huge. Do we test upgrades in lower environments for these enterprise wide agents? Or do we trust the OEMs on the basis of their net worth or company face value,” asks Vijay Verma, SVP & Head Cyber Security Engineering, Jio Platforms Limited.

Level of Microsoft’s culpability

Of all the operating systems, why should Microsoft only be a victim to such an update? Why did it not impact other operating systems?

“There should be a clear explanation from Microsoft. Why was such a patch designed in haste for Microsoft platforms? What was the specific threat that they were trying to protect? Was Microsoft aware of this? If this was a problem with Microsoft platforms, which carried the potential of being exploited then why did Microsoft not release a patch? What is the arrangement Microsoft has with OEMS for releasing patches,” asks Dube.

“This is a black swan event. As per their recent Post Incident Report, Crowdstrike pushed a patch on Windows, without knowing that there was a bug in the Content Validator, which resulted in passing the validation despite containing the problematic content data. This content when processed by the Crowdstrike sensor, resulted in an out-of-bounds memory read triggering an exception that has caused BSD. And there are no magic bullets here. The weekend was tough for IT teams. Delta Airlines are struggling to recover even after 5 days. The elephant in the room is the possibility of fraud and bad actors are preparing to attack as people recover because they would be most vulnerable then. I feel that today is when you should invest in breach-ready segmentation, if you have not done it yet. The key is to focus on being breach ready right after this situation blows over,” says Agnidipta Sarkar, VP CISO Advisory, ColorTokens.

All these questions need to be clarified in detail. However, irrespective of the action taken by Crowdstrike and Microsoft, utmost care should be exercised while rolling out patches. Also, it is time to create an efficient framework where all patches as well as signatures can be tested before their rollouts.