August 16, 2024 | Dennis T Bailey
In July 2024, CrowdStrike, a leading cybersecurity firm, experienced a significant outage due to a faulty update to its Falcon sensor for Windows. This issue stemmed from a configuration update intended to enhance the platform’s protection mechanisms against new threats. However, a logic error in the update caused widespread system crashes, leading to blue screens of death (BSOD) on approximately 8.5 million devices globally. The outage affected various critical industries, including finance, transportation, and healthcare, resulting in grounded planes, disrupted hospital operations, and significant financial losses for many companies.
Such an outage, although internal, raises concerns about the potential impacts of supply chain attacks where attackers could target essential software used by numerous organizations to cause widespread disruptions. This incident underscores the importance of preparing for and mitigating the risks associated with overreliance on critical software, whether it’s security software like CrowdStrike, operating systems like Windows, or any other widely used software.
Understanding Supply Chain Attacks
Supply chain attacks involve infiltrating systems through vulnerabilities in third-party providers or widely used software components. These attacks have far-reaching consequences, as evidenced by the recent 3CX supply chain attack in March 2023. In this attack, North Korean-linked hackers, known as Kimsuky, compromised the software supply chain of 3CX through a double supply chain attack. Initially, the hackers infiltrated the financial software firm Trading Technologies and embedded backdoor code into their X_Trader application. When a 3CX employee installed this infected app, it allowed the hackers to gain access to 3CX’s network. They then compromised a server used for software development and corrupted a 3CX installer application, spreading malware to potentially hundreds of thousands of 3CX customers. This double supply chain attack demonstrates the complex and far-reaching impacts such incidents can have.
Risks of Over-Reliance on Single Vendors
Although not an attacker-driven incident, the CrowdStrike outage highlights the issues with supply chain dependence on a single vendor. When a critical provider experiences a disruption, the effects can cascade through its client base, leaving many organizations vulnerable. This reliance creates a single point of failure, which attackers could exploit to maximize their impact. Even trusted solutions are not immune to flaws, and relying solely on one provider can introduce significant risks to an organization’s security posture.
Strategies for Enhancing Resilience Despite Platform-Specific Outages
To mitigate these risks, corporate security management should consider several key strategies:
Backup and Recovery Plans
One essential strategy is to ensure regular and reliable backups of critical systems and data. Implementing a robust disaster recovery plan can help quickly restore affected systems to a pre-incident state. This includes having offline backups that are not affected by online outages. For example, regularly backing up all critical data using solutions like Veeam or Acronis and storing these backups both on-site and in the cloud can ensure they are accessible even if local systems fail.
Controlled Environment Testing
Testing all updates in a controlled staging environment before deploying them to production systems can help identify potential issues without risking widespread disruption. However, it’s important to note that rapid response updates from CrowdStrike are typically pushed automatically, and organizations may not have the opportunity to test these updates beforehand. Using a subset of non-critical machines to test updates from CrowdStrike and other vendors can help detect any potential issues before a full rollout.
Implementing Advanced Network Security Measures
Comprehensive network security, such as firewalls and intrusion detection/prevention systems (IDS/IPS), can monitor for and mitigate threats that might arise from compromised endpoints. Deploying Cisco’s IDS/IPS solutions, for example, can detect and prevent malicious activities on the network, providing a safety net even if endpoint protections fail.
Developing and Regularly Updating Incident Response and Business Continuity Plans
These plans should include steps for dealing with complete system failures and protocols for quickly isolating affected systems, communicating with stakeholders, and restoring services. A detailed incident response plan that outlines roles, responsibilities, and specific actions to take in an endpoint protection failure can be crucial. Including communication strategies and fallback options ensures that the organization can continue to operate even during a significant disruption.
Maintaining Strong Vendor Relationships and Clear Communication Channels
Having direct lines of communication with vendors like CrowdStrike can expedite the remediation process during outages. Regular meetings with CrowdStrike representatives can help stay updated on potential issues and ensure quick support in case of emergencies.
A Fictional Company’s Response to a Global Active Directory Attack
Imagine TechSecure, a mid-sized financial services company that heavily relies on Microsoft Active Directory (AD) for identity and access management. In this scenario, a sophisticated global cyberattack targets vulnerabilities in AD, leading to widespread outages and unauthorized access across multiple organizations. Here’s how TechSecure effectively manages the crisis using the strategies outlined above:
Backup and Recovery
TechSecure’s IT team immediately initiates their disaster recovery plan. They start by restoring critical AD data from their most recent offline backups, ensuring that essential services can continue to operate. These backups, managed through Acronis, allow for a swift recovery of vital directory information.
Incident Response
The company’s incident response team isolates the affected AD servers to prevent the spread of the issue. They follow a predefined incident response protocol, which includes communicating the situation to all employees and stakeholders, providing clear instructions on how to proceed.
Network Security
Despite the AD failure, TechSecure’s network remains secure thanks to their robust Cisco IDS/IPS solutions, which continue to monitor and protect against network-based threats. This network security layer helps prevent any further compromise during the incident.
Cross-Platform Solutions
While their AD systems are compromised, TechSecure relies on alternative identity management systems for critical operations. They switch to a backup identity management system provided by Okta, ensuring that user authentication and access controls remain functional. This cross-platform approach ensures that not all operations are halted, allowing some business functions to continue unaffected.
Vendor Communication
TechSecure immediately contacts Microsoft for support, leveraging their strong vendor relationship to expedite the resolution process. Regular updates from Microsoft help TechSecure’s IT team stay informed and adjust their response as needed.
Business Continuity
With a well-defined business continuity plan in place, TechSecure’s critical business operations are quickly rerouted to alternative systems and processes. This plan includes manual processing of financial transactions and temporary use of cloud-based services to maintain customer service and operational efficiency.
Conclusion
The CrowdStrike outage serves as a stark reminder of the importance of preparing for supply chain attacks and the risks associated with overreliance on critical software. By diversifying security solutions, managing supply chain risks, and implementing robust incident response, business continuity, and monitoring systems, organizations can mitigate these risks and enhance their overall security posture.
For more detailed information on the 3CX supply chain attack and its impact, visit Trend Micro’s detailed report.