Special Report: IT Disruptions Continue as CrowdStrike Sees Crisis Receding

In the US, more than 1,500 flights were canceled for the third successive day on Sunday, with Atlanta-based Delta Air Lines in particular struggling even as CrowdStrike said the outages are easing.

CrowdStrike said a “significant” number of the 8.5m devices affected by last week’s global IT outage are back online

CrowdStrike said it was also testing a new technique to reboot systems more rapidly amid experts' warnings that a full recovery from Friday's IT failure could take weeks.

In the US, more than 1,500 flights were canceled for the third successive day on Sunday, with Atlanta-based Delta Air Lines in particular struggling, while in the UK, 45 flights were canceled on Saturday.

Delta Chief Executive Ed Bastian said in a message to customers Sunday that flight cancellations continued as the airline tried to recover its systems and restore operations. He noted that the pause in Delta's operations resulted in more than 3,500 Delta and Delta Connection scrubbed flights. Delta has been offering waivers to affected customers.

Bastian noted that one of their crew tracking-related tools was affected and unable to effectively process the unprecedented number of changes triggered by the system outage.

NHS England warned of delays in the UK as healthcare services recovered from the outage. It said patients with appointments this week should continue to attend unless told not to. (Dan Milmo / The Guardian)

Related: Reuters, Associated Press, BBC News, BBC News

Over the weekend, more details surrounding the massive outage triggered by a CrowdStrike update continued to emerge, with CrowdStrike's CEO apologizing for the incident and the company issuing more technical information about how a faulty update to its Falcon system for Windows hosts occurred.

As organizations worldwide struggled to remediate the problem, Crowdstrike CEO George Kurtz apologized for the outage. Kurtz said in a letter to CrowdStrike customers and partners, "I want to sincerely apologize directly to all of you for today's outage. All of CrowdStrike understands the gravity and impact of the situation. We quickly identified the issue and deployed a fix, allowing us to focus diligently on restoring customer systems as our highest priority." He added, "We know that adversaries and bad actors will try to exploit events like this. I encourage everyone to remain vigilant and ensure that you're engaging with official CrowdStrike representatives."

CrowdStrike then released more granular technical details on how the file update bricked Windows machines. CrowdStrike said, "On July 19, 2024, at 04:09 UTC, as part of ongoing operations, CrowdStrike released a sensor configuration update to Windows systems. Sensor configuration updates are an ongoing part of the protection mechanisms of the Falcon platform. This configuration update triggered a logic error, resulting in a system crash and a blue screen (BSOD) for impacted systems.

The sensor configuration update that caused the system crash was remediated on Friday, July 19, 2024 05:27 UTC."

The configuration files are "Channel Files" and are part of the behavioral protection mechanisms used by the Falcon sensor. CrowdStrike said, "Updates to Channel Files are a normal part of the sensor's operation and occur several times a day in response to novel tactics, techniques, and procedures discovered by CrowdStrike. This is not a new process; the architecture has been in place since Falcon's inception."

The company explained that "Channel File 291 controls how Falcon evaluates named pipe1 execution on Windows systems. Named pipes are used for normal, interprocess or intersystem communication in Windows."

The update was designed to target newly observed, malicious-named pipes being used by common C2 frameworks in cyberattacks. The configuration update triggered a logic error that resulted in an operating system crash.

The company said, "CrowdStrike has corrected the logic error by updating the content in Channel File 291. No additional changes to Channel File 291 beyond the updated logic will be deployed. Falcon is still evaluating and protecting against the abuse of named pipes."

CrowdStrike CrowdStrike has published a new "Remediation and Guidance Hub" that collects details related to its faulty update.

Despite CrowdStrike's quick contrition and rapid action, some experts fault the company for not following industry practices of adequately testing software updating procedures. "What it looks like is, potentially, the vetting or the sandboxing they do when they look at code, maybe somehow this file was not included in that or slipped through," said Steve Cobb, chief security officer at Security Scorecard, which also had some systems impacted by the issue.

Security researcher Patrick Wardle said, "It's very common that security products update their signatures, like once a day... because they're continually monitoring for new malware and because they want to make sure that their customers are protected from the latest threats." The frequency of updates "is probably the reason why (CrowdStrike) didn't test it as much," he said.

Patrick Anderson, CEO of Anderson Economic Group, says the meltdown's cost could easily top $1 billion.

Lawsuits against CrowdStrike and demands for compensation will undoubtedly follow Friday's catastrophe. CrowdStrike has not mentioned whether it intends to compensate affected customers. (CrowdStrike, CrowdStrike, CrowdStrike, CrowdStrike, CrowdStrike, Zeba Siddiqui / Reuters, Chris Isidore / CNN )

Related: CISA, 404 Media, Bloomberg, Wired, Wall Street Journal, The Independent, Krebs on Security, The Record, Neowin,  TechCrunch, CSO, CrowdStrike, PCMag, The Guardian, Tom's Guide, Forbes, SC Media, StateScoop, East Bay Times, CNBC, CRN, WeLiveSecurity, Stack Diary, New York Times, International Business Times, Livemint, Business Today, Forbes, Los Angeles Times, New York Times, The Economic Times, TweakTown, Cybernews.com, Pixel Envy, The Cyber Express, Cyber Daily, Newser, SecurityWeek, iTnews, Wired, GovTech. Where's Your Ed At?, Reuters, Washington Post, Jen Easterly, Bloomberg, South China Morning Post, r/technology, Decipher, Data Breach Today, CSO Online, New Zealand Herald, CBS News, Security Week, The Verge, The Verge

Microsoft said the global tech outage related to CrowdStrike affected nearly 8.5 million Microsoft devices and released an automated tool to remove CrowdStrike's faulty update.

"While the percentage was small, the broad economic and societal impacts reflect the use of CrowdStrike by enterprises that run many critical services," Microsoft said.

Microsoft said CrowdStrike has helped develop a solution to help Microsoft's Azure infrastructure accelerate a fix. It added that it was working with Amazon Web Services and Google Cloud Platform and sharing information about the effects Microsoft saw across the industry.

Microsoft released a custom WinPE recovery tool to find and remove the faulty CrowdStrike update, helping to avoid the laborious process of manually removing it. "The signed Microsoft Recovery Tool can be found in the Microsoft Download Center: https://go.microsoft.com/fwlink/?linkid=2280386."

To use Microsoft's recovery tool, IT staff need a Windows 64-bit client with at least 8 GB of space, administrative privileges on this device, a USB drive with at least 1 GB of storage, and a Bitlocker recovery key if required (Lynn Doan and Matt Day / Bloomberg and Lawrence Abrams / Bleeping Computer)

Related: Microsoft, Microsoft, BBC News, Neowin, GeekWire, Thurrott, iPhone in Canada Blog, PC Mag, TechCrunch, The Verge, Bloomberg, Engadget, Wall Street Journal, Reuters, Bleeping Computer

As expected during times of crisis, the CrowdStrike outage sparked bad actors to exploit the turmoil to engage in scams and other malicious activities, particularly CrowdStrike support scams targeting some IT professionals working on remediation who might be desperate for solutions.

"We know that adversaries and bad actors will try to exploit events like this," CrowdStrike founder and CEO George Kurtz said in his apology statement.

"I encourage everyone to remain vigilant and ensure that you're engaging with official CrowdStrike representatives. Our blog and technical support will continue to be the official channels for the latest updates."

In an alert, the Cybersecurity and Infrastructure Security Agency said, "Of note, CISA has observed threat actors taking advantage of this incident for phishing and other malicious activity. CISA urges organizations and individuals to remain vigilant and only follow instructions from legitimate sources.

CISA recommends that organizations remind employees to avoid clicking on phishing emails or suspicious links."

The UK's NCSC warned about increased phishing related to the CrowdStrike outage.

Clare O'Neil, Australia's home affairs minister, said, "What we are seeing some reporting of is attempts to conduct phishing through the incident that just occurred."

She said small businesses, particularly, were receiving emails from people pretending to be CrowdStrike or Microsoft and seeking bank details to access a reboot to fix the error.

"I ask Australians to be really cautious over the next few days about attempts to use this for scamming or phishing," she said. "If you see an email, if you see a text message that looks a little bit funny, that indicates something about CrowdStrike or IT outages, just stop. Don't put any details." (Lily Hay Newman / Wired, CISA, The Guardian)

Related: Associated Press, NCSC, Newser, Latestly, TechCrunch, Reuters, RTE, Decipher, PerthNow, Data Breach Today, The 420, CyberGuys, Audacy, Mirror, Financial Times, Bleeping Computer

More significant questions about the CrowdStrike outage are emerging about the fragility of the modern digital ecosystem and the belief that some key players hold too much power.

"In a globally interconnected economy, we need to ensure that we have the resilience" when an event like this happens, said Anne Neuberger, the deputy national security adviser for cyber and emerging technologies.

"This is not something that is new, but it has been accelerated by technology and by the interconnectivity," Sir Jeremy Fleming, the recently retired leader of GCHQ, said.

Regulators and lawmakers across the political spectrum raised the alarm that the sprawling outage showcases the danger of so much power concentrating on one firm, Microsoft. Over the past two years, critics have alleged that Microsoft, with a near monopoly on office productivity systems in many sectors, has too much power and has failed to pay sufficient attention to the security of its products.

Lawmakers on at least three congressional panels, the House Oversight, House Homeland Security, and House Energy and Commerce committees, requested that Microsoft and CrowdStrike brief members about how the outage took place and how it impacted agencies. (David Sanger / New York Times and Cristiano Lima-Strong and Cat Zakrzewski / Washington Post)

Related: The Guardian, The Atlantic

Crypto firms and mainland China were seemingly unaffected by the CrowdStrike software update.

According to reports, no crypto-related firms had reported any errors or outages in their services.

Pumpfun, a meme coin platform run on the Solana blockchain that has itself experienced outages in the past, posted on X (formerly Twitter), "The London Stock Exchange is down but pump dot fun is up, Do you understand?"

SunnySide Digital founder and CEO Taras Kulyk said Bitcoin was safe because it relies on no Windows machines.

China remained unscathed because very few organizations rely on CrowdStrike for threat detection and will not buy software from an American firm that has been vocal in the past about Beijing's cyber-security threat.

Moreover, China is not as reliant on Microsoft as the rest of the world. The dominant cloud providers are domestic companies such as Alibaba, Tencent, and Huawei. (Protos, Jason Nelson / Decrypt and Nick Marsh / BBC News)

Related: Crypto Times, Cointelegraph, Cryptonews, Decrypt, Slashdot, Ecns