A faulty software update issued by security giant CrowdStrike has resulted in a massive overnight outage that’s affected Windows computers around the world, disrupting businesses, airports, train stations, banks, broadcasters and the healthcare sector.
CrowdStrike said the outage was not caused by a cyberattack, but was the result of a “defect” in a software update for its flagship security product, Falcon Sensor. The defect caused any Windows computers that Falcon is installed on to crash without fully loading.
“The issue has been identified, isolated and a fix has been deployed,” said CrowdStrike in a statement on Friday. Some businesses and organizations are beginning to recover, but many expect the outages to drag on into the weekend or next week given the complexity of the fix. CrowdStrike CEO George Kurtz told NBC News that it may take “some time for some systems that just automatically won’t recover.” In a later tweet, Kurtz apologized for the disruption.
Here’s everything you need to know about the outages.
What happened?
Late Thursday into Friday, reports began to emerge of IT problems wherein Windows computers were getting stuck with the infamous “blue screen of death” — a bright blue error screen with a message that displays when Windows encounters a critical failure, crashes or cannot load.
The outages were first noticed in Australia early on Friday, and reports quickly came in from the rest of Asia and Europe as the regions began their day, as well as the United States.
Within a short time, CrowdStrike confirmed that a software update for Falcon had malfunctioned and was causing Windows computers that had the software installed to crash. Falcon lets CrowdStrike remotely analyze and check for malicious threats and malware on installed computers.
At around the same time, Microsoft reported a significant outage at one of its most used Azure cloud regions covering much of the central United States. A spokesperson for Microsoft told TechCrunch that its outage was unrelated to CrowdStrike’s incident.
What is CrowdStrike and what does Falcon Sensor do?
CrowdStrike, founded in 2011, has quickly grown into a cybersecurity giant. Today the company provides software and services to 29,000 corporate customers, including around half of Fortune 500 companies, 43 out of 50 U.S. states and eight out of the top 10 tech firms, according to its website.
The company’s cybersecurity software, Falcon, is used by enterprises to manage security on millions of computers around the world. These businesses include large corporations, hospitals, transportation hubs and government departments. Most consumer devices do not run Falcon and are unaffected by this outage.
One of the company’s biggest recent claims to fame was when it caught a group of Russian government hackers breaking into the Democratic National Committee ahead of the 2016 U.S. presidential election. CrowdStrike is also known for using memorable animal-themed names for the hacking groups it tracks based on their nationality, such as: Fancy Bear, believed to be part of Russia’s General Staff Main Intelligence Directorate, or GRU; Cozy Bear, believed to be part of Russia’s Foreign Intelligence Service, or SVR; Gothic Panda, believed to be a Chinese government group; and Charming Kitten, believed to be an Iranian state-backed group. The company even makes action figures to represent these groups, which it sells as swag.
CrowdStrike is so big it’s one of the sponsors of the Mercedes F1 team, and this year even aired a Super Bowl ad — a first for a cybersecurity company.
Who are the outages affecting?
Practically anyone who during their everyday life interacts with a computer system running software from CrowdStrike is affected, even if the computer isn’t theirs.
These devices include the cash registers at grocery stories, departure boards at airports and train stations; school computers, your work-issued laptops and desktops at office, airport check-in systems, airlines’ own ticketing and scheduling platforms, healthcare networks, and many more. Because CrowdStrike’s software is so ubiquitous, the outages are causing chaos around the world in a variety of ways. A single affected Windows computer in a fleet of systems could be enough to disrupt the network.
TechCrunch reporters around the world are seeing and experiencing outages, including at points of travel, doctor’s offices and online. Early on Friday, the Federal Aviation Administration put in effect a ground stop, effectively grounding flights across the United States, citing the disruption. It looks like so far the national Amtrak rail network is functioning as normal.
What is the U.S. government doing so far?
Given that the problem stems from a company, there isn’t much that the U.S. federal government can do. According to a pool report, President Biden was briefed on the CrowdStrike outage, and “his team is in touch with CrowdStrike and impacted entities.”
The pool report said Biden’s team is “engaged across the interagency to get sector by sector updates throughout the day and is standing by to provide assistance as needed.”
In a separate tweet, Homeland Security said it was working with its U.S. cybersecurity agency CISA, CrowdStirke, and Microsoft — as well as its federal, state, local and critical infrastructure partners — to “fully assess and address system outages.”
There will no doubt be questions for CrowdStrike (and to some extent Microsoft, whose unrelated outage also caused disruption overnight for its customers) from government and congressional investigators.
For now, the immediate focus will be on the recovery of affected systems.
How do affected customers fix their Windows computers?
The major problem here is that CrowdStrike’s Falcon Sensor software malfunctioned, causing Windows machines to crash, and there’s no easy way to fix that.
So far, CrowdStrike has issued a patch, and it has also detailed a workaround that could help affected systems function normally until it has a permanent solution. One option is for users to “reboot the [affected computer] to give it an opportunity to download the reverted channel file,” referring to the fixed file.
In a message to users, CrowdStrike detailed a few steps customers can take, one of which requires physical access to an affected system to remove the defective file. CrowdStrike says users should boot the computer into Safe Mode or Windows Recovery Environment, navigate to the CrowdStrike directory, and delete the faulty file “C-00000291*.sys.”
TechCrunch will keep this report updated throughout the day.
TechCrunch’s Ram Iyer contributed reporting.
