CrowdStrike changes internal testing after global outage

• Experts estimate that this incident was the largest IT outage in history, and it provided a wake-up call to companies about how reliant they've become on single vendors.
• Airlines were canceling flights through Monday. Hospitals had to reschedule non-urgent medical procedures. Some local 9-1-1 systems went down, and even Starbucks mobile ordering stopped working.

State of play: The new post provides the first in-depth look at where CrowdStrike went wrong when pushing out what was meant to be a routine software update to its antivirus tools on Friday.

Zoom in: CrowdStrike was attempting to deploy an update designed to collect information about new threat tactics that went out early Friday morning.

• The update went through CrowdStrike's "Rapid Response" protocols, where the company quickly deployed a tested software update to all customers at once. Typically, this type of process is used to remediate potential cyber threats.
• However, faulty data in the update "passed validation despite containing problematic content data," the company said in its post. The data then trigged a so-called "out-of-bounds memory read" that bricked devices, creating the Blue Screen of Death.
• "This unexpected exception could not be gracefully handled, resulting in a Windows operating system crash," the CrowdStrike wrote in its findings.
• The company is now adding another level of testing to its internal processes and will start staggering the release of these types of updates across customers to prevent a similar incident from happening again.

Between the lines: CrowdStrike is currently trying to bounce back after the devastating wave of outages.

• A source familiar with the situation says that a significant majority of customers are now up and running.
• CrowdStrike has also been holding war rooms with Microsoft to remediate the problem as quickly as possible, and Microsoft has not had any conversations about revoking the level of access CrowdStrike products have to Windows systems, the source added.

Yes, but: There's no good solution to prevent another catastrophic, widespread tech outage.

• Adding too many layers of internal testing could slow down how quickly companies can patch networks or block malicious activity on their systems.
• But not having enough testing can allow more faulty data to sneak in and cause another outage.

What to watch: Security experts have so far praised CrowdStrike's new blog posts, saying it has a rare level of details.

• But some customers still say the company isn't doing enough to make amends.
• TechCrunch reported Wednesday that CrowdStrike was sending $10 UberEats gift cards to affected partners. However, some of those cards don't appear to working either.
• A CrowdStrike spokesperson told Axios that it did not send gift cards to customers or clients. "We did send these to our teammates and partners who have been helping customers through this situation. Uber flagged it as fraud because of high usage rates," the spokesperson wrote in an email.

Editor's note: This story has been updated with a statement from CrowdStrike and corrected to reflect that it was faulty content data (not faulty code) that CrowdStrike shipped, that the company was trying to collect information about new threat tactics used in the software update that went out early Friday morning, and that it sent gift cards to affected partners (not to customers).