Igor Kralj/PIXSELL/Sipa USA via AP Images
The dreaded “blue screen of death”
On the “perfect phone call,” then-President Donald Trump famously asked newly elected Ukrainian President Volodymyr Zelensky to look into a Republican conspiracy theory about the California cybersecurity firm CrowdStrike. The Democratic National Committee had hired CrowdStrike to respond to a security breach on its servers, but according to Trump, the company was secretly owned by a Ukrainian oligarch who had instructed his employees to fabricate evidence that the intrusion was carried out by Russian government–linked hackers, to conceal the fact that the emails had been leaked by a low-level DNC voter outreach data manager named Seth Rich, who’d been mysteriously murdered two months after the breach.
There was, as is often the case, a kernel of plausibility to one part of the theory. While no evidence surfaced to link Rich to the leaked emails, and Fox News had to pay a seven-figure settlement to Rich’s family for spreading the erroneous claim, CrowdStrike co-founder Dimitri Alperovitch would soon retire from the software industry to become a full-time professional neo–Cold Warrior, founding a think tank that predicted Russia’s invasion of Ukraine and writing a book about the coming world war over Taiwan. But cybersecurity tends to attract the geopolitically paranoid, and CrowdStrike, whose seed funding had come from private equity firm Warburg Pincus and Google, had just gone public on the NASDAQ stock exchange and its ownership was public: Its largest single shareholder was the money management empire BlackRock.
More notably, CrowdStrike had just been accused in court of orchestrating a conspiracy linked to the DNC cyber attack, though it didn’t involve any murders. According to the plaintiff, an independent cybersecurity software testing service called NSS Labs, the hack had exposed the incompetence of CrowdStrike, which the DNC had hired to stop the breach in May 2016 but which “missed a spot” that enabled hackers to hang out undetected in the servers for another five months afterward.
According to the lawsuit, in a pique of damage control, CrowdStrike allegedly colluded with some competing software developers and a nonprofit standards organization whose leadership they controlled to blackball a group of independent third-party software testing outfits that specialized in testing and identifying defects in so-called “endpoint protection” software, a variety of cybersecurity software CrowdStrike pioneered. The complaint says that, through the nonprofit Anti-Malware Testing Standards Organization (AMTSO), the companies had promulgated a bogus new set of narrow parameters by which third-party testing shops were allowed to test their products—essentially, a Mutual Enshittification Pact—and promised to boycott and/or sue any third-party testing shops that lobbied for more expansive or rigorous testing standards.
By all appearances, the scheme worked: By the end of 2020, the testing shop that brought the case had shut down, CrowdStrike was a $50 billion company—and nightmarishly disruptive ransomware attacks had become a near-daily occurrence.
“The fact is that there is very little empirical evidence that any endpoint detection software does anything approaching what the marketing claims that it does, much less prevent any of the catastrophic security breaches” that have become so increasingly commonplace, says an attorney and cybersecurity consultant who tweets under the moniker Brian in Pittsburgh and asked the Prospect not to use his full name for professional reasons. “Some large companies hire consultants to test-drive software packages against one another, but the results of those sorts of tests are almost never made public.”
On Wednesday, CrowdStrike posted a detailed technical explanation of the software update that had caused what some have described as the biggest IT outage in history.
The result, he says, is an information vacuum that combined with the software industry’s historical exemption from product liability laws and “enormous investor pressure to generate constant earnings growth,” inevitably resulting in the corrupted software update that canceled and delayed thousands of flights, surgeries, and electronic transactions last Friday.
On Wednesday morning, CrowdStrike posted a detailed technical explanation of the “content configuration update” that had caused what some have described as the biggest IT outage in recorded history. The company’s narrative blamed the meltdown on a bug in the “content validator for rapid response content” that “validated” the update in spite of the “problematic content” that caused the glitch. “In other words,” explained Brian in Pittsburgh, “they didn’t test the update. Instead they put it through the ‘content validator’ they designed for ‘rapid response content’ they don’t want to bother to have to test.” Shipping just one untested update to eight million computers, many of which power vital essential infrastructure, is “grossly unacceptable,” he said. “And the fact that no one’s shocked by it should be really shocking to us.”
But if the conspiracy NSS Labs described in its complaint holds even a kernel of truth, it sheds a lot of light on how CrowdStrike emerged in less than a decade as a company big and powerful enough to mint two multibillionaires and bring the world to its knees, despite a flagship product that almost no one understands, which demonstrably failed at its highest-profile assignment, and whose recent flub suggests practices so sloppy they call to mind much older, more corrupted enterprises like Boeing, or Abbott’s contaminated baby formula factory. Two cybersecurity experts told the Prospect that their industry was even worse. “This industry is pervaded by an incredible degree of secrecy and rot,” says Brian in Pittsburgh, “and that will persist until we create a neutral, adequately funded body … to investigate these disruptions.”
THE SEEDS OF CROWDSTRIKE, according to its co-founder, New Jersey–born Seton Hall University accounting major George Kurtz, were sown when Kurtz was the chief technology officer of McAfee, the ubiquitous—and notorious—manufacturer of antivirus software that’s often difficult to distinguish from a virus itself. (McAfee’s eponymous cokehead founder, now dead, is the subject of numerous of his own conspiracy theories.) Kurtz was on a plane watching another passenger start up his computer and wait—and wait—for the software to finish scanning his computer. As the man sat there patiently, the story goes, Kurtz died inside with each minute that passed, and vowed to come up with something more convenient: a cybersecurity app that would work not to filter out the ubiquitous but often benign scourge of malware but prevent the security breaches malware enabled, and would exist entirely on the cloud.
In April 2010, a botched software update eerily similar to Friday’s triggered a massive worldwide Windows glitch; within a few months, Kurtz was gone. He landed in a post at Warburg Pincus, where as a designated “entrepreneur in residence” he pitched the company on a “private cyber intelligence agency,” or “Digital Blackwater,” in the phrase of former CIA director Michael Hayden. Companies would hire Kurtz’s creation to stop security breaches, but also to preemptively set up decoys and traps in hopes of thwarting future malicious actors. Warburg put up $26 million to get the idea off the ground, and CrowdStrike was born.
But Digital Blackwater was just the first phase of Kurtz’s plan, as he explained in an interview earlier this year with the Logan Bartlett Show podcast. In 2016, CrowdStrike unveiled its flagship “endpoint protection platform,” or EPP, software Falcon Host, just in time for an explosion in a new strain of ransomware attacks, carried out by random freelance criminals outfitted with hacking software leased from a (typically Russian) “gang.” The company took off.
“What I knew from security was if you had the relationship, particularly on the services side, you’re gonna be able to sell the software. That isn’t always the case in other industries. But in security it is,” he explained. “Because it’s such a trusted relationship, when someone has a breach, it’s like going to your doctor, you have a problem that needs to be solved, and when somebody solves it you’re grateful … stop the bleeding [and] you become the hero.”
By 2018, CrowdStrike had exceeded $100 million in annual revenue; the following year, gross sales hit a quarter-billion dollars and it filed to go public on the NASDAQ stock exchange, where shares nearly doubled on their first day of trading. But CrowdStrike had a secret: According to documents filed in two lawsuits, Falcon had some serious flaws.
CrowdStrike had retained NSS in April 2016—the month before the DNC hired CrowdStrike—to submit the Falcon modules to a battery of private tests to determine their vulnerabilities. The results of those tests aren’t known, but one can probably extrapolate from what happened in 2017 after NSS informed CrowdStrike that its public group testing division—which claimed to aspire to become the Consumer Reports of security software, and was strictly forbidden from sharing software or data with its private testing division—had independently purchased some Falcon modules and submitted them to a battery of standardized tests alongside some competing products, the results of which it planned on releasing at the annual RSA security conference.
CrowdStrike immediately sued NSS, demanding a temporary restraining order to enjoin the lab from releasing the results. The company claimed that the test amounted to a theft of trade secrets, and that a public release would result in “irreparable harm” to its business. A federal judge disagreed and dismissed the TRO the day before the conference was scheduled to begin.
Ben Gray/AP Photo
Travelers wait in line at Hartsfield Jackson International Airport in Atlanta, July 19, 2024, as a major internet outage disrupted flights across the world.
But CrowdStrike had another trick up its sleeve, according to the antitrust complaint NSS would file the following year. At a trade organization conference in Poland the year of the DNC hack, an amended version of the complaint claims, CrowdStrike co-founder Dimitri Alperovitch hosted a meeting with fellow security software vendors “with the express intent, purpose and effect of obtaining agreement among the competitors to refuse to do business with companies [that] attempt to perform public tests of their products using testing methodologies other than those agreed to by the EPP Vendor Conspirators.”
Together, the companies formulated a new set of rules the testing agencies were required to follow if they wanted to test their software, including a minimum advance notice of five business days before tests commenced, a requirement that testing agencies allow software vendors to do certain tests over before the results were publicized so long as they insisted the malfunction was “anomalous,” and strict parameters on what kind of tests they were allowed to administer. CrowdStrike and its allies further agreed to “refuse to deal with any cybersecurity testing service that did not adhere” to their new “standards.” As a Symantec executive argued in an email to fellow AMTSO members urging them to vote in favor of the new standards, “If you want the money Symantec will pay for those tests, you will have to follow the standards. If a tester doesn’t like that, too bad. We will find one of their competitors who will.”
NSS and at least three other testing labs balked at the new provisions, which the complaint argues “defeat the purpose of independent third-party testing by giving the EPP Vendor Conspirators the ability to cheat the tests” to which they were being subjected. “Instead of using the Draft Standard to improve product offerings and protect the end user, vendors have repeatedly used it as a tool to demonstrate their dissatisfaction with tests where they have underperformed,” NSS wrote in a letter to AMTSO president Dennis Batchelder, arguing that the organization’s entire reason for existing was being turned on its head by the new rules. Another testing company wrote to AMTSO that unnamed software vendors had started routinely threatening to sue the testing lab to prevent it from releasing test results. “Is it ethical behavior to reply to a ‘test notification’ with legal threats, without trying to discuss a solution first?” that company asked. “Should AMTSO allow vendors who are members to threaten labs who are members with legal action to prevent them testing their products?”
The response from the AMTSO brass was swift and unambiguous. “You are prohibited from discussing the ‘status or substance of any ongoing or threatened litigation,’” AMTSO general counsel Jaimee King responded, copying all AMTSO members. When the testing company protested that its query had been general and unrelated to specific litigation, a board member and Symantec executive piped up: “The wording is ‘ongoing or threatened litigation.’” And in case that was too subtle a hint, AMTSO insisted on voting to enact the new rules in a public forum, over the objections of one software company that worried dissidents would be subject to the “fear of intimation” [sic] if the ballots were not kept secret.
While the CrowdStrike brass was working overtime to suppress unflattering test results, it was letting some serious bad guys slip through the cracks of clients’ firewalls, according to a report in The Daily Caller. Not only did it fail for months to catch one of the servers “Fancy Bear” was using to spy on the DNC—a tidbit originally published in Donna Brazile’s memoir, according to The Daily Beast—the company failed the following year to detect a security breach in the servers of its fellow client, the National Republican Congressional Committee, that led to the hacking of four senior officials’ emails. That breach was instead detected by a separate security consultant after the NRCC hired CrowdStrike. “I just hate CrowdStrike so much,” a former senior Democrat information security professional said at the time. “Their incompetence just makes me sigh.”
A CYBERSECURITY EXPERT WHO HAS WORKED with most of the players involved in the lawsuit and wished to remain anonymous says there was likely some validity to CrowdStrike’s assertion that NSS’s tests were flawed. “Falcon was, in those early days anyway, a best-in-class software,” he says, noting that CrowdStrike was the first and remains the only cybersecurity software developer to offer customers a “warranty tier” of service called Falcon Complete that reimburses clients for the costs of a disruption if a malicious actor manages to breach its firewalls. (Brian, the Pittsburgh consultant, dismisses Falcon Complete as “kind of a marketing ploy” that is “full of loopholes and fine print.”)
But there was general agreement that the scourge is systemic. “The problem is that cybersecurity software is a uniquely unaccountable sector of an unaccountable industry, and in an unaccountable industry there’s always going to be the inclination to allow standards to relax, and lean more on sales and marketing,” said the cybersecurity expert. Exacerbating matters, he continued, is that most of the testing labs, NSS included, had been founded to test antivirus software, which is a much more straightforward proposition than testing EPP software.
“In antivirus, you are shooting to block 100 percent of the viruses that are known to exist; in EPP, you’re trying to get 95 percent of those and hopefully another 95 percent of the threats you don’t know about yet but tend to conform to certain patterns, and you’re trying to do it with the fewest network resources possible. You can go about designing EPP software in an almost infinite number of ways, which makes it exponentially more challenging to iron out an objective methodology for testing one [EPP software] against another.”
“The problem is that cybersecurity software is a uniquely unaccountable sector of an unaccountable industry.”
The NSS lawsuit dropped with little fanfare in September 2018, a few months before CrowdStrike filed to go public. The following spring, CrowdStrike settled with NSS out of court, in an undisclosed arrangement that required NSS to retract its negative test results on the company, a few weeks before CrowdStrike had one of the year’s most successful IPOs, attaining a market cap on par with Symantec’s on its first trading day despite boasting just 5 percent of the software giant’s revenues.
But the Trump Justice Department breathed life into the complaint against AMTSO and its other conspirators a few weeks later, filing a “statement of interest” urging the judge not to dismiss the case. The judge didn’t have to respond: Apparently running out of funds, NSS sold itself to a small private equity firm that fall and dropped the suit against AMTSO in December. Meanwhile, the NRCC discovered its own emails had been hacked and called in an old foe to fix it: CrowdStrike.
In January 2020, AMTSO CEO Batchelder privately conceded that the NSS case had had some merit. “I do feel that vendors … that AMTSO helps vendors gang up,” he told members on a secretly recorded conference call transcribed in a court filing. “When you have these e-mails that go back and forth on AMTSO it allows the vendors to join in and express that, yes, I also agree or plus one or gang up or we all should work together. It actually can bring out the worst in the vendors because they don’t really have the ability to gang up privately without causing the kinds of problems.”
Alas, it was too late to save NSS, which abruptly shut down a few months after the call, unconvincingly blaming COVID-19. Calls to its former CEOs were not returned. CrowdStrike also did not return a request for comment.
MEANWHILE, CROWDSTRIKE’S MARKET VALUE grew almost as rapidly as the cyber attack scourge it had been founded to vanquish. During the 14 months after the start of the pandemic, Kurtz sold $250 million worth of shares and turned much of his attention to his second career racing exotic cars. Over the past five years, the CEO has sold stock 49 times, with his last sale on June 21 netting him more than $20 million. Chief Security Officer Shawn Henry, too, has made roughly $30 million selling more than 85,000 shares over the past year, the last 4,000 of which he disposed of in a planned transaction on Monday, July 15.
Four days later, just before 1:00 a.m. Eastern, the company sent out a faulty “sensor configuration update” to tens of thousands of clients that caused millions of computer screens to flash an error message apparently known as the “blue screen of death.” The buggy update grounded thousands of planes, suspended patient care at hundreds of hospitals and doctors’ offices, trapped millions of emails, and disrupted access to millions of consumer bank accounts. It wasn’t, Kurtz was quick to insist, the result of a cyberattack; the bug in the update was homemade, likely the result of shoddy testing protocols. An (ultimately vindicated) user on a subreddit for system administrators claimed CrowdStrike had been sending out updates without testing them for some time; another claimed CrowdStrike had laid off hundreds of engineers and quality inspectors the year earlier.
CrowdStrike, for its part, says it is still conducting “root cause analysis” of the fuckup and will report back soon; two House Republicans wrote a not-angry-just-disappointed letter to the company demanding Kurtz report back with his availability for the obligatory congressional flogging by Wednesday.
“The thing I found most absurd was the way they kept insisting it ‘wasn’t a hack.’ As though that would matter to anyone who had to postpone emergency surgery or who couldn’t access their bank account because of what may be the biggest IT disruption in recorded history,” mused the cybersecurity expert.
“These software companies ascend and once they’re ascendant, they spend more money on race cars and action figures and advertisements at airports as opposed to the people and processes that are going to keep those airports operating smoothly and securely …” he paused. “It’s like the fall of Rome, every time.”
