AP Photo/Matt Rourke, File
Zucked: Waking Up to Facebook Catastrophe
By Roger McNamee
Penguin
Custodians of the Internet: Platforms, Content Moderation, and the Hidden Decisions That Shape Social Media
By Tarleton Gillespie
Yale University Press
The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power
By Shoshana Zuboff
Public Affairs
This article appears under the title "The Trouble with Tech" in the Spring 2019 issue of The American Prospect magazine. Subscribe here.
Not long after whistleblower Edward Snowden's first revelations appeared in The Guardian and Washington Post in the summer of 2013, longtime digital-freedom activist and writer Cory Doctorow declared that we had finally passed the point of “peak indifference to surveillance.” That fall, upset at rising rents and declining public services, people in San Francisco and Oakland starting blockading and sometimes throwing rocks at the private buses taking privileged tech workers to their Silicon Valley jobs. But the backlash to Big Tech and its massive power didn’t really gather force until the news broke in the spring of 2018 about Cambridge Analytica’s stealthy acquisition of the personal data of 87 million Facebook users.
Now America's love affair with Big Tech is clearly over, and we're finally starting to re-evaluate the terms of the relationship. But it remains to be seen whether we will merely patch things up or try to start over on completely new terms. This is not a small question.
At stake is whether we continue to let giant tech companies, along with many other Fortune 500 firms, function as the de facto owners and organizers of the 21st century's most valuable underdeveloped resource—our personal information and the connections we make with it
—or if we act to reassert control over what should rightfully be ours. To get to that answer, we need to look afresh at what today's data barons actually do.
The reach of Big Tech and concerns about its power stretch far beyond America's borders. The industry commands territories bigger than the population of nation-states. Facebookistan has 2.3 billion users. Googlestan has more than a billion users of each of seven services (Gmail, Android, Chrome, Maps, Search, YouTube, and the Google Play Store). Microsoftistan boasts 700 million running Windows 10. And other platforms are also extending their reach: Amazon, which is the starting point for more than half of all online shopping searches in the United States, is now partnering with carmakers to install Alexa in their dashboards; Pokémon Go, a Google offshoot, has demonstrated that online gaming can move crowds in real life to “sponsored locations”; and iRobot has upgraded its Roomba robot cleaners with mapping capabilities in order to vacuum up their owners' home floor plans.
As the tech platforms expand into the far corners of the world and every aspect of our lives, public policy and public understanding have trouble keeping up, but several new books provide important insights. In the delightfully titled Zucked, the venture capitalist Roger McNamee recounts his role in the early development of Facebook and his disillusionment with the company. Tarleton Gillespie’s Custodians of the Internet takes us inside the decision-making at tech firms about what communications they allow and what they block. And Shoshana Zuboff’s The Age of Surveillance Capitalism—the most important book I have ever read on the intersection of technology, politics, and society—offers a radical reinterpretation of the changes the tech industry has wrought.
As a recently woke Silicon Valley capitalist, McNamee has become a traitor to his class, or at least to the young man he once mentored, Mark Zuckerberg, who, according to McNamee, has a lot to thank him for. Years before Facebook went public, McNamee urged Zuckerberg not to sell it to Yahoo for a reported $1 billion, and then later suggested he hire Google ad whiz Sheryl Sandberg as his chief operating officer to help him figure out how to make money. But while the enterprise certainly did well, it did not do the good that McNamee expected. “It turns out that connecting 2.2 billion people on a single network does not naturally produce happiness for all.” The environment that Facebook created has encouraged people to form “into clusters or tribes,” which has led to polarization in the United States and mob violence, even genocide, in other societies.
In early 2016, McNamee grew concerned about the spread of divisive political content on Facebook driving wedges between Hillary Clinton's and Bernie Sanders's supporters, and then watched the unexpected success of the “Leave” campaign in the Brexit vote. Convinced that Facebook's News Feed algorithm was amplifying fear and anger and that bad actors were gaming the system, McNamee tried to sound an alarm with his protégé. But he was fobbed off on one of Zuck's lieutenants, who treated his warnings as a “public relations problem.” So he decided to go public with his criticisms, teaming up with former Google ethicist Tristan Harris to create a new organization, the Center for Humane Technology, and to educate lawmakers on Capitol Hill.
As McNamee writes, Facebook is, among other things, “a massive artificial intelligence that influences every aspect of user activity, whether political or otherwise. Even the smallest decisions at Facebook reverberate through the public square.” One man stands at the center of those reverberating decisions, but how does that decision-making process work, and what rules does it follow?
The rule-making at Facebook and other platforms about allowable content is the subject of Gillespie's Custodians of the Internet. Gillespie’s custodians are the moderators at social networks, search engines, blogging services, photo- and video-sharing sites, opinion hubs, dating apps, collaborative knowledge tools, recommendation and ratings sites, goods exchanges, fundraising platforms, and video-gaming worlds—in other words, the platforms that consume much of our collective time online. “All platforms moderate,” Gillespie writes, even though they downplay how much they interfere. The process of moderation, he shows, is “a prism for understanding what platforms are, and the ways they subtly torque public life.”
As Gillespie notes, the hyperscale of today's online platforms makes individualized content moderation an impossible task, and even with a combination of automated detection using artificial intelligence, community flagging of objectionable content, and editorial review, plenty of objectionable stuff makes it online. To take a relatively benign example, the best “nudity detection algorithm,” he observes, claims to spot nude pictures with a 94 percent accuracy rate and a false-positive rate of just 5 percent. With more than a billion active monthly users of YouTube and 350 million photos uploaded every day to Facebook alone, that means plenty of naked pictures make it onto these sites. “Platforms dream of electric shepherds,” Gillespie wryly notes. “But it is not clear that the human labor necessary to support such automated tools can ever go away.”
The result, which he explicates with great precision throughout the book, is a new industry of content moderation, with a small elite of policymakers atop the major platforms writing the rules for a burgeoning and largely invisible archipelago of digital sweatshops. That's where hundreds of thousands of so-called “janitors”—click-workers making a few dollars an hour in Third World countries, or little more than poverty wages here in America—spend their days dealing with a never-ending flow of traumatizing content.
Moderators inevitably make contestable judgments. Here, for example, are some of the contradictions in Facebook's policies that Gillespie cites:
Hate speech is prohibited, but only for protected categories of people (specific races, religions, sexualities, nationalities), and “migrants” constitute only a quasi-protected category, so although dehumanizing statements about them should be removed, cursing at them, calling them thieves, and urging them to leave the country do not amount to hate speech. Holocaust denial is allowed, but geo-blocked from countries that outlaw it—not all fourteen countries with laws prohibiting Holocaust denial, just the four that have pursued the issue with Facebook explicitly.
The custodians of online platforms face an inescapable dilemma: If they don't curate content enough, they risk losing users. And if they curate too much, they risk stifling their own growth and losing favor on Wall Street. Curating is not a role that Big Tech's leaders wanted, as Gillespie points out. They promised an open, participatory culture, but as the promises have soured, the companies are “grappling with how best to be stewards of public culture, a responsibility that was not evident to them at the start.”
Like McNamee, Gillespie frames the trouble with tech as a paradise lost, as though platform makers were just innocent victims of their own success. Who could have known how hard it would be to police sites with billions of users? Who could have imagined that fine-tuning the user experience to show people content they find most engaging would lead to filter bubbles, increased societal strife, and the spread of conspiracy theories? This narrative is overly friendly to Big Tech as it faces its moment of democratic reckoning. Furthermore, it misses the central truths about the forces we are dealing with. To understand those forces, we must turn to Zuboff's The Age of Surveillance Capitalism.
For Zuboff, companies like Facebook and Google are not friendly providers of free services such as search engines or social connectivity that just happen to be struggling with the unintended consequences of massive growth. They are the 21st century's robber barons. Only instead of plundering the natural environment, they are plundering us.
More or less by accident, they discovered a source of fabulous riches and now, by design, they have created a whole new kind of capitalism, rooted in their ability to track, digest, and with increasing precision, predict and even behaviorally modify everything that we do
.
Our posts, blogs, tweets, and other actions online are what Zuboff calls “the first text.” That first text provides us with unprecedented opportunities for connection and knowledge. But that first text also supplies the basis for a second, shadow text. “Everything that we contribute to the first text, no matter how trivial or fleeting, becomes a target for surplus extraction.”
Once you understand the titans of tech as surveillance capitalists bent on capturing as much unguarded human experience as possible, many of their previously curious business decisions become more legible. Google's 2006 purchase of YouTube for $1.65 billion before it had any revenue, its massive expenditures mapping the world for Google Street View, Facebook's $1 billion purchase of Instagram and $19 billion purchase of WhatsApp, Verizon's $4.4 billion purchase of AOL, Microsoft's pivot to providing free cloud services rather than selling licensed software—all of these choices and others too numerous to mention make sense when you think of them not as risky speculation but as the shrewd colonization of raw materials and the protection of supply lines. Maps create empire, Zuboff reminds us, citing the cartographer John B. Harley.
And the plunder is intensifying, driven by an economic logic now understood by many industries if not the general public. As a senior engineer at a major tech company heavily investing in the so-called Internet of Things told Zuboff:
“Imagine you have a hammer. That's machine learning. It helped you climb a grueling mountain to reach the summit. That's machine learning's dominance of online data. On the mountaintop you find a vast pile of nails, cheaper than anything previously imaginable. That's the new smart sensor tech. An unbroken vista of virgin board stretches before you as far as you can see. That's the whole dumb world. Then you learn that any time you plant a nail in a board with your machine learning hammer, you can extract value from that formerly dumb plank. That's data monetization. What do you do? You start hammering like crazy and you never stop, unless somebody makes you stop. But there is nobody up here to make us stop. This is why the ‘internet of everything' is inevitable.”
Surveillance capitalism, Zuboff argues, took off when the founders of Google, Larry Page and Sergey Brin, faced an existential crisis as the first dot-com boom imploded in early 2000. As graduate students who discovered the value of using web linking patterns to deliver authoritative answers to search questions, they had built a dominant search engine. But they didn't know how to monetize all the traffic Google was drawing, and they were philosophically critical of advertising and its corrupting effects. “But as the evidence mounted that ads could save the company from crisis,” Zuboff writes, “their attitudes shifted. Saving the company also meant saving themselves from being just another couple of very smart guys who couldn't figure out how to make real money, insignificant players in the intensely material and competitive culture of Silicon Valley.”
They were also under immense pressure from their venture capitalist backers, one of whom, Michael Moritz of Sequoia Capital, later revealed that Google's first source of income, selling licenses to the search engine to larger companies, wasn't going all that well. “Cash was going out of the window at a feral rate,” Zuboff quotes him as saying. Once Page and Brin agreed to start mining user data to better match ads to their interests, the company's fortune was made.
As Google learned how much value it stood to unlock from user data, it advanced on two fronts simultaneously, moving from crawling the web to crawling the physical world, while carefully deflecting attention from its real agenda and asserting that the death of privacy was a technological inevitability. Thus, as Zuboff notes, Google chairman Eric Schmidt, upon being asked on CNBC in 2009 why Google retains our search histories indefinitely, replied, “The reality is that search engines including Google do retain this information for some time,” as if this was an engineering necessity. “In truth,” she writes, “search engines do not retain, but surveillance capitalism does. Schmidt's statement is a classic of misdirection.” Had we been able to see these encroachments not as privacy violations alone but as acts of digital dispossession, perhaps the rise of surveillance capitalism would have faced some real roadblocks.
Instead, its growth was greased by Democratic politicians like Barack Obama, who turned his administration into a Google annex and who, Zuboff writes, “used his proximity to Schmidt to cement his own identity as the innovation candidate poised to disrupt business as usual in Washington.” Senator Ron Wyden was primarily responsible for writing Section 230 of the 1996 Telecommunications Act, a safe-harbor provision that was designed to encourage early platforms such as CompuServe and Prodigy to take active steps to moderate content without holding them legally liable for all that they enabled users to publish. “What Wyden and his colleagues could not have anticipated, and still do not grasp,” Zuboff charges, “is that the logic [of the early internet service provider platforms] no longer holds. … They no longer merely host content but aggressively, secretly, and unilaterally extract value from that content.”
We may individually want to escape this new enclosure movement, but Zuboff is right.
We are caught in a Faustian bargain. Being connected is essential for participating in life, but the internet has been commandeered by commerce, and “commerce is now subordinated to surveillance capitalism.”
The solutions on offer, she adds, are not anywhere close to addressing the problem at hand. Calls to break up Google or Facebook might simply establish more surveillance capitalist firms, and “demanding privacy from surveillance capitalists or lobbying for an end to commercial surveillance on the internet is like asking Henry Ford to make each Model T by hand.” Europe's new General Data Protection Regulation offers hope for a different future, but Zuboff's eagle eye catches Facebook's Zuckerberg characteristically promising to operate according to GDPR's spirit while quietly moving 1.5 billion Facebook users who used to be clients of its Ireland subsidiary back to the mothership in the United States, in order to avoid the law's reach.
In the last third of her book, Zuboff wrestles with the philosophical implications of a society run by distant technocrats with “God-like” knowledge of humanity's desires and actions, delving into the intellectual roots of such thinking in the works of Harvard behaviorist B.F. Skinner. She theorizes that surveillance capitalists are not digital totalitarians but a new form of overlord, “instrumentarians” indifferent to the day-to-day effects of their platforms as long as the raw materials flow into their analytical maws unmolested. This section could have been a whole second book, and readers may find some of her argument repetitive. But these are quibbles against a masterwork of synthesis.
The Age of Surveillance Capitalism is not an obituary for democracy. Zuboff has provided the latticework on which to hang useful new initiatives. “We have yet to invent the politics and new forms of collaborative action … that effectively assert the people’s right to a human future,” she writes. But now that she has named the system, described it, analyzed it, and helped us understand it, we may begin to change it.
