Twitter Triggers

The push and pull between social media and our moral emotions

A version of this article has been published by Nautilus

In December 2013, Justine Sacco, then the senior director of corporate communications at IAC, sent out a string of tweets over the course of a long, multi-leg journey from New York to South Africa. She started with sardonic observations about the minor trials of travel—one about a smelly passenger at JFK Airport, another about London’s peculiar food and predictably inclement weather. Then came this one, which she sent shortly before boarding her plane for the final, 11-hour flight to Cape Town:

“Going to Africa. Hope I don’t get AIDS. Just kidding. I’m white!”

As she turned off her phone and settled in to get some sleep, she had good reason to expect that that tweet, like the ones before it, would simply fade away into the hectic, ever-updating ether of Twitter. She had only 170 followers, after all.

But it did the exact opposite. Over the next 11 hours, Sacco became the number one worldwide trending topic on Twitter, as tens of thousands of users across the globe filled her feed with their outrage. When she turned on her phone after landing in Cape Town, she found herself receiving the full brunt of the online community’s capacity for public shaming.

Sacco’s public persona was destroyed. She was fired from her job and saw much of her social circle—both online and offline—wither away.

I thought there was no way that anyone could possibly think it was literal,” she told journalist Jon Ronson a few weeks after the incident. “To put it simply, I wasn’t trying to raise awareness of AIDS or piss off the world or ruin my life. Living in America puts us in a bit of a bubble when it comes to what is going on in the third world. I was making fun of that bubble.”

Regardless of whether one accepts that defense, Sacco’s case is yet another reminder that nuance is often one of the first victims of viral outrage. Her tweet is just one of countless examples of provocative online behavior being stripped from its context and drawing a seemingly disproportionate social punishment.

To understand how online outrage can so easily bloat to a farcical scale, it helps to zoom out and examine how the architecture of social media exploits our sense of right and wrong.

Pictured: Twitter

Attention is often conceptualized as a commodity. The idea has been around for decades, but it’s been drawing more and more support since the advent of smartphones, which have driven the market competition for your eyeballs to new heights. The algorithms that undergird the flow of information on social media are, like the sensationalist print media and incendiary talk radio that came before them, designed to maximize ad revenue by engaging consumers’ attention to the fullest extent possible. Or as novelist John Green puts it, “Twitter is not designed to make you happier or better informed. It’s designed to keep you on Twitter.”

Columbia Law professor Tim Wu calls this “attention harvesting.” And as a business model, it’s extremely lucrative.

Now, the concept of attention harvesting may not come as a surprise. Most of us are aware on some level that those persistent, weirdly personal ads on our computers and mobile devices have a lot to do with how internet giants like Twitter, Facebook, and Google generate their massive revenue streams. What we may not be aware of, though, is exactly how those platforms manage to hold our attention well enough to make their ads so profitable.

Scientists have been at this question for several years, studying people’s activity online and revealing interesting trends as to what makes content eye-catching and more likely to go viral. Jonah Berger and Katherine Milkman from UPenn, for example, analyzed 7,000 articles from the New York Times and found that one of the main determinants of whether readers shared a story via email was whether it was emotionally arousing. In other words, what drove a story’s popularity online wasn’t how surprising or useful it was, but rather how well it stirred people up.

Billy Brady from NYU built on Berger and Milkman’s work by analyzing hundreds of thousands of tweets in an effort to understand the role of moral emotions in social networks. Moral emotions are the feelings associated with our sense of right and wrong, like pride, embarrassment, and outrage. Brady and his colleagues found that tweets about political topics were much more likely to go viral if they contained words that are both morally and emotionally charged, like “evil,” “shame,” “fight,” “punish,” and “faith.”

What’s more, Brady’s analysis revealed that viral political tweets propagated almost exclusively among people with similar ideological leanings, which adds to the pile of evidence that we spend much of our online lives inside echo chambers.

The constancy with which moral-emotional content spreads online has turned much of Twitter—and, by the looks of it, other platforms—into what writer Samuel Ashworth described as “an endlessly self-renewing bonfire of outrage and confusion,” in which preachers whip their respective choirs into frenzies of righteous fury all day every day, and in which a hastily written tweet to 170 followers can become a global scandal. And given how profitable it is, social media companies have little financial incentive to scale it back.

“Morality is something that we see as central to being human,” said Molly Crockett, a psychology professor at Yale, on the podcast Very Bad Wizards. “And I think it’s really important to ask ourselves and to have a serious conversation about how we feel about our moral emotions being used to make a lot of money for tech companies.”

In a recent article in Nature Human Behavior, Crockett argued that the constant triggering of moral outrage on social media not only makes money for tech companies, but also alters how we experience and express moral outrage. Provocative content is ubiquitous, the tools to socially react to it have never been more accessible, and outrage expression often garners positive reinforcement in the form of likes, retweets, and shares. All of this together, Crockett wrote, may cause people to undergo “outrage fatigue,” whereby the intensity of the outrage they feel gradually fades. Or, in light of research showing that venting anger can stoke more anger, it may do the opposite, amplifying successive expressions of outrage.

One final, particularly troubling possibility Crockett mentioned is that social media may uncouple the expression and experience of moral outrage. “And just as a habitual snacker eats without feeling hungry,” she wrote, “a habitual online shamer might express outrage without actually feeling outraged.”

Pet shaming: an ethical (and adorable) alternative to Twitter sanctimony

Though there remains much to learn about the interplay between social media and moral emotions, the existing research points to a broader issue: social media companies either ignoring or downplaying the immense influence their products can exert on users’ emotions and on public discourse in general. Perhaps the most glaring example is from November 2016, when Mark Zuckerberg confidently dismissed the idea that Russian hackers exploited his platform to influence the outcome of the presidential election as “pretty crazy.”

The Facebook CEO has since come around on that issue, and now he, his competitors, and all who use their products face a difficult question: To what extent do social media companies have a moral obligation to improve the way we communicate with each other?

“I think most people assume…that if democracy is going to be healthy, we need civil discourse to be healthy also.” said Brady. “And so if there is data that continues to come in that shows that social media is amplifying (the expression of negative moral emotions), then I think they do have a moral obligation.”

Many social media companies now appear to agree with Brady and are making efforts to address some of the concerns that research like his raises. Twitter, for example, recently announced an open call for proposals on how to improve “conversational health” on its platform. And last October, Reddit rolled out a more robust policy for actively monitoring its discussion boards.

But some efforts to improve online communication have been flops, like Facebook’s short-lived feature that allowed users to flag fake news. And in view of the recent Cambridge Analytica scandal, which has exposed serious flaws in Facebook’s privacy policy, many are wondering whether social media companies can even be trusted to monitor themselves, let alone global public discourse.

There are no easy solutions to any of these issues. And the longer they go unsolved, the longer our moral emotions will be subject to monetized technological forces that nobody fully understands. And frankly, that’s an outrage.

3 Replies to “Twitter Triggers”

  1. Hi Scott,

    Sorry for the delay in getting my opinion on this to you. However, it just so happens that the delay and events since then have made me sort of shift my attitude on this.

    First, this: Or as novelist John Green puts it, “Twitter is not designed to make you happier or better informed. It’s designed to keep you on Twitter.” Great quote.

    Second and primarily, the part of your article that I wanted to respond to was regarding the moral obligation of social media companies to improve the way we communicate with each other. The reaction that I told you I would have is I think they have basically zero obligation to do that. My rationale for this is that there are other controlling forces in an individual’s life that have a significantly greater responsibility in teaching someone how to communicate more effectively, and that putting any of that responsibility on a website/social media network is asking that website to step so far outside the bounds of its general purpose. It reminds me of the logic behind someone eating a diet of unhealthy food and then wanting to sue a fast food place for making such unhealthy food. Its not McDonald’s job to teach you not to eat french fries everyday. Ergo, it’s also not Twitter’s job to make you learn how to communicate without being an asshole. I think the obvious critique of my argument is that Twitter/Facebook/Insta etc. are becoming so overwhelmingly popular that they are rising to be the essential medium of our time whereas fast food is not so pervasive in everyone’s lives. It’s a fair critique too, the bigger and ever-present/necessary a business becomes, I can see why people want to assign it greater responsibility. I just don’t like to place many restrictions on businesses when it comes to their ability to conduct themselves how they see fit and to make money any way they can (within the bounds of legal limits).

    Having said all of that, I have to admit that I’ve enjoyed something a great deal recently that involves a website doing that. I recently signed up for The Athletic (really enjoying it so far, I’d advocate checking it out). And although I’ve not read anywhere that the website is for sure doing this, I suspect that they are curating their public comments to make all of the discourse generally be about the topic and removing personal attacks, inane comments, etc. I know they are not the first website to do this, just the first one I have great exposure to. I have to say, it makes reading the comments x1000 better. The civil discussion and different opinions on a highly trafficked article is like a whole extra article worth of content.

    In conclusion, I think I would have to make my official position that I think these websites have no moral “obligation” to help people communicate more amicably, I am all for it if they independently choose to do so.


    1. Thanks for sharing your thoughts.

      I’m very much of the opinion that you mentioned in your comment: that these tech companies have grown to control such a massive share of our communication that it’s no longer in our best interest to treat them like just another cog in our economic system. I share your belief that we should think very carefully before infringing on a company’s right to seek profit how they please within the confines of the existing law, but I think tech companies’ downsides are significant and well-backed by empirical research to an extent that warrants oversight.

      I’ve heard Facebook and the like compared to big tobacco. Maybe the analogy is a bit dramatic, but in many ways I think it’s appropriate. Setting aside for a moment all the concern about data privacy and public discourse, it’s well-documented that using Facebook is generally pretty bad for your mental health. If a) a product is damaging to your health, b) the company behind it doesn’t really acknowledge that fact, and c) unlike in fast food, using that product is almost a necessity in public life, then I think oversight becomes necessary. When you throw in all the other negatives, like how FB fuels ideological polarization (including, according to the UN, aggravating the ongoing Rohingya genocide) and has shady privacy policies, the argument for oversight gets even stronger.

      I don’t know how the “oversight” should look. It could be helpful to regulate certain tech companies like public utilities (an idea also supported by Steve Bannon, so…take that?). Or maybe we see how tech companies’ emerging self-policing efforts turn out and then decide whether they’re adequate. But I think leaving the companies entirely to their own devices (and us consumers to our own, pun extremely intended) with little-to-no accountability is about the worst we could do.


  2. Quick thoughts:
    – Excellent pun.
    – The shady privacy policies bit is more convincing to me. I think when you sign up for one of those sites you should expect a certain amount of privacy; certainly, you should get at least the amount of privacy that the site represents you will have.
    – I don’t agree with you that these sites are “almost a necessity in public life”. I think having an email address is probably a necessity, but I don’t know that anything else on the internet is a necessity. With that in mind, I do not rule out my fast food comparison. I think the tobacco comp is also not bad though, I don’t believe that the damage to your health is on the same scale as smoking cigs though.
    – I wonder if the oversight for social media sites should just be the same as cigarettes? In order to log on you see: “Warning: Looking at Social Media websites for more than one half hour per day can lead to depression.” I don’t know if I’m kidding?


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: