Twitter's Sensitive Content Algorithm

Is it too sensitive?

This post was originally published on August 20, 2017.

In March, Twitter’s sensitive content algorithm began its testing phase, a feature that warns users of “potentially sensitive content,” according to Sarah Perez at TechCrunch.

This was the first time I have personally seen such a warning and the algorithm appears to have some failings.

I noticed that a Twitter account called “Raging Bull NO TRUMP,” or @ToDropADime2 had begun following me. When I went to investigate the account by clicking, the account’s profile gave me this warning:

I was curious as to why an account that appeared to be anti-Trump would be blocked by another account that likely shares its attitude toward the president. Back in March, TechCrunch reported:

Twitter tells us the new feature works similarly to how other sensitive content on Twitter gets flagged, based on users’ settings.

Currently, the company permits content that contains violence or nudity, but it draws the line at “pornography or excessive violence in live video, or in your profile image or header image,” according to its page on sensitive media. It doesn’t mention profanity, racism, bigotry and other types of offenses, however. But sensitive content is not limited to “violence or nudity,” we’re told.

Users can choose to mark themselves as someone who tweets sensitive content through their “Privacy and Safety” settings.

I checked my user settings:

As you can see, I have the Hide Sensitive Content setting checked because I don’t want to see porn or gratuitous violence on Twitter. Note, however, that the setting claims to prevent potentially sensitive content from displaying in search results.

I did not find this account through search results; I found it within my notifications because it had followed me.

I went ahead and clicked Yes, View Profile to investigate why it had been blocked from me. Here’s what the Raging Bull Twitter profile looks like:

While I can see why fans of the president wouldn’t like this account, I am not one of those people.

I scrolled through the Raging Bull feed and found what you might expect to find from a political activist opposing the president. But aside from prolific use of  the F-word, nothing I saw appeared to be “sensitive content” to me.

Even after I clicked through Twitter’s warning, I noticed individual tweets with photos or video appeared blocked as well. Here’s an example:

I went back to my settings to allow sensitive media to be shown. Here’s what the same tweet looks like with the GIF displayed:

Hardly offensive, right?

Is Twitter’s Sensitive Content Algorithm Too Sensitive?

It appears the only “sensitive” content Raging Bull posts is the frequent use of the F-word.

There are two explanations for the warning:

1. Swear words trigger it, or

2. Getting reported or blocked by other users triggers it.

If it is only swear words that trigger the warning, that’s an awfully low bar in my book. It seems to me it would be trivial for Twitter to offer a text box setting into which those delicate souls who cant abide by an errant curse can indicate to the service which offending words they wish not to see.

The last explanation is pure speculation on my part. I seriously doubt the Raging Bull account has checked its own settings to declare it tweets potentially sensitive material, so that leaves the likelihood that Trump fans are complaining.

TechCrunch continues:

Twitter users can report tweets to the Twitter team for review. In this case, if the tweet is determined to be potentially sensitive, Twitter will label the content appropriately — or remove it, if it’s a live video. It may also adjust your account setting for you, so your future tweets are marked accordingly, if it deems it necessary.

For repeat violations, Twitter may permanently adjust that setting on your behalf, it says.

The process for marking entire profiles as sensitive follows a similar set of guidelines and processes, including the fact that Twitter can take an active role in identifying these accounts, based on the content of the account’s tweets.

These are the options for reporting a Twitter account:

I imagine Twitter has an algorithm that flags accounts that reach a certain threshold of complaints or of getting blocked.

But that could easily be gamed, with supporters of Trump reporting or blocking this account en mass.

Is Twitter’s Sensitive Content Algorithm Unsophisticated?

Yes, methinks.

Based on what Twitter knows about me (or should know), the platform would have no reason to warn me about the Raging Bull account:

• I frequently post articles that are critical of the president,

• I frequently retweet others’ posts that are critical of Trump,

• I follow many accounts that are critical of Trump, and

• The media sources I share are likely the polar opposite of those of a Trump supporter.

The only real signal that might throw Twitter’s algorithms for a loop is the fact that I follow many conservatives from my own state and many of them follow me, as well.

Nevertheless, it seems to me Twitter has ignored many of the signals it should be paying attention to when deciding whether or not to warn me about an account like Raging Bull’s.

From that, I would conclude that Twitter is either not collecting the data it should be collecting to understand its users, its algorithms are not very sophisticated, or both.