Insults, personal and identity attacks — Content Moderation

Page contents

1. Definitions: Insults and hate speech in user content
2. What's at stake
3. How to detect insults and hate speech
4. What actions should be taken with insulting and hatred content

Definitions: Insults and hate speech in user content

Insulting or hate speech is a very broad topic as it includes various situations like:

• insults directed towards individuals and personal attacks, i.e. a form of speech deliberately intended to be rude and to attack a specific person
• insults directed towards public entities like a company or a public figure
• hate speech and identity attacks, i.e. speech that attacks or disparages a person or group of persons on the basis of protected characteristics such as sexual orientation / gender, origin / ethnicity / nationality, religion, skin color, disability but also physical appearance, social environment, language or age

Online attacks could be considered as violence or abuse. In its Hate Crime Report of 2021, Galop (a UK’s LGBT+ anti-abuse charity) indicated that among 523 people who have experienced abuse because of their gender or sexual orientation, 60% say that the abuse occurred online.

Type of anti-LGBT+ violence or abuse experienced by 523 people (Galop Hate Crime Report 2021)

What's at stake

Defining this category is a challenge as it is not easy to define its boundaries: what should be considered as a legitimate opinion? What could be the subject of a debate between users? The answers to these questions really depend on the platform you want to build.

Because public entities such as politicians or companies often lead to heated debate between users and because freedom of expression generally prevails on big platforms and applications, social media sites typically allow content that criticizes businesses and politicians as long as it complies with their guidelines about hate speech, harassment, and other improper content.

More generally, below are some key points on why it is important to address the issue of personal and identity attacks.

Content inciting hatred or calling for violence against people or protected groups are often based on any of the following attributes: age, caste, disability, ethnicity, gender identity and expression, nationality, race, immigration status, religion, sex, sexual orientation, victims of a major violent event and their kin or veteran status.

Most user-generated texts that are considered hate speech express hateful ideas without naming a specific person and this is also often the case for insulting messages. But as Meta indicated in a 2017 post, such content could "feel very personal and, depending on someone's experiences, could even feel dangerous". This kind of comment therefore shouldn't be considered as a simple opinion.

There is mostly no legal obligation to detect insults and attacks but in some countries in the EU for example, it is illegal to encourage and promote hate. In addition, users of a platform or an application probably wouldn’t tolerate such content and might decide to leave the platform if no action is taken to handle the situation.

Finally, detecting insults and hate speech is one of the most common tasks in the text moderation field. There are more and more papers on this topic and especially on hate speech detection. This confirms that this topic should be detected as it is a very common issue for all platforms and applications.

What actions should be taken with insulting and hatred content

Detected insulting or hatred messages should be appropriately treated. When detecting such content, platforms and applications should first do a triage to make the difference between the possible targets of the message.

Attacks directed towards individuals and protected groups

When a hateful or insulting text is detected, platforms can use more or less severe sanctions depending on the case. This can range from simply removing the insulting text to suspending the account of the author of the text from the platform, for a limited time or permanently. For some extreme cases where insults or hate speech escalate to an actual threat, platforms could report them to the relevant local authorities. In its hateful conduct policy, Twitter also suggests the downranking and reduced visibility of the tweets violating the policy.

Several criteria should be taken into account when deciding on the sanction:

• Is the user already known for posting insulting and / or hateful content?

To answer this question, it would be interesting to set up a moderation log to allow all moderators to report any incidents encountered for a specific user. It would help to identify users who have previously used hateful language or insults.

• Is the content clearly expressing a bad intention?

If the content is ambiguous (for example if it has two meanings), it is more complicated to penalize a user.

• Is it explicitly directed against a person?

Insulting or hateful content directed against a named individual may be considered a direct threat.

• Is the level of toxicity of the text very high?

For example, wishing for the death of a community is probably more serious than making a bad / poor joke about that same community.

Attacks directed towards public figures or companies

Platforms detecting content criticizing public entities such as politicians or companies should make sure the content is respecting the community guidelines. Here are some examples of contents that are typically allowed:

• Opinions and comments on the actions or policies of a politician or corporation, or on any other public figure, as long as they are respectful and do not involve threats or personal attacks.
• News articles or reports that provide critical analysis or commentary on a politician's or company's actions or policies, or on any other public figure, as long as they are factual and not defamatory.
• Satirical content that pokes fun at a politician's or company's actions or policies, or at any other public figure, as long as it does not contain hate speech or other forms of inappropriate content.
• Memes or other forms of humorous content that criticize a politician's or company's actions or policies, or any other public figure, as long as they are not defamatory or do not contain hate speech.
• Protests and actions organized against a politician or a company, or against any other public figure, as long as they are not coordinate attacks to boycott or call for violence, whether the public entity belongs to a specific protected group or not.

Detected threats, insults, hate speech, defamatory content against public entities should be treated as personal attacks and attacks towards protected groups.

More generally, when building the rules of an online community, it is important to encourage positive behavior from users. In their advices to build a safe community, Discord says that setting an example as a community leader or practicing de-escalation by offering a second chance to a user who causes trouble are also solutions.

Read more

Illegal traffic and trade — Content Moderation

This is a guide to detecting, moderating and handling illegal traffic and trade in texts and images.

Joining the Online Dating & Discovery Association: Towards Safer Connections

This is a step to further enhance end-user safety in the online dating realm.

Head back to the Knowledge Center