Online promotion, spam and fraud — Content Moderation

Definitions: Online promotion, spam and fraud in user content

Spam refers to "unsolicited usually commercial messages (such as emails, text messages, or Internet postings) sent to a large number of recipients or posted in a large number of places" with the intent to further promote their misleading contents.

In a survey from Orbit Media, QuestionPro and Foundation asking more than 1000 people about spam, 35% indicated seeing spam comments every day on social media (see graphic below).

For platforms and applications, it mainly corresponds to unwanted advertising or promotion, but also problematic behaviors such as:

• personal promotion, i.e. comments from users promoting their personal work, website or content, but also promoting false information and services, and asking for likes, shares, follows or clicks
• company promotion, i.e. comments from companies and brands promoting their products and solutions, and asking for likes, shares, follows or clicks
• sexual promotion, i.e. comments promoting sexual services (including false services), websites or content
• scams or fraud and phishing attempts, i.e. comments from fraudsters that can compromise your financials or personal details if you get tricked
• and more specifically catfishing, i.e. a type of phishing where a user creates a fake profile and identity, generally on a dating or social networking website, with the goal of seducing a victim into a sexual or emotional relationship to extort money or personal data in the end

Moreover, these behaviors occur through a platform circumvention method: users are led to leave the platform to go to another one.

How to detect online promotion, spam and fraud

Apps and platforms generally resort to four levels of detection measures to detect online promotion and spam, as well as fraud attempts:

• user reporting
• human moderators
• keyword-based filters
• ML models

User reporting

Platforms should consider relying on the help that other users can provide: they can report potential spam, promotion or platform circumvention attempts to the trust and safety team so that these are reviewed by moderators and handled by the platform. It's a simple way to do moderation, the user just needs a way to do the report. The main disadvantage is the delay needed to properly handle a potentially large volume of reports.

To simplify the handling of user reports and make the difference between the types of issue, it is important that users have the option to categorize their reports, with categories such as scam / phishing, sexual promotion or company promotion.

Human moderators

Human moderators are also good at detecting this kind of content, especially because spam and promotion comments are often similar. But note that human moderation has some common issues like the lack of speed and consistency.

When it comes to fraud attempts such as catfishing, it can be challenging to identify such cases since the attackers frequently use techniques to make their profiles seem real. Yet there are some warning signs that content moderators can watch out for, like:

• Inconsistencies in the profile: catfishers frequently use fictitious photos or take pictures from other people's profiles. A catphishing attempt may be detected if the profile image appears to be too good to be true. Also, if the person's information looks contradictory or inconsistent, it can be a red flag that the profile is fake.
• Too much, too soon: catfishers frequently take swift action to build a relationship with their victim. Early on in the dialogue, they might complement or express affection to them. It can be an indication of catphishing if someone seems to be moving too quickly or seems overly invested in the relationship from the start.
• Refusal to meet in person: a catfisher may cite a multitude of justifications to avoid seeing their victim in person. It may be a hint that the profile is fake if the person is reluctant or unwilling to meet up in person.
• Requests for money or personal information: catfishers may attempt to trick their victims into giving them money or personal information. Early on in a relationship, if someone asks for money or sensitive personal information, it might be a catfishing attempt.

Keyword-based filters

Keywords are actually not a very good way to detect such content as there are no specific words allowing to filter the data but more syntactic patterns such as:

want ...? like this post and click on ...

follow my account for ...

List of words such as follow, share or click could still be used to try to get a shorter sample of data to be verified by human moderators, but it would probably miss many examples and catch too many false positives (i.e. comments containing these words but that are not spam nor promotion).

ML models

The use of ML models would possibly be more relevant than a keyword approach for this topic. It would help detect such content with annotated datasets containing spam and promotion comments, but also circumvention attempts.

Automated moderation can also be used to moderate images and videos submitted by users by detecting for instance cases of impersonation or scams.