<< Back to Blog

Best moderation strategy: Humans or Deep Neural Networks?

AI touching humans

Moderating your user-generated images is important to protect your brand and keep your users and advertisers happy. What are the different approaches at hand and how do you choose the appropriate solution? In this post we will go through the key elements that you should have in mind when you define your moderation strategy.

Three different approaches

In this post, we consider the 3 common approaches to content moderation for websites, apps and platforms that have user-generated content:

  • Operator-based moderation - content is reviewed one-by-one by human operators, either your own employees or through specialized subcontractors
  • Automated moderation - you rely on an algorithm to classify and moderate your content
  • User-based moderation - you rely on your community of users to report and flag unwanted content

With this in mind let's go through the main questions you should ask yourself.

What user experience do you want

This is probably the most important question, as we're sure you don't want to mess with your users' experience. So the question is to define what is acceptable - or not - for your users' experience. The main elements being waiting time, error rates, unwanted exposures and privacy protection.

Target waiting time. Very quick moderation where you get a response within a second lets you implement a seamless upload flow and provide instant feedback to end users. This what you can expect of automated moderation tools. Operator-based ones will provide answers with longer waiting times: a few minutes for the most expensive ones and up to a few hours.

The waiting time will also depend on the opening hours you set. If images are submitted at a very specific time, you can ask for operators to, and have alonger waiting times during nights or week-ends. If your users are spread out worldwide or if usage is not concentrated in specific time slots, you would probably need to go for 24/7 moderation.

Finally, an other way to look at waiting times is to determine if it would be acceptable for you to display unmoderated images. This is far from perfect but a useful fallback solution when your moderation times are long.

Rate of false positives. False positives are photos that are rejected even though they should be accepted. Same solutions as for the false negatives.

Rate of false negatives. False negatives are photos that are accepted by moderation even though they shouldn't. This is also known as unwanted exposure, wherein some of your users may inadvertently be exposed to unwanted content.

If you want to limit unwanted exposure, you would of course have to sort out user-based moderation. And if you need an extremely low rate of false negatives (<0.1% for instance) you should probably combine different approaches such as automated moderation with crowdsourced reports to identify any misses, or use extended operator moderation with double or triple checks where photos are reviewed by 2 or 3 different people to reduce errors.

Privacy. If content is reviewed by human operators before getting published, this has to be clear to your users and in your terms and conditions. This is acceptable in most cases, though in some cases users may dislike that, see below.

What's your content and the related legislation

Semi-public or private content should not be moderated just like public content. When uploading semi-public photos - such as in closed groups - or private photos - such as in one-to-one messaging - users expect their content to be viewed only by those they have chosen. If you rely on human moderation for those cases, you need to make sure that users are aware.

What are your obligations

If you have ads running on your site, whether it's through ad networks or direct ad sales, you have to protect your advertisers' brands by making sure the ads will never show alongside unwanted content. Ad networks such as Google Adsense are very strict and can ban publishers who let a few adult images next on the same page as ads. Even if the image is submitted by users.

If the images or shown in a mobile app, you have to abide by the Apple App Store or Google Play Store terms and conditions. Apple's terms state that you must perform some kind of moderation to make sure no adult / violent / illegal content is displayed.

Google's terms require that you make sure no adult content is displayed as long as your app is not X-rated

Who are your users

If your site or app accomodates under-aged users, you have a legal (and moral) obligation to protect them through proper moderation

Complexity and specificity of your moderation rules

If your moderation rules are not straightforward, you probably won't be able to rely on User-based moderation as you cannot expect your users to make reports based on complex rules. Human-based and automated moderation are usually good fits. Sightengine has developed fine-tuned moderation tools to cater to complex rules that customers have been setting up.

Examples of complex rules would be "accept photos of people in swimsuits or bikinis but not photos of women in lingerie" or "flag photos of women showing their cleavage in a suggestive way".

If you have moderation rules that are very specific to you, then it is unlikely that automated moderation tools have been or could be developed for your specific situation. In any case, reach out and we would be happy to see if a custom solution can be built.

Examples of specific rules that we've been asked for was to detect and filter images containing very specific logos.

Conclusion

OperatorAutomatedUser reports
Seamless upload flowNOYESNO
Limited budgetNOYESYES
Semi-public or private imagesNOYESYES
< 0.1% false negatives or false positivesYESNONO
Support complex rulesYESYESNO
Support very specific rulesYESNONO
Appropriate for content next to adsYESYESNO
Appropriate for content on App Store / Play StoreYESYESNO
Sightengine

Sightengine is an Artificial Intelligence company that develops image moderation APIs to empower business owners and developers. Sightengine's powerful image and video analysis technology is built on proprietary state-of-the-art Deep Learning systems and is made available through simple and clean APIs.

SIGN UP