Text Moderation in Images/Videos

Overview

The Visual Text Moderation API is useful to determine if an image or video contains unwanted text such as profanity, contact information, scam risk or other restricted content.

This model is based on our pattern-matching algorithms. It will return flagged words and expressions across multiple fine-grained categories. The response can then be used to reject, flag or review the image / video.

The Text Moderation Engine is a lot stronger than word-based filters. It uses advanced language analysis to detect objectionable content, even when users specifically attempt to circumvent filters. It covers obfuscation techniques such as repetitions, insertions, spelling mistakes, leet speak and more. Learn more on our Text Moderation Engine.

Just like our other Image Moderation APIs, this API performs the analysis entirely automatically. There are no humans reviewing your content. This helps us achieve very fast turnaround times — typically a few hundreds of milliseconds — and very high scalability.

Principles

The Text Moderation API for Images works in several steps:

1. Detection of text items contained in the image
2. Recognition of the text (this is equivalent to transforming text into string objects)
3. Analysis of the recognized text, through our rule-based text moderation engine

Use-cases

• Prevent users from adding insults, profanity, racial slurs or sexually suggestive text in an image
• Remove photos that contain PII such as an email address or phone number
• Flag users who include links / URLs in their images
• Prevent users from mentioning their social accounts in images
• Adapt moderation to your own rules by allowing or forbidding restricted content on your platform (allowing content trade but removing violent content for instance)

Categories

The rules are grouped into categories to help you implement custom filters based on the type of flagged content.

Profanity Detection in Images

Profanity Detection allows you to detect insults, discriminatory content, sexual content or other inappropriate words and phrases in your images. Each flagged word or expression is associated with an intensity level that can be used for even more fine-grained moderation. See our Rule-based Profanity Detection model for more information.

• Sexual
  text.profanity.sexual
  • Term or expression that refers to sexual acts
  • Term or expression referring to sexual organs, body parts or bodily fluids typically associated with sexual acts
  • Example: here's my d*ck
• Discriminatory
  text.profanity.discriminatory
  • Discriminatory and derogatory content
  • Mostly hate speech that instigates violence or hate against groups based on specific characteristics such as religion, national or ethnic origin, sexual orientation or gender identity
  • Example: he's a tard
• Insult
  text.profanity.insult
  • Words or phrases that undermine the dignity or honor of an individual
  • Signs of disrespect that are generally used to refer to someone
  • Example: you fatso!
• Inappropriate
  text.profanity.inappropriate
  • Swear words
  • Slang
  • Familiar / informal or socially inappropriate / unacceptable words or phrases to describe something or to talk to someone
  • Example: what the cr@p?
• Grawlix
  text.profanity.grawlix
  • String of typographical symbols that are typically used in place of obscenity or profanity
  • Example: you #@$%!!

Image containing flagged profanity

Contact Information Detection in Images

PII Detection

The Personal Information Detection model detects PII (personally identifiable information) in your visual content.

• Email
  text.contact.pii.email
  • Email addresses
  • Includes obfuscated email addresses
  • Example: rick[@]gmail . com
  

  Image with a flagged email address
• Phone number
  text.contact.pii.phone_number_**
  • Phone numbers that are valid numbers in the countries specified through the opt_countries parameter
  • Includes obfuscated numbers
  • Example: +1(605)493-3483

  Notes:

  • You can select the countries to be covered through the opt_countries parameter. Provide a comma-separated list of the ISO 3166 2-letter country codes, for instance us for the United-States, fr for France. See the full list of supported countries.
  • If you do not specify any country, the API will default to the following list of countries: United States us, France fr, United Kingdom gb.
  

  Image with a flagged US phone number
• Username
  text.contact.pii.username
  • Name or handle that uniquely identifies a user within a digital system or platform, typically social networks
  • Example: iamthebest007
• SSN
  text.contact.pii.ssn
  • US social security number
• IP
  text.contact.pii.ip
  • IPv4 addresses, the most frequent and oldest format for IP addresses, typically written as 4 numbers separated by dots, for instance 52.222.158.75
  • IPv6 addresses, the newer format that allows for a greater variety of formatting options, for instance 2600:9000:2247:6000:8:a1f0:7e00:93a1

Link / URL Detection

Links and URLs will be detected and flagged as such in the image.

• Link / URL
  text.contact.link
  • URLs to external websites and pages
  • Includes obfuscated URLs and links
  • Example: http://harmfulsiteexample.com

  Notes:

  • In addition to detecting URLs in Images or Videos, you can also moderate the link to determine if the link is unsafe, deceptive or known to contain otherwise unwanted content (such as adult content, gambling or drugs). More details are available on the URL and link moderation page.
  • Links can also appear as QR codes on images. See our QR code Moderation API for more information.
  

  Image with a flagged link to a twitter handle

Social account Detection

• Social
  text.contact.social
  • Mentions of social networks and social accounts alone
  • Attempts to send a user to another mentioned social media platform
  • Example: facebook, fb, whatsapp, snapchat, give me your instagram, insta, send ur snap, etc.

  Notes:

  • Logos of said social networks won't be flagged by this model.
  • The most common social networks are supported by default. Other social networks can be made available on a custom basis. Reach out for more.
  

  Image with a flagged mention of a social network or account

Other circumvention and spam Detection

• Other circumvention and spam
  text.contact.other
  • Attempts to send a user to another unmentioned platform
  • Includes content usually considered as spam
  • Examples: wanna message?, follow 4 follow

Scam risk Detection in Images

• Content trade
  text.scam.content-trade
  • Requests or messages encouraging users to send, exchange or sell photos or videos of themselves
  • Examples: can u send a pic?
• Financial
  text.scam.financial
  • Requests or messages encouraging users to send money
  • Examples: gimme 200chf

Other Restricted Content Detection in Images

• Extremism
  text.restricted.extremism
  • Words, expressions or slogans related to extremist ideologies, people or events
  • Example: join the kkk
• Weapon
  text.restricted.weapon
  • Names or terms that are related to guns, rifles and firearms (models, brands, etc.)
  • Examples: ak-47, ak 101, imbel, etc.
• Medical
  text.restricted.medical
  • Names related to medical drugs used in specific situations such as depression, anxiety, insomnia, obesity, pain or erectile disfunction
  • Examples: xanax, diazepam, percocet, viagra, etc.
• Drug
  text.restricted.drug
  • Names related to recreational drugs
  • Examples: lsd, ket, xtc, n-bomb, etc.
• Self-harm
  text.restricted.self-harm
  • Terms related to suicide and self-inflected injuries
  • Examples: i should kms
• Violence
  text.restricted.violence
  • Expressions of violence such as kicking, punching or harming someone, or threatening to do so
  • Examples: smash ur head

Custom disallow or allow lists

You can use custom disallow lists (also known as blacklist or ban list) and allow lists along with with Text-in-image moderation and Text-in-video moderation.

To create a new text list, go to the dedicated page on your dashboard.

Once your text list has been created, you will have the opportunity to add entries to the text list.

View your custom list

Each entry is a text item that you want to detect, along with meta-data to help our engine determine how and when this entry should be detected. The following information will be needed:

• Text: this is the text item you want to detect.
• Language: this is the language to which the text item applies. If it appears in a text item with a different language, it will not be flagged. If you need a text item to be detected in all languages, select all
• Match type: this is the way the detection engine should detect. Choose exact match if you want the engine to detect the exact string, with no modifications (apart from case modifications).
  Choose standard match if you want the engine to detect variations such as phonetic variations, repetitions, typos, leet speak etc... This will typically cover millions of variations and make sure your users cannot simply obfuscate the word to fool the engine.
• Category: this is a suggested categorization of the text item, to help you sort and filter entries.

Add a new entry to your custom list

While most users only need to create a single text list, you can create multiple text lists to apply different moderation criteria to different types of text items.

In order to use a custom list, you will have to specify the corresponding id of the list in the opt_textlist parameter in your API calls.

Languages

English is the default language used for text moderation. This means that if you do not specify anything, the API engine will assume that the text is in english and will process it as such.

If you know with high confidence what language is used in the message, for instance because your users tend to mostly speak one language, you can set the language with the opt_lang parameter. To do so, use the ISO 639-1 codes for languages:

If you are unsure about the language used by a user, you can specify multiple languages as a comma-separated list. For instance en,fr,es for users that might write in english, french or spanish. The API will then automatically detect the language and apply the corresponding rules. We recommend specifying the shortest possible list as this will yield better results both in speed and accuracy.

Other languages are available upon request. Please get in touch.

Use the model (images)

If you haven't already, create an account to get your own API keys.

Moderate text in images

Let's say you want to moderate the following image:

You can either share a URL to the image, or upload the image file.

Option 1: Send image URL

Here's how to proceed if you choose to share the image URL:

curl -X GET -G 'https://api.sightengine.com/1.0/check.json' \
    -d 'models=text-content' \
    -d 'api_user={api_user}&api_secret={api_secret}' \
    --data-urlencode 'url=https://sightengine.com/assets/img/examples/example-text-ocr-3.jpg'

# this example uses requests
import requests
import json

params = {
  'url': 'https://sightengine.com/assets/img/examples/example-text-ocr-3.jpg',
  'models': 'text-content',
  'api_user': '{api_user}',
  'api_secret': '{api_secret}'
}
r = requests.get('https://api.sightengine.com/1.0/check.json', params=params)

output = json.loads(r.text)

$params = array(
  'url' =>  'https://sightengine.com/assets/img/examples/example-text-ocr-3.jpg',
  'models' => 'text-content',
  'api_user' => '{api_user}',
  'api_secret' => '{api_secret}',
);

// this example uses cURL
$ch = curl_init('https://api.sightengine.com/1.0/check.json?'.http_build_query($params));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
curl_close($ch);

$output = json_decode($response, true);

// this example uses axios
const axios = require('axios');

axios.get('https://api.sightengine.com/1.0/check.json', {
  params: {
    'url': 'https://sightengine.com/assets/img/examples/example-text-ocr-3.jpg',
    'models': 'text-content',
    'api_user': '{api_user}',
    'api_secret': '{api_secret}',
  }
})
.then(function (response) {
  // on success: handle response
  console.log(response.data);
})
.catch(function (error) {
  // handle error
  if (error.response) console.log(error.response.data);
  else console.log(error.message);
});

Option 2: Send image file

Here's how to proceed if you choose to upload the image file:

curl -X POST 'https://api.sightengine.com/1.0/check.json' \
    -F 'media=@/path/to/image.jpg' \
    -F 'models=text-content' \
    -F 'api_user={api_user}' \
    -F 'api_secret={api_secret}'

# this example uses requests
import requests
import json

params = {
  'models': 'text-content',
  'api_user': '{api_user}',
  'api_secret': '{api_secret}'
}
files = {'media': open('/path/to/image.jpg', 'rb')}
r = requests.post('https://api.sightengine.com/1.0/check.json', files=files, data=params)

output = json.loads(r.text)

$params = array(
  'media' => new CurlFile('/path/to/image.jpg'),
  'models' => 'text-content',
  'api_user' => '{api_user}',
  'api_secret' => '{api_secret}',
);

// this example uses cURL
$ch = curl_init('https://api.sightengine.com/1.0/check.json');
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $params);
$response = curl_exec($ch);
curl_close($ch);

$output = json_decode($response, true);

// this example uses axios and form-data
const axios = require('axios');
const FormData = require('form-data');
const fs = require('fs');

data = new FormData();
data.append('media', fs.createReadStream('/path/to/image.jpg'));
data.append('models', 'text-content');
data.append('api_user', '{api_user}');
data.append('api_secret', '{api_secret}');

axios({
  method: 'post',
  url:'https://api.sightengine.com/1.0/check.json',
  data: data,
  headers: data.getHeaders()
})
.then(function (response) {
  // on success: handle response
  console.log(response.data);
})
.catch(function (error) {
  // handle error
  if (error.response) console.log(error.response.data);
  else console.log(error.message);
});

API response

The API will then return a JSON response with the following structure:

                  
                  
{
  "status": "success",
  "request": {
      "id": "req_22Qd0gUNmRH4GCYLvYtN6",
      "timestamp": 1512483673.1405,
      "operations": 1
  },
  "text": {
      "personal": [
        {
          "type": "phone_number_us",
          "match": "+1 800 222 2408"
        }
      ],
      "link": [],
      "social": [],
      "profanity": [],
      "social": [],
      "extremism": [],
      "medical": [],
      "drug": [],
      "weapon": [],
      "content-trade": [],
      "money-transaction": [],
      "spam": [],
      "violence": [],
      "self-harm": [],
      "ignored_text": false
  },
  "media": {
      "id": "med_22Qdfb5s97w8EDuY7Yfjp",
      "uri": "https://sightengine.com/assets/img/examples/example-text-ocr-3.jpg"
  }
}


              

See response description

Use the model (videos)

Moderate text in videos

Option 1: Short video

Here's how to proceed to analyze a short video (less than 1 minute):

curl -X POST 'https://api.sightengine.com/1.0/video/check-sync.json' \
  -F 'media=@/path/to/video.mp4' \
  -F 'models=text-content' \
  -F 'api_user={api_user}' \
  -F 'api_secret={api_secret}'

# this example uses requests
import requests
import json

params = {
  # specify the models you want to apply
  'models': 'text-content',
  'api_user': '{api_user}',
  'api_secret': '{api_secret}'
}
files = {'media': open('/path/to/video.mp4', 'rb')}
r = requests.post('https://api.sightengine.com/1.0/video/check-sync.json', files=files, data=params)

output = json.loads(r.text)

$params = array(
  'media' => new CurlFile('/path/to/video.mp4'),
  // specify the models you want to apply
  'models' => 'text-content',
  'api_user' => '{api_user}',
  'api_secret' => '{api_secret}',
);

// this example uses cURL
$ch = curl_init('https://api.sightengine.com/1.0/video/check-sync.json');
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $params);
$response = curl_exec($ch);
curl_close($ch);

$output = json_decode($response, true);

// this example uses axios and form-data
const axios = require('axios');
const FormData = require('form-data');
const fs = require('fs');

data = new FormData();
data.append('media', fs.createReadStream('/path/to/video.mp4'));
// specify the models you want to apply
data.append('models', 'text-content');
data.append('api_user', '{api_user}');
data.append('api_secret', '{api_secret}');

axios({
  method: 'post',
  url:'https://api.sightengine.com/1.0/video/check-sync.json',
  data: data,
  headers: data.getHeaders()
})
.then(function (response) {
  // on success: handle response
  console.log(response.data);
})
.catch(function (error) {
  // handle error
  if (error.response) console.log(error.response.data);
  else console.log(error.message);
});

Option 2: Long video

Here's how to proceed to analyze a long video. Note that if the video file is very large, you might first need to upload it through the Upload API.

curl -X POST 'https://api.sightengine.com/1.0/video/check.json' \
  -F 'media=@/path/to/video.mp4' \
  -F 'models=text-content' \
  -F 'callback_url=https://yourcallback/path' \
  -F 'api_user={api_user}' \
  -F 'api_secret={api_secret}'

# this example uses requests
import requests
import json

params = {
  # specify the models you want to apply
  'models': 'text-content',
  # specify where you want to receive result callbacks
  'callback_url': 'https://yourcallback/path',
  'api_user': '{api_user}',
  'api_secret': '{api_secret}'
}
files = {'media': open('/path/to/video.mp4', 'rb')}
r = requests.post('https://api.sightengine.com/1.0/video/check.json', files=files, data=params)

output = json.loads(r.text)

$params = array(
  'media' => new CurlFile('/path/to/video.mp4'),
  // specify the models you want to apply
  'models' => 'text-content',
  // specify where you want to receive result callbacks
  'callback_url' => 'https://yourcallback/path',
  'api_user' => '{api_user}',
  'api_secret' => '{api_secret}',
);

// this example uses cURL
$ch = curl_init('https://api.sightengine.com/1.0/video/check.json');
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $params);
$response = curl_exec($ch);
curl_close($ch);

$output = json_decode($response, true);

// this example uses axios and form-data
const axios = require('axios');
const FormData = require('form-data');
const fs = require('fs');

data = new FormData();
data.append('media', fs.createReadStream('/path/to/video.mp4'));
// specify the models you want to apply
data.append('models', 'text-content');
// specify where you want to receive result callbacks
data.append('callback_url', 'https://yourcallback/path');
data.append('api_user', '{api_user}');
data.append('api_secret', '{api_secret}');

axios({
  method: 'post',
  url:'https://api.sightengine.com/1.0/video/check.json',
  data: data,
  headers: data.getHeaders()
})
.then(function (response) {
  // on success: handle response
  console.log(response.data);
})
.catch(function (error) {
  // handle error
  if (error.response) console.log(error.response.data);
  else console.log(error.message);
});

Option 3: Live-stream

Here's how to proceed to analyze a live-stream:

curl -X GET -G 'https://api.sightengine.com/1.0/video/check.json' \
    --data-urlencode 'stream_url=https://domain.tld/path/video.m3u8' \
    -d 'models=text-content' \
    -d 'callback_url=https://your.callback.url/path' \
    -d 'api_user={api_user}' \
    -d 'api_secret={api_secret}'

# if you haven't already, install the SDK with 'pip install sightengine'
from sightengine.client import SightengineClient
client = SightengineClient('{api_user}','{api_secret}')
output = client.check('text-content').video('https://domain.tld/path/video.m3u8', 'https://your.callback.url/path')

// if you haven't already, install the SDK with 'composer require sightengine/client-php'
use \Sightengine\SightengineClient;
$client = new SightengineClient('{api_user}','{api_secret}');
$output = $client->check(['text-content'])->video('https://domain.tld/path/video.m3u8', 'https://your.callback.url/path');

// if you haven't already, install the SDK with 'npm install sightengine --save'
var sightengine = require('sightengine')('{api_user}', '{api_secret}');
sightengine.check(['text-content']).video('https://domain.tld/path/video.m3u8', 'https://your.callback.url/path').then(function(result) {
    // The API response (result)
}).catch(function(err) {
    // Handle error
});

Moderation result

The Moderation result will be provided either directly in the request response (for sync calls, see below) or through the callback URL your provided (for async calls).

Here is the structure of the JSON response with moderation results for each analyzed frame under the data.frames array:

            
                  
{
  "status": "success",
  "request": {
    "id": "req_gmgHNy8oP6nvXYaJVLq9n",
    "timestamp": 1717159864.348989,
    "operations": 21
  },
  "data": {
    "frames": [
      {
        "info": {
          "id": "med_gmgHcUOwe41rWmqwPhVNU_1",
          "position": 0
        },
        "text": {
          "personal": [],
          "link": [],
          "social": [],
          "profanity": [],
          "social": [],
          "extremism": [],
          "medical": [],
          "drug": [],
          "weapon": [],
          "content-trade": [],
          "money-transaction": [],
          "spam": [],
          "violence": [],
          "self-harm": [],
          "ignored_text": false
        },
      },
     ...
    ]
  },
  "media": {
    "id": "med_gmgHcUOwe41rWmqwPhVNU",
    "uri": "yourfile.mp4"
  },
}


            

You can use the classes under the text object to analyze text content in the video.

Any other needs?

See our full list of Image/Video models for details on other filters and checks you can run on your images and videos. You might also want to check our Text models to moderate text-based content: messages, reviews, comments, usernames...