Audio Moderation
audio-moderationAutomatically detect profanity, hate, personal information, violence, fraud and other unwanted spoken content in videos
Overview
The Audio Moderation Model helps you determine if a video or audio file contains unwanted spoken content such as sexual content, discriminatory content, PII, insults, signs of fraud, or otherwise unwanted or inappropriate language.
The analysis works across multiple languages. The audio track is analyzed across 19 moderation categories, from profanity and hate to personal information, drugs, violence and grooming.
For each detection, the API returns the matching words or expressions along with timestamps, so that you can locate them in the video.
Audio moderation works by analyzing your file's audio track. If you need to detect written text in videos or images, you should use our Text Moderation in Images and Videos instead.
In this guide, you will learn how to moderate audio in video and audio files.
Moderation categories
The categories to be evaluated must be specified in the audio_categories parameter of your API request, as a comma-separated list. The response will only contain results for the selected categories.
Sexual & Suggestive
sexual- Terms or expressions referring to sexual acts, sexual organs, body parts or bodily fluids typically associated with sexual acts.
- Each match is assigned a severity level ranging from low to high.
Insults & Personal Attacks
insult- Words or phrases that insult, undermine or attack the dignity or honor of an individual.
- Signs of disrespect that are generally used to refer to someone.
- Each match is assigned a severity level ranging from low to high.
Discriminatory & Hate
discriminatory- Discriminatory, hateful, or derogatory content targeting individuals or groups based on protected characteristics (e.g. religion, ethnicity, nationality, sexual orientation, gender identity).
- Each match is assigned a severity level ranging from low to high.
Inappropriate Language & Swear Words
inappropriate- Swear words, slang, vulgar language or socially inappropriate expressions used to describe something or to address someone.
- Each match is assigned a severity level ranging from low to high.
Email Addresses
email- Email addresses, including spoken forms such as john dot doe at gmail dot com.
Phone Numbers
phone_number- Phone numbers that are valid numbers in the United States us, France fr or the United Kingdom gb.
- Each match comes with the country of the phone number.
Link & URLs
link- URLs to external websites and pages.
- Additional information allows you to determine whether link is unsafe, deceptive or known to contain otherwise unwanted content (e.g. adult content, gambling, or drugs). More details are available on the URL and link moderation page.
Extremism
extremismWords, expressions or slogans related to extremist ideologies, people or events.
Weapons & Firearms
weaponNames or terms related to guns, rifles, firearms and other weapons (e.g. ak47, kalashnikov, mk-14).
Medical Drugs
medicalNames of prescription or over-the-counter pharmaceutical drugs (e.g. xanax, viagra, flurazepam).
Recreational Drugs
drugNames related to recreational drugs (e.g. cannabis, weed, xtc, cocaine).
Self-Harm & Suicide
self_harm- Terms related to suicide.
- Mentions of self-inflicted injuries.
- Each match is assigned a severity level ranging from low to high.
Violence & Threats
violence- Expressions of violence such as kicking, punching or harming someone, or threatening to do so.
- Each match is assigned a severity level ranging from low to high.
Platform Evasion & Circumvention Attempts
platform_evasion- Attempts to send or lure a user to another platform.
- These attempts are either for a social media platform (what's your ig?, add me on snapchat) or a device-related feature (give me your phone number, wanna facetime?).
Off-Platform Mentions
off_platform_mention- Names of social media platforms.
- The most common social networks are supported by default (facebook, whatsapp, snapchat, instagram, etc.), as well as their variations (ig, insta, snap, etc.).
Content Trade & Solicitation
content_tradeRequests or messages encouraging users to send, exchange or sell photos (send a pic of you, can i see a face pic?) or videos of themselves (do you sell videos?).
Money Requests & Transactions
money_transactionRequests or messages encouraging users to send money (give me 8 dollars, send your credit card, e-transfer money).
Grooming
grooming- Minor-identifying questions.
- Sexualized statements about minors.
- Each match is assigned a severity level ranging from low to high.
Minor Presence
minor- Self-declared underage status.
- Contextual clues indicating the presence of a minor in a conversation (e.g. grade level mention).
Language support
The spoken language is detected automatically during transcription, and the moderation rules of that language are then applied. You do not need to specify the language in your request.
The following languages are supported:
| Language | Code |
| English | en |
| Chinese | zh |
| Danish | da |
| Dutch | nl |
| Finnish | fi |
| French | fr |
| German | de |
| Italian | it |
| Korean | ko |
| Norwegian | no |
| Polish | pl |
| Portuguese | pt |
| Romanian | ro |
| Russian | ru |
| Spanish | es |
| Swedish | sv |
| Tagalog / Filipino | tl |
| Turkish | tr |
If the detected language is not in this list, English moderation rules are applied.
Other languages are available upon request. Please get in touch regarding your language needs.
Severity levels
Severity is used to rank harmful content from mild to extreme. It is available at both the match level and the category level for the categories listed above as having a severity.
| Severity | Description |
| high | The highest severity level. Content that is harmful or dangerous in most contexts and typically requires direct action. |
| medium | Content that may be acceptable in limited contexts (e.g., adult-only environments) but generally requires monitoring. |
| low | Mild language that may be acceptable in many contexts. |
The severity of a category is the highest severity among its matches.
Moderate Audio in a Stored Video
If you haven't already, create an account to get your own API keys.
To moderate the audio part of a stored video, just add audio-moderation to the models parameter in your API calls, and list the categories you want to detect in the audio_categories parameter. You can of course add multiple models at the same time, for instance if you want to detect things like nudity or graphic content along with unwanted spoken content. Head to the Model Reference for the full list of available models.
To moderate a video, you can either send a public URL pointing to this video or send the video's raw bytes. Here's how to proceed to send the raw bytes:
curl -X POST 'https://api.sightengine.com/1.0/video/check-sync.json' \
-F 'media=@/path/to/file.mp4' \
-F 'models=audio-moderation' \
-F 'audio_categories=sexual,insult,discriminatory,phone_number,platform_evasion,off_platform_mention' \
-F 'api_user={api_user}' \
-F 'api_secret={api_secret}'
# this example uses requests
import requests
import json
params = {
# specify the models you want to apply
'models': 'audio-moderation',
'audio_categories': 'sexual,insult,discriminatory,phone_number,platform_evasion,off_platform_mention',
'api_user': '{api_user}',
'api_secret': '{api_secret}'
}
files = {'media': open('/path/to/file.mp4', 'rb')}
r = requests.post('https://api.sightengine.com/1.0/video/check-sync.json', files=files, data=params)
output = json.loads(r.text)
$params = array(
'media' => new CurlFile('/path/to/file.mp4'),
// specify the models you want to apply
'models' => 'audio-moderation',
'audio_categories' => 'sexual,insult,discriminatory,phone_number,platform_evasion,off_platform_mention',
'api_user' => '{api_user}',
'api_secret' => '{api_secret}',
);
// this example uses cURL
$ch = curl_init('https://api.sightengine.com/1.0/video/check-sync.json');
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $params);
$response = curl_exec($ch);
curl_close($ch);
$output = json_decode($response, true);
// this example uses axios and form-data
const axios = require('axios');
const FormData = require('form-data');
const fs = require('fs');
data = new FormData();
data.append('media', fs.createReadStream('/path/to/file.mp4'));
// specify the models you want to apply
data.append('models', 'audio-moderation');
data.append('audio_categories', 'sexual,insult,discriminatory,phone_number,platform_evasion,off_platform_mention');
data.append('api_user', '{api_user}');
data.append('api_secret', '{api_secret}');
axios({
method: 'post',
url:'https://api.sightengine.com/1.0/video/check-sync.json',
data: data,
headers: data.getHeaders()
})
.then(function (response) {
// on success: handle response
console.log(response.data);
})
.catch(function (error) {
// handle error
if (error.response) console.log(error.response.data);
else console.log(error.message);
});
Moderation will happen while you wait for the API response. Once moderation has been done, the API will return a JSON response with the moderation results under the data.audio object.
If unwanted content is found, the detected_categories array lists the categories that were found, and the detections object gives the details for each of these categories. Here is an example response for a video in which someone says "you are such a fucking idiot, call me at 212 555 0198 or add me on snapchat":
{
"status": "success",
"request": {
"id": "req_1OjggusalNb2S7MxwLq2h",
"timestamp": 1509132120.6988,
"operations": 5
},
"media": {
"id": "med_1OjgEqvJtOhqP7sfNe3ga",
"uri": "/path/to/file.mp4"
},
"data": {
"audio": {
"detected_categories": [
"sexual",
"insult",
"phone_number",
"platform_evasion",
"off_platform_mention"
],
"detections": {
"sexual": {
"severity": "high",
"details": [
{
"severity": "high",
"match": "fucking",
"start_ms": 1600,
"end_ms": 1950
}
]
},
"insult": {
"severity": "low",
"details": [
{
"severity": "low",
"match": "idiot",
"start_ms": 2000,
"end_ms": 2350
}
]
},
"phone_number": {
"details": [
{
"match": "212 555 0198",
"start_ms": 3600,
"end_ms": 4800,
"country": "us"
}
]
},
"platform_evasion": {
"details": [
{
"match": "add me on snapchat",
"start_ms": 5200,
"end_ms": 6800
}
]
},
"off_platform_mention": {
"details": [
{
"match": "snapchat",
"start_ms": 6400,
"end_ms": 6800
}
]
}
}
}
}
}
For each match, you will get the matching words, and corresponding timestamps (start_ms and end_ms) expressed in milliseconds from the start of the video. Depending on the category, matches also come with:
- a severity, for categories that have severity levels. The category itself then also has a severity, which is the highest severity among its matches.
- a country, for phone numbers.
- a category, for links. See the URL and link moderation page.
If the audio does not contain any unwanted content, or if the video has no audio track, then an empty detected_categories array will be returned. Here is an example:
{
"status": "success",
"request": {
"id": "req_1OjggusalNb2S7MxwLq2h",
"timestamp": 1509132120.6988,
"operations": 5
},
"media": {
"id": "med_1OjgEqvJtOhqP7sfNe3ga",
"uri": "/path/to/file.mp4"
},
"data": {
"audio": {
"detected_categories": [],
"detections": []
}
}
}