Introducing Lip-Sync Detection

A lip-sync deepfake alters a real video to make someone appear to say something they never said.

Sightengine's new Lip-Sync Detection model analyzes a video and determines whether the lips and facial expressions of the people in it have been altered by AI to change what they appear to say.

What is a lip-sync deepfake?

A lip-sync deepfake starts from a real video. Unlike a face swap, the person's identity is preserved. What changes is the mouth, jaw and facial expressions, which are regenerated to match a new audio track.

That new audio can be an edited version of the person's own speech, a recording of someone else, or an entirely AI-generated voice.

Why it matters

Lip-sync tools have become cheap, fast and accessible, and the videos they produce are increasingly hard to spot with the naked eye. These videos can be used to spread false statements, impersonate executives or bypass identity checks.

• Misinformation. Politicians, journalists or public figures shown making statements they never made, often timed to elections, crises or breaking news.
• Impersonation fraud. Fake video messages from executives requesting urgent transfers, or celebrities "endorsing" investment scams and fraudulent products.
• Video identity verification. Re-animated videos used to get through video KYC and onboarding checks that ask users to speak on camera.
• Newsrooms and fact-checking. Verifying user-submitted or viral footage before it is published or debunked.
• Trust & Safety. Harassment, defamation and non-consensual manipulated content targeting real people on social platforms.

Why lip-sync needs a specialized model

A lip-sync video is a hybrid: mostly genuine camera footage, with a small AI-regenerated patch. Because much of the footage remains unchanged, these edits can be difficult for a general-purpose detector to catch.

• AI-generated video detection is built to catch fully synthetic footage, where every pixel comes from a generative model such as Sora, Veo or other text-to-video and image-to-video tools. In a lip-sync video, the background, the body and most of the face are real camera footage. Only the mouth area is regenerated, then blended back into the original frames and re-encoded. The overall signal is overwhelmingly authentic, so a whole-frame detector mostly sees a real video.
• Face swap detection is built to catch identity-level manipulations, where one person's face is replaced with someone else's. With lip-sync, the identity never changes: it is the same person with the same face, and only the mouth movements and expressions differ.
• What's left to catch is small and plays out over time. The manipulation is concentrated on the lips, teeth and jaw, and on how they move from one frame to the next. Detecting it reliably requires a model trained specifically on lip-sync manipulations.

This is why Lip-Sync Detection is a dedicated model. It doesn't replace AI-generated video detection or face swap detection: it covers the gap between them.

Covers open-source and commercial generators

Lip-sync attacks come from a fast-growing ecosystem of tools, from open-source research projects that anyone can run locally to polished commercial platforms. The model is trained to detect manipulations from both, including:

• Open-source models such as Wav2Lip, SadTalker, VideoReTalking, LatentSync, MuseTalk, Hallo2 and many more.
• Commercial platforms such as HeyGen, Synthesia, Hedra, Kling, Sync, OmniHuman and many more.

How it works

1. Submit a video to the Sightengine API.
2. The model analyzes the faces. It examines the faces visible throughout the video, looking for traces of lip-sync manipulation.
3. Get the result. The API returns a lip-sync score: a float from 0 to 1, where values closer to 1 indicate a high likelihood that the video has been manipulated.

You can tune the threshold on the score to match your precision/recall requirements, for instance with a stricter threshold for automated decisions and a looser one to route videos to human review.

Using it with other detection models

Deepfakes rarely rely on a single technique. Sightengine provides a dedicated model for each type of manipulation, so that you can combine them to match the threats you face:

• Face Swap Detection: the person's identity has been replaced with someone else's.
• Lip-Sync Detection: the identity is kept, but the speech and expressions have been altered.
• AI Speech Detection: the voice itself is synthetic or cloned.
• AI-generated Video Detection: the entire video was produced by a generative model.

These models are complementary. A lip-synced video is often paired with a cloned voice: running Lip-Sync Detection and AI Speech Detection together lets you catch both the visual and the audio side of the same deepfake. Adding Face Swap Detection covers attacks that change who is on screen, not just what they say.

Get started

Lip-Sync Detection is available now, on request.

• Contact our team to get access and discuss your use case.
• Learn more about our other deepfake forensics tools: Deepfake Detection and AI Voice Detection.

Read more

AI-generated images - The Trust & Safety Guide

This is a complete guide to the Trust&Safety challenges, regulations and evolutions around GenAI for imagery.

Joining the Online Dating & Discovery Association: Towards Safer Connections

This is a step to further enhance end-user safety in the online dating realm.

Head back to the Knowledge Center