Auto-dubbing using AI Technologies
From television adverts to social media posts, the audio-visual medium dominates the consumption of media in our age. With the increase in streaming services, there has been a boom in people consuming not just regional media but also international media, in various languages from various culture. Shows such as Money Heist and Dark, have proved that people everywhere in the world, want to experience cultures and ideas from all corners of Earth. But due to the obvious language barrier most people cannot enjoy what they wish to enjoy, especially people who are unfamiliar with our lingua franca, English. The process of subtitling can be a very hectic process and the viewer also loses immersion, so dubbing is the most common method of translation. But it is a very expensive and time-consuming process. Our model, with an auto-dubbing solution can alleviate this problem creating new fields of experience.
The Philippines
Client
ePublishing and Content Management Industry
CLIENT
Our client is an Research and Consultancy company based in the Philippines who provide their excellent services all around the globe. They wanted a system that could automate the process of dubbing but with seamless human arbitration for content fluidity and ease of understanding regarding archived American Television Shows. These shows are functionally edutainment, they not only provide entertainment but also can be used for hybrid language training. The reason why Dubbing is chosen as the more viable alternative for customer and viewer expansion because it is universally preferred over subtitles, hence cementing the manner of how new viewership consume media.
CHALLENGE
A complicated model of this manner certainly has a number of clear challenges that have to be tackled, they are :
-
Tonality Errors - Due to mechanistic automation, there’s a chance of the translation losing any emotion tone further than what is observed by our analysis. Certain subtleties might be lost.
-
Cultural Contextuality Errors - Certain aspects of cultural orientations used in media can’t be translated seamlessly using automation, external help is needed to match the context with the content.
-
Video-Audio Demuxing/Muxing Errors - Compatible codecs have to be utilized to make the audio-video merging process to be universal and seamless.
-
Speech Artificiality - Nominally, it is considered that most text-to-speech voices sound very unnatural to the unaccustomed ear and this could definitely break immersion.
SOLUTION
Solution Summary:
-
Auto-captioning is utilized to transcript the content of the audio.
-
Extract emotional and feature content from both the audio and the video present in the video.
-
Use a third-party service to translate the contents of the aforementioned into a coherent translation.
- Utilize text-to-speech to convert this text to translated speech.
- Modulate the speech contextually and emotionally so that it maintains a natural flow.
- Manual modulation can also be done to orient the accuracy of the translated speech.
With the challenges in mind, we have created flexible solutions to deal with them:
- The video is auto-subtitled via transcription operations, and the emotional content of the video is analysed and extracted using machine-learning technologies.
-
The audio has its important auditory features such as tone, pitch, intensity, and analysis extracted to extract the core features and tendencies related to audio.
-
The subtitles are then fed through a text-to-speech processing library which converts the extracted text to targeted audio speech translations.
- The emotional and intensity operations processed from both the video and the audio should be integrated to the translated audio and extracted video.
- Certain minor modulations can be applied after the dubbing process is finished to create a more structurally and contextually intact dub.
RESULT
This process of automated dubbing is highly effective and has a massive churn rate with a consistent output. It simplifies the complicated process of dubbing to a click of a button. This process is also equally cost effective and reduces needless double takes which is the biggest reason for the dubbing process to be time consuming.
By utilizing this model, the dubbing process is streamlined and more cleaner from the outset suitable for
creating a perfect environment for our client to produce content that can be consumed universally with all
the emotions and quirks intact.