Last week, two New York-based startups — ElevenLabs and Captions — each released a video dubbing tool powered by artificial intelligence (AI) that aims to preserve the speaker’s voice in different languages. The tools allow users to reach audiences in other language markets quickly and cheaply, as well as to hear what their voice might sound like in different languages.
ElevenLabs, a voice cloning and synthesis company, unveiled its AI Dubbing online tool, which can automatically translate spoken audio and video content into 20 languages, including Hindi, Portuguese, Japanese, Ukrainian, Polish, and Arabic.
CEO and co-founder of ElevenLabs Mati Staniszewski said in a statement, “AI Dubbing will help audiences enjoy any content they want, regardless of the language they speak. And it will mean content creators can easily and authentically access a far bigger audience across the world.”
Meanwhile, AI video startup Captions debuted a free iOS app called Lipdub that can translate speech in prerecorded videos into 28 languages, even altering the speaker’s lip movements to match the other language. As a fun addition, Lipdub also offers English slang options like Gen Z, Texan, and pirate.
“The entire team at Captions could not be more excited about what Lipdub will unlock for everyone across the globe,” writes COO and co-founder Dwight Churchill on the company blog. “It’s a first of its kind in terms of quality, performance, and use case.”
With either tool, all you have to do is upload a file and you’ll receive a dubbed version within minutes. Both tools attempt to recreate the speaker’s voice by preserving the emotion and intonation in the original audio.
Startups aren’t the only companies working on AI video dubbing tools, with YouTube planning to introduce its tool called Aloud next year. Content creators on YouTube will have the option to click to generate a dub of their video and can then decide whether to publish it.