This would be a lovely option to have and I find it hilarious that anyone imagines that subtitles are some kind of problem we are trying to get out of.<p>Adding text to movies has a huge array of advantages: it preserves the original performances, it makes the dialog accessible to people with hearing or auditory processing disabilities, they can be multiplexed in a way that audio cannot, etc. Obviously it would be wonderful to have an 'autodub' option! But the times where one would use it seem quite narrow.
I would much rather we use AI to improve subtitling. Better auto speech recognition, per-word millisecond synchronization, positioning based on speaker/screen content-- all these could improve the subtitled experience and would be way less creepy than deepfake voices.
Completely besides whether this is possible (I think it is; it's an offline learning problem with huge resources/economic potential and it's getting attention) I actually prefer subtitles...<p>Coming from Germany, this wasn't always the case, but nowadays watching dubbed content feels so weird and impoverished, I don't even want to imagine what it'd be like knowing that the voice is fully synthetic.
I grew up consuming movies and TV shows with subtitles so it has never bothered me. My theory is that if you're not used to them, then they are really annoying. Most of my American friends do not like them.<p>There is something about watching a movie/show in its original language that you cannot understand. I think its awesome to be immersed in a different world, getting the sounds, inflections, etc from a different language of yours. I will always prefer subtitles to dubbed versions if possible.<p>I understand some people don't, but I think it's just a matter of being exposed to it. And probably the younger, the better.
This article is a thinly veiled promotion for Veritone's MARVEL.ai which doesn't even do translation into foreign languages. It's a text-to-speech encoder that has a variety of voices trained for different languages. The actual translations and subtitles need to be done most likely still by human workers or human-supervised machine translation. The article never really makes the point of the title which is that foreign-language subtitles will be ending any time soon.<p>There should be a disclaimer on this advertorial piece.
AI is also able to create sub titles. So, people will have a choice. Many people prefer subtitles.<p>Dubbing movies is common in big countries where the locals speak nothing else than their own language. Here in Germany, you have to pay attention if you go to the cinema but they do have original language movies as well. Lots of Germans go to see those instead of the badly dubbed versions. I tossed out my TV ages ago because I don't speak German very well and don't like watching dubbed content. Especially if that content is originally in English which I actually can understand.<p>One annoyance with the local netflix here is that while they have most content with the original sound track, they often swap out a lot of the text in these movies with German text. E.g. movie fragments showing a news paper or somebody's text messages, or the famous rolling text in star wars. And of course a lot of American movies or series sometimes feature people not speaking English. Netflix solves this by offering German sub titles. And sometimes those are actually baked into the movie.<p>Par of the issue is that there's a big dubbing industry that is incentivized to keep on dubbing everything whether people like it or not. AI will decimate how profitable that business is.
I like listening to the original audio of foreign movies and it's partly how I learned English, and how I pick up random words from languages I can't really speak.<p>I also respect actors and actresses enough to appreciate their performance includes their original voices.<p>I'm a fast reader and subtitles are like a second nature to me, to the point I don't "remember" having read them (likewise I sometimes don't remember if I read a novel in English or Spanish unless a particular turn of phrase caught my attention).<p>About the only thing I agree with is that dubbing (barring some honorable exceptions) sucks. I did listen to a part of La Casa de Papel's English dub -- the horror!<p>In short: do not want this.
A couple points here.<p>1. AI might be able to prevent bad dubbing/subbing, but it will not be able to compete with _good_ dubbing/subbing in our lifetimes. There's an artistry there that no AI can match.<p>2. I love subtitles and prefer original audio + English subtitles in anything foreign-language. Besides the voices matching up to the movement of the actors' lips and thus not breaking my immersion, I also adore hearing how familiar emotions are reflected in languages I do not know. It makes me feel more connected to humanity.
I think it's a great option but their is an art to translations. This applies especially to books (I'm reading an English translation of <a href="https://en.wikipedia.org/wiki/Zorba_the_Greek" rel="nofollow">https://en.wikipedia.org/wiki/Zorba_the_Greek</a> which is very well done) and while you do have less dialog with movies, I wonder if some subtleties will be missed, particularly around humour.
My wife is a translator and based on our conversations and her experience, subtitles are needed because people want to have the original audio + text, foreign subtitles are needed for obvious reasons, and human translators are needed for the "tricky" parts like new words and expressions, cultural context expressions, stuff that can't be directly translated, among other things.
That's why in many countries a translated piece counts as a new/original piece, and the translator has the rights to claim some of the books IP.
> More Netflix subscribers watched dubbed versions of "Squid Game" than subtitled versions.<p>The default of you load it is dubbed, right? That’s what it started for me. So this stat probably has more to do with the default than anything else. They’re making a weak case that this problem even needs solving.
Couple thoughts:<p>- The hard part about subtitles is not translation it's contextualization. Languages are not variable replacements and words have different meanings based on the context of the sentence. Languages have nuance, phrases, and slang unique to their cultures which further adds difficulty.<p>- coupled with the above, permutations can get large with translated dubs. Which subtitle are you supposed to show? Most translated subs are done off the original audio track and the dubs are done separately. So for example with an English show, the English subs match the English track, but the translation will be off for the Spanish audio track. And the Spanish dubs will not match the Spanish subs either.<p>- lip sync on dubs is a hard problem for many reasons ( full face and body emotion, how words are said in the native tounge, etc..) and will significantly improve the viewing experience. You can do creative things like have fewer full face shots to help with this. It's also important to get the audio mix right so it doesn't sound like an extra track. Keep in mind you now need more videos to cache and store and the ability to swap video streams when you swap audio streams.<p>- animated content offers other approaches than leveraging nn/deepfakes because the assets are already rigged. The use of game engines offers some really interesting opportunities here.
This will not happen for many, many years to come.<p>That is, companies will try it, there will be some bad and ridiculous results, there will be pusheback and the tech will largely be ignored for a decade or so.<p>Main problems will be intonation and inflection. What is a question to an English speaker is a regular sentence to a Russian. And so on. "Petabytes of data from primarily English-speaking libraries" mean jack shit if you wnt to actually do voice translation into/from foreign languages
I want the opposite: subtitles on everything, just not directly on the picture.<p>Opera houses have subtitles on a separate text-only screen (or screens embedded in seats). Why can’t I have a device like that at home?<p>A narrow subtitle display that sits below the TV, sort of like the Touch Bar on the previous-generation MacBooks. I’d buy that.
I second the importance of subtitles. I grew up in a country where dubbing is a huge industry and everything is dubbed. I’ve also lived in another country where almost nothing is dubbed. After moving to the US and seeing many of the films I had previously only known in their dubbed version, I can no longer stand dubbing. I can tell that something is dubbed without having to look at the screen. I’d never want to not have subtitles — I’d rather not watch at all…<p>That being said, as I understand the article this marvel ai thing is supposed to replicate the original performance using the original voice in another language? Of course that would be fantastic a solution, but I doubt that the technology is anything near what it wants to do…
Whilst this is good, as long as it is an option and the user still has the choice to have the original language with subtitles then great. Many enjoy films in other languages they don't fully understand for many reasons, learning the language, nuance of the language and tonality, many things. After all, a Shakespeare love scene in Klingon is just not going to cut the ambience like many other languages.<p>But one thing that bugs me, why is it that even today - we have to have programs with a hard-overlay of sign language interpreter in a you have it if you want it or not and the selection is limited. For those who depend upon such sign language, wouldn't it be great if they could watch anything and some AI would do the sign language for them. You would of thought that such a task would be easy, and the market is there but yet, it seems to elude us.<p>So my hope is that whilst this is useful, it may well open up avenues like the above in which these tools can be leveraged in more practical ways for a group of users who in effect have what they can and can not enjoy, somewhat overly limited as they are at the mercy and whims of the broadcasters who in many instances, only do them to quota tick some regulation put in place to at least not totally shut out a group of people who can't hear at all.
Even as a native English speaker, I prefer watching English speaking shows with English subtitles.<p>I find I don’t need the volume as loud and I get retain more of the dialogue.
What’s really surprising to me is that synching subtitles isn’t a solved problem.<p>Why can’t someone just loosely transcribe without time stamps and sync it to the video?
I speak a minority language that is primarily in spoken form by native speakers, very limited in number. I wonder, how AI technologies can be utilized to enhance the content or auto generating speech. I find it hard. Plus, the economic incentives for doing this work is highly limiting because of a small / limited user base.
The notion that some people prefer dub over sub is not new, but it baffles me that it is to the level of urgent business concern. From what I see in other comments, is it an American thing?<p>We should invest in making people’s ear accustomed/used to hear other languages. It has great intercultural benefits.
This would be interesting if it worked - I recently re-watched The Expanse in German with subtitles on but it wasn’t very helpful as a lot of the time they simply gave up and used the English terms. Some are a bear to translate - try saying “donkey balls” in German, “eseleier”? The captions reverted to English and I didn’t really pick up what the voice track said but wasn’t even close.<p>I had much better luck with Star Trek TOS, very little deviation between the English and German dialog, and the subtitles matched up well with the spoken dialog. Much better for learning - though “er ist tot Jim” doesn’t quite have the same punch IMO.<p><a href="https://www.youtube.com/watch?v=BOtx4Jk-OaQ" rel="nofollow">https://www.youtube.com/watch?v=BOtx4Jk-OaQ</a>
I feel the same way about this as I do about reading poetry in translation, or listening to the (fortunately rare) opera sung in translation. There's something integral about the native qualities of the original language that I can't imagine would survive the AI process.
I’m highly skeptical about AI-replacing-subtitle-as-a-whole. We all know what’s the problem: the quality. We already know that translation is difficult. We already know that acting is difficult.<p>The only feasible outcome here is deepfaking the voice from the recordings from localized voice actors to the actual movie actors. Perhaps, in the future, a voice actor might be doing all of (1) translation (2) voice acting (3) deepfaking in one shot. One person might be able to dub the whole movie - <i>alone</i>.<p>But, for sure, we won’t be able to remove human from the process.
The article seems more against manual dubbing and related difficulties than actual subtitles. The best subtitles I've seen included relevant context on cultural jokes I would have completely missed had there not been explanation text. One example would be recently while watching a Korean drama with dubs that didn't address cultural items and beauty standard, the content itself came off very differently than the intention seemed.
As a programmer I think that this is awesome. But... I'm also thinking that we may be missing opportunities.<p>For example, I prefer to see subtitled movies in their original language (I'm a Spanish native speaker). But there are some dubs that are awesome, even better than the original: for example, Shrek. I'm concerned that eventually studios may just throw the script through Google translate and call it a day.
The article would be far more convincing if it argued that AI could end foreign-language <i>dubbing</i>. Distaste for poor dubbing only partly explains the demand for subtitles. Film aficionados would scoff at the idea of abandoning the original audio, and even casual appreciators of film recognize the value of hearing the original tones and accents.
Oh come on everybody, this is a PR pseudo-advertisement for Veritone. The piece neglects to acknowledge that auto-dubbing is already a thing. No longer research has highly accurate voice synthesis of any specific voice with very limited source material. Realistic voice synthesis is very actively researched, and is already well into auto deep fake territory.
An adjacent concern: Ads bad, yeah, but I appreciate the growing trend of subtitling them. I'm surprised there isn't some existing regulation on that, in the same way that streaming services are required to include them. The lower the cost of accessibility features, the more inclusive the media we consume can become.
On a tangential note: As far as training data for text and speech driven machine learning models go, isn't srt files truly remarkable?<p>They are time stamped hence annotated precisely, hash and file matching makes media content to subtitle matching exact and overall there are language variations.
It is striking to me that this article doesn't discuss the negative impact such a technology will have on actors and translators. In fact, the author makes the surprising claim that there is a shortage of translators and voice actors, which is very hard to believe.
I'm prefer subtitles as opposed to poorly voiced and dubbed versions that are prevelant. Also how will marvel.ai add nuance that is missing in subtitles? The missing nuance is not always about tonality but the context that is missed by machine translation.
Oh. This would be terrible. Not only from a loss of cultural and language preservation, but also losing the actor’ performance. I really dislike watching dubbed shows. If this somehow became a thing it would be sad
Absolutely coming.<p>Goupon founder Andrew Mason's new startup Descript does a fantastic job editing podcasts from text. This same sort of tooling is absolutely going to make it into film.<p>This sort of stuff is already trivially possible with zero effort:<p><a href="https://fakeyou.com/w2l/result/WR:t5m9xayqcwkjf9wng3nb4sxgzhy8p" rel="nofollow">https://fakeyou.com/w2l/result/WR:t5m9xayqcwkjf9wng3nb4sxgzh...</a><p>In ten years, we'll be able to reposition actors and fix flubbed takes. We won't have to visit set, use expensive lenses, or do many of the things that make Hollywood the domain of the rich and well funded.<p>In thirty years, we'll just tell the machines what we want to watch.
AI can generate voices similar to the original actor, but the script will still have to be translated and proofread by expert translators. No AI can fully replace them.
Most people here said it already; that would be a very bad development imho. But one that will happen as, at least in my distributed developer team (all continents, many countries) everyone young hates reading; chat, email, books, subtitles; they all hate it and want to listen and talk, not read & write. So it makes sense. Hope there will be choice always; I grew up with English movies with Dutch subtitles which taught me English and saw, at the same time, how awful it was to watch Rambo saying 'Ich bin John Rambo' with a flinty German dubbed voice. The horror.
Deep fake audio plus alternate video tracks to lip/face synchronize too?<p>Why not just deep fake the entire movie without using any live actors or sets too?
Like those hard to follow auto translated generated captions on YouTube, but spoken...<p>It's a good idea but I'm not sure how well that would work.