Audio AI: isolating vocals from stereo music using Convolutional Neural Networks

This is an awesome project, but it seems it was done without reference to academic literature on source separation. In fact, people have been doing audio source separation for years with neural networks.<p>For instance, Eric Humphrey at Spotify Music Understanding Group describes using a U-Net architecture here: <a href="https:&#x2F;&#x2F;medium.com&#x2F;this-week-in-machine-learning-ai&#x2F;separating-vocals-in-recorded-music-at-spotify-with-eric-humphrey-51c2f85d1451" rel="nofollow">https:&#x2F;&#x2F;medium.com&#x2F;this-week-in-machine-learning-ai&#x2F;separati...</a> - paper at <a href="http:&#x2F;&#x2F;openaccess.city.ac.uk&#x2F;19289&#x2F;1&#x2F;7bb8d1600fba70dd79408775cd0c37a4ff62.pdf" rel="nofollow">http:&#x2F;&#x2F;openaccess.city.ac.uk&#x2F;19289&#x2F;1&#x2F;7bb8d1600fba70dd7940877...</a><p>They compare their performance to the widely-cited state of the art Chimera model (Luo 2017): <a href="https:&#x2F;&#x2F;www.ncbi.nlm.nih.gov&#x2F;pmc&#x2F;articles&#x2F;PMC5791533&#x2F;#R24" rel="nofollow">https:&#x2F;&#x2F;www.ncbi.nlm.nih.gov&#x2F;pmc&#x2F;articles&#x2F;PMC5791533&#x2F;#R24</a> with examples at <a href="http:&#x2F;&#x2F;naplab.ee.columbia.edu&#x2F;ivs.html" rel="nofollow">http:&#x2F;&#x2F;naplab.ee.columbia.edu&#x2F;ivs.html</a> - from the examples, there&#x27;s significantly less distortion than OP.<p>Not to discourage OP from doing first-principles research at all! But it&#x27;s often useful to engage with the larger community and know what&#x27;s succeeded and failed in the past. This is a problem domain where progress could change the entire creative landscape around derivative works (&quot;mashups&quot; and the like), and interested researchers could do well to look towards collaboration rather than reinventing each others&#x27; wheels.<p>EDIT: The SANE conference has talks by Humphrey and many others available online: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;channel&#x2F;UCsdxfneC1EdPorDUq9_XUJA&#x2F;videos" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;channel&#x2F;UCsdxfneC1EdPorDUq9_XUJA&#x2F;vid...</a>

What motivates people to invent phrases like &quot;perceptual binarization&quot; when googling &quot;audio binary mask&quot; literally gives you citations in the field that have been doing this for years?<p>For example, 2009 musical sound separation based on binary time frequency masking.<p>Or more recent stuff using deep learning. Also the field generally prefers ratio masks because they lead to better sounding output.

Hello, a little self promotion, you can see it our experiment with some deep neural networks doing real-time audio processing in the browser, using tensorflow.js<p><a href="http:&#x2F;&#x2F;gistnoesis.github.io&#x2F;" rel="nofollow">http:&#x2F;&#x2F;gistnoesis.github.io&#x2F;</a><p>If you want to see how it&#x27;s done it&#x27;s shared source : <a href="https:&#x2F;&#x2F;github.com&#x2F;GistNoesis&#x2F;Wisteria&#x2F;" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;GistNoesis&#x2F;Wisteria&#x2F;</a><p>Thanks

Does anyone know if this is related to the new iZotope RX 7 vocal isolation &amp; stemming tools? It does seem to be talking about something similar, especially when it mentions using the same technique to split a song into instrument stems.<p>(Or to put it another way - there is commercial music software released in the last year that lets you do this yourself now.)<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=kEauVQv2Quc" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=kEauVQv2Quc</a><p><a href="https:&#x2F;&#x2F;www.izotope.com&#x2F;en&#x2F;products&#x2F;repair-and-edit&#x2F;rx&#x2F;music.html" rel="nofollow">https:&#x2F;&#x2F;www.izotope.com&#x2F;en&#x2F;products&#x2F;repair-and-edit&#x2F;rx&#x2F;music...</a>

I used to work in an audio processing research center back in 2003, and colleagues next to me were able to isolate each instrument in a stereo mix live using the fact that they were &quot;placed&quot; on different spot in the stereo plane.<p>Don&#x27;t ask me how they did that, it was close to magic to me at that time, but i&#x27;m sure it wasn&#x27;t neural networks. Although it probably involved convolution, as it is the main tool for producing audio filters.<p>If anyone has more info on the fundamental differences of the neural network approach compared to the &quot;traditional&quot; one, i&#x27;d be thankful.

Trivia: Avery Wang, the guy who invented the Shazam algorithm and was their CTO did his PhD thesis on this topic:<p><a href="https:&#x2F;&#x2F;ccrma.stanford.edu&#x2F;~jos&#x2F;EE201&#x2F;More_Recent_PhD_EEs_CCRMA.html" rel="nofollow">https:&#x2F;&#x2F;ccrma.stanford.edu&#x2F;~jos&#x2F;EE201&#x2F;More_Recent_PhD_EEs_CC...</a><p>``Instantaneous and Frequency-Warped Signal Processing Techniques for Auditory Source Separation&#x27;&#x27; (1994)

Please stop publishing on Medium. I&#x27;m getting error &quot;You read a lot. We like that. You’ve reached the end of your free member preview for this month. Become a member now for $5&#x2F;month to read this story&quot;.<p>Not gonna do that.

There is a lagged autoregressive technique used in forensic analysis that allows 3d reconstruction using 1d (mic) sound.<p>A CNN should be able to back that out too, and do other things like regenerate a 3d space. In the right, high-fidelity, acoustic tracks could be the spatial information to reconstruct a stage and a performance. It would be neat&#x2F;beautiful&#x2F;(possibly very powerful) to back video out of audio in that way.

The presentation of this project alone is a visual tour de force to say nothing of the technical quality. Beautiful and easily digestible post. As with any interesting, non-toy applied ML problem, the dataset generation is really where the innovation is. It gets a neat little graphic at the end. As far as how the author characterizes the problem, I think the word he&#x27;s looking for is &quot;semantic segmentation&quot; - he&#x27;s trying to classify each pixel of the spectrograph as vocal&#x2F;non-vocal. I&#x27;d be curious if he could drop the dataset into pix2pix-style networks and achieve the same results.

Question: has any progress been made in removing reverb?<p>There are many, many historical recordings (and modern ones made in less-than-ideal circumstances) that suffer badly from reverb. Seems like a valuable use-case that -ought- to be in reach today.

Just wanted to mention there&#x27;s some folks doing realtime source separation (not sure exactly how they&#x27;ve implemented it) with a DNN for reduction of background noise in, eg: Skype conversations.<p>I&#x27;m not involved with them in any way, but I&#x27;ve been amazed with its ability to cancel out coffee-shop style noise.<p>Check out <a href="https:&#x2F;&#x2F;krisp.ai&#x2F;technology&#x2F;" rel="nofollow">https:&#x2F;&#x2F;krisp.ai&#x2F;technology&#x2F;</a> - Mac&#x2F;Windows. I wish they had Linux support!<p>Edit: Appears they don&#x27;t have Windows support yet.

Clicked into to the article because I was curious how the training set was created. Using the acapella version is an amazing idea! Wished the article went more in-depth about this section.

Is it possible yet to take a recording of singing and generate a model of the singer for synthesis, like a Vocaloid?

Question: Currently building earphones with great active-noise-cancellation is a secret kept within few companies.<p>This means they&#x27;re expensive($300 headphones from Bose etc).<p>Do neural network make this simpler ?<p>And do you think they can be applied cheaply enough,say for $99 headphones ?<p>I assume this will sell really well, and justify creating a dedicated chip, with time.

Soon enough there will be an AI filter that will take any old hacky, coughing, wheezing singer running around on stage, singing out of tune - and turn it into virtuoso chops. Maybe even derived from their own voice.<p>Which will give entirely new meaning to &#x27;lip synching&#x27;.

A fun thing to do with this would be to slurp the lyrics from one song - the beats from another, some other stream from another and remix the “threads” together into something new.<p>Basically a giant equalizer that allows you to dim or brighten each channel from multiple sources.

This project I&#x27;ve found to be very useful if you want access to something that like what the article describes. <a href="http:&#x2F;&#x2F;isse.sourceforge.net" rel="nofollow">http:&#x2F;&#x2F;isse.sourceforge.net</a>

I&#x27;d like to try using this kinda thing to build an automated beat saber map. The ability to orchestrate the beats very specifically would make for excellent mappings.<p>Alas so many projects, too little time!

Sounds pretty good but exhibits the same artifacts&#x2F;phasing that I&#x27;ve heard with other source separation. Good for forensics etc but I wouldn&#x27;t use this for music production

There was a similar demo (I think from Google) here on HN sometime last year that was far more impressive. I can&#x27;t seem to find it though. Anybody know what it was?

Are there any hearing aid manufacturers taking this approach? Quite incredible.