Hi HN, author here. A few comments on how I came up with this idea. I've been trying to find a "proper" connection between audible sound and visible shape, a connection that would not only preserve all the information, but would also properly visualize the "symmetry" in sound, so that messy sound would turn into messy images and harmonic sound would turn into visually appealing images. The latter part is hard, as perception of "musical harmony" is vaguely defined and subjective. Nevertheless, after quite a few attempts, I came across a particularly simple FFT-based technique that produces impressive and unexpected results. Below is the summary of my finding.<p>Music is a temporal ornament. There are many types of ornaments, e.g. the 17 types of wallpaper tesselations, but few of them look like music. However there is one particular type of ornament that resembles music a lot - I mean those “mandala” images. I don’t know how those are produced, but I noticed a connection between those images and music:<p>- The 1st obvious observation is that a mandala is drawn in polar coordinates and is 2PI periodic. Sound is periodic too, so I thought the two facts are related.<p>- The 2nd observation is that patterns on those images evolve over the radial axis. Ans so is music is a sequence of evolving sound patterns.<p>- The 3rd observation is that a 2PI periodic function trivially corresponds to a set of frequencies. We usually use FFT to extract the frequencies and another FFT to restore the 2PI periodic function. Thus, a single radial slice of a mandala could encode a set of frequencies. If this is correct, a mandala is effectively an old school vinyl disk.<p>Putting these observations together, we naturally arrive with the idea of using ACF. More details in the linked github project.
I'm very interested in this, but I can't seem to get the demo page working in latest Firefox:<p>> AudioContext.createMediaStreamSource: Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported.<p>Edge worked though. Didn't try Chrome.<p>Edit: This is beautiful!<p>I have checked out a variety of songs, and I feel the visualization is rather dominated by whatever frequency is loudest. (E.g. bass sound -> three- to five-fold symmetry and most detail obscured by it).<p>Have you considered applying something like the <a href="https://en.wikipedia.org/wiki/Equal-loudness_contour" rel="nofollow">https://en.wikipedia.org/wiki/Equal-loudness_contour</a> somewhere in the process to more evenly weight the frequency contributions according to human hearing perception? Not sure if would have the intended effect, but I'd be curious what happens.
Autocorrelation has been used in psychoacoustics at least since Licklider's work in the 1950s. But I'm not sure if I've seen this style of visualization before. It looks a bit like the output of a strobe tuner.
I've been playing some music on the livedemo. I like it a lot. The harmony changes are easy to see as concentric slices slighty rotated. On heavy drum music the rythm also leaves a trail of spaced circles easy to recognize. You might be onto something here. Colored waveforms, like those used in djing software, might be better for locating a point in a song, but I've never been able to locate a chord change on those based on the waveform representation alone. I would totally love to try to "mix" some of those mandalas, or try an editor software using this visualization where I can move around a virtual needle, copy paste slices, etc. Not saying It will be better than using a waveform, but I would experiment with it given the chance.<p>Very nice job, congratulations!
Seems to work really well! Playing Gasoline by Audioslave, it really misses the high intensity peaks.<p>It interprets the trumpet in Miles Davis - It Never Entered my Mind as dark ripples and is beautiful in it's own way.<p>It's a long way from the last time I used a visualizer on Winamp, nice!
Is this somehow related or similar to oscilloscope music?
<a href="https://youtu.be/qnL40CbuodU" rel="nofollow">https://youtu.be/qnL40CbuodU</a>
Interesting maths. I'm enjoying the graphics, they are beautiful to watch unfold, but bear little relation to the full range that is heard in the audio examples. Not in volume, or varying pitch. This technique seems to be a way of showing an aspect of sound via graphics that mean the two are unrelated in immediately meaningful ways. The word 'Periodic' is mentioned, but that isn't seen.<p>Still, great experiment, and interesting results!
Very nice. Curious how colors are chosen?<p>Can’t try it on my dated iPhone - you need to vendor prefix the AudioContext with webkitAudioContext if AudioContext is undefined.
Somehow I have memories of the 1990s and audio visualisations coming back... Fantastic. I'd love to be able to plug this into mpv for playing audio files. I wonder how hard this would be (or how many resources it would use...).<p>I really enjoyed your work: the write-up was clear and the demo page worked well.
Very great work.
I'm also interested to see this patterns "unrolled".
Not on a circle but on timeline (like spectrogram). Where x-axis is time and y-axis is unrolled pattern (0..2pi).