This touches on something I've thought about for years, which is what I refer to as "synesthetic computing" for lack of a better term. It's the idea that as humans, we are innately endowed with these <i>incredible</i> abilities that no computer can possibly match, like our ability to understand social situations and hierarchies, and our ability to process and understand complex emotions. We could potentially unlock whole new models of computation by leveraging them. Imagine a data visualization environment where instead of just looking at rows of spreadsheet data, each data point had a "personality", and was allowed to form "social groups" autonomously with other data points. Those social groups could then be analyzed in the same way we do with actual human social interactions. Their reasons for grouping could be looked at emotionally and understood through intuition. And those intuitions could even be fed into machine learning models as a heuristic for hyperparameter optimization.
Similar idea in neuroscience - listen to neural spikes during task/training/experiment to help troubleshoot experiment setup and get a secondary source of info before visualizing data[0].<p>0. Sonifying and Visualizing Neural Data <a href="https://ccrma.stanford.edu/~mindyc/256a/final/" rel="nofollow">https://ccrma.stanford.edu/~mindyc/256a/final/</a>
Back in the day, I'd tee a supposedly quiet log file to /dev/snd0 on my workstation so I could take a nap under the desk and be sure to be woken up when the particles hit the fabric, so to speak..
At the National Museum of Computing in the UK (<a href="https://www.tnmoc.org/" rel="nofollow">https://www.tnmoc.org/</a>) I saw an old minicomputer where, if I recall correctly, the low order bit(s?) of the program counter had been hooked up to a speaker for some very simple auditory debugging.<p>My memory is that e.g. infinite loops would be easy to identify as a persistent tone, etc as repeated small changes to the program counter happened.
This isn't too far from using a logic probe to test hardware circuit. It has a tone for LOW and a tone for HIGH, and you can listen to what state pins are in, even if they're switching faster than your eyes can see the LED flicker, you can hear the pitch changes.
It has been around for a while as far as I know, I was introduced to the concept back in the '80s. It was called "cat belling." Memory leaks and memory pressure, frame rate drops (it was especially useful for that), texture loads & swaps, game AI processing. We used it for determing if the game was executing correctly by how often it passed certain check points in the code. Also tracking the number of sprites in play vs the number on-screen. You could listen for the rhythm of the various sections of the code, in what was to most people, a cacophony of noise.<p>After thought: I should probably try and work some of that concept in to my next music track release.
There was a rumor about a place that configured OS alert sounds as orchestra instruments. So that as folks were working it would sound like an orchestra tuning up. Instead of lots of random alert tones.
I did similar experiments longbtine ago, but wasn't satisfied with results. Still, I believe this idea has great future if done right.<p>One principle I was experimenting with was "related events should create related sounds." A particular case is hierarchical events, like parent-child operations or requests, that should create "hierarchical sounds" that are octaves apart. For example, the parent request produces note A4, so its child request should be A5. Some cleverness needed to express different types of requests with different, but related sounds. Errors should be clearly audible too.<p>Real life applications need to work with huge amounts of data: so you could hear a lot in a few seconds. I recommend you to watch a movie about a airplane company investigating a crash by listening to recorded noises: a professional listener gets to hear a lot of insights, and gets subsequently killed for that (forgot the movie name).<p>Also, think of a huge file with logs: requests, errors, and all sorts of garbage in it. A few millions of records. How do you hear it in 1 minute and get something useful out of it?<p>Another association is dial-up modems from the 90s: those would take minutes to connect to the internet, "logging" their progress to speakers, and with some experience, you could distinguish diffrent stages of connection or errors.
The article says each operation gets a unique note, but I wonder how the notes were selected? It would be interesting to apply some musical theory to the process so that, for example, operations which normally run together form pleasing cords, while warning or errors appear more discordant. One could also experiment with using different voices/instruments in addition to plain notes.
Love the idea of bringing the other senses into help. Check out ye olden Peep [0]. I can’t imagine what it would sound like now with modern volumes of logging.<p>0 - <a href="https://sourceforge.net/projects/peep/" rel="nofollow">https://sourceforge.net/projects/peep/</a>
And
<a href="https://www.usenix.org/legacy/publications/library/proceedings/lisa2000/full_papers/gilfix/gilfix_html/index.html" rel="nofollow">https://www.usenix.org/legacy/publications/library/proceedin...</a>
came here for 'lights on the cm2' and was disappointed. you could in fact indentify the application and stage of computation by the patterns. it was sometimes useful in debugging, not that you got any real information, but if it didn't look right you might was well stop and look further instead of waiting 20 minutes for it to finish.