MindEye2: Shared-Subject Models Enable fMRI-to-Image with 1 Hour of Data

From the article,<p>&gt; The present work demonstrates that it is now practical for patients to undergo a single MRI scanning session and produce enough data to perform high-quality reconstructions of their visual perception.<p>&gt; Such image reconstructions from brain activity are expected to be systematically distorted due to factors including mental state, neurological conditions, etc.<p>&gt; This could potentially enable novel clinical diagnosis and assessment approaches, including applications for improved locked-in (pseudocoma) patient communication (Monti et al., 2010) and brain-computer interfaces if adapted to real-time analysis (Wallace et al., 2022) or non-fMRI neu-roimaging modalities.<p>&gt; As technology continues to improve, we note it is important that brain data be carefully protected and companies collecting such data be transparent with their use.

While decoding dreams, as some here suggested, would certainly be an intriguing venue of research, I believe, decoding the minds eye of people who lack the minds eye would be an even more interesting topic. How accurate are their &quot;view&quot;, for a lack of a better word? Would there even be an image to capture?

The fact that this is possible is so bizarre to me. Impressive work.

What about doing the opposite: get an image and stimulate the brain to perceive it ?

Looking at the result (figure 4), the model looks more like a good classifier than an image retriever, the generated images have the same subject as the original but are very different, even the colors of the subjects.

Why does anyone think this type of scientific inquiry should be pursued at all.

I hope someone will do a similar experiment with visualization (imagination) only. Some people are better at this than others so maybe focus on those people who easily create vivid images in their mind&#x27;s eye.<p>Or maybe text.

It&#x27;s remarkable that this works, and it&#x27;s equally remarkable that the authors succeeded using a (deliberately) naive approach with the fMRI data. I was just teaching the visual system to students over the last few weeks: cognitive neuroscience&#x2F;neuropsychology can tell us a lot about what brain systems should be involved here. It&#x27;s such a tempting research domain to dive into using a more sophisticated approach, but then again, maybe the authors&#x27; simpler data-driven method is a strength.

So here is a question: what would happen if this process faced a person who was specifically hostile to it? Could you &quot;trick&quot; the process by intensely thinking of, for instance, a yellow ball while viewing a cat picture, as one example?

This is insane! If capture latency is reduced, can this be used to visualize dreams?

As someone with damage to my visual cortex with a very specific effect I’m very curious what this would come up with for me. Not something useful on its own, but more understanding means more possibilities for treatments.

Very cool work. Can something similar be done for audio?

very exciting tech. also very scary, if we imagine how authority can misuse this.

Impressive