Scientists Reconstruct Rough Images of Movie Clips People are Watching From Brain Scans

Posted on September 25, 2011

Scientists from the University of California, Berkeley, have used brain scans to reconstruct rough computer reconstructions of videos that people are watching. Here is a brief explanation of the experiment:

The goal of the experiment was to design a process for decoding dynamic natural visual experiences from human visual cortex. More specifically, we sought to use brain activity measurements to reconstruct natural movies seen by an observer. First, we used functional magnetic resonance imaging (fMRI) to measure brain activity in visual cortex as a person looked at several hours of movies. We then used these data to develop computational models that could predict the pattern of brain activity that would be elicited by any arbitrary movies (i.e., movies that were not in the initial set used to build the model). Next, we used fMRI to measure brain activity elicited by a second set of movies that were completely distinct from the first set. Finally, we used the computational models to process the elicited brain activity, in order to reconstruct the movies in the second set of movies. This is the first demonstration that dynamic natural visual experiences can be recovered from very slow brain activity recorded by fMRI.

The reconstructed images partially resemble the video people are watching while their brains are being scanned. The researchers say brain activity was sampled every one second, and each one-second section of the viewed movie was reconstructed separately. Take a look:

Here is another YouTube video that shows reconstructions from brain activity in three subjects:

In an interesting FAQ here the researchers explain what the future applications could be for the technology.

In addition to their value as a basic research tool, brain-reading devices could be used to aid in diagnosis of diseases (e.g., stroke, dementia); to assess the effects of therapeutic interventions (drug therapy, stem cell therapy); or as the computational heart of a neural prosthesis. They could also be used to build a brain-machine interface.

The FAQ also addresses whether this kind of technology (if refined) could be used in detective work and in court cases.

The potential use of this technology in the legal system is questionable. Many psychology studies have now demonstrated that eyewitness testimony is notoriously unreliable. Witnesses often have poor memory, but are usually unaware of this. Memory tends to be biased by intervening events, inadvertent coaching, and rehearsal (prior recall). Eyewitnesses often confabulate stories to make logical sense of events that they cannot recall well. These errors are thought to stem from several factors: poor initial storage of information in memory; changes to stored memories over time; and faulty recall. Any brain-reading device that aims to decode stored memories will inevitably be limited not only by the technology itself, but also by the quality of the stored information. After all, an accurate read-out of a faulty memory only provides misleading information. Therefore, any future application of this technology in the legal system will have to be approached with extreme caution.

The research paper was published here in Current Biology.

THE MIND
PSYCHOLOGY