Scientists at the University of Texas at Austin have developed a non-invasive AI system called a semantic decoder that can translate a person’s brain activity into a stream of text. The system has the potential to aid patients who have lost their ability to communicate due to degenerative diseases, paralysis or a stroke.
The study involved participants listening to several hours of podcasts while in an fMRI scanner that measures brain activity. Once trained, the AI system can generate a stream of text when the participant listens to or imagines telling a new story. The resultant text captures general thoughts or ideas rather than an exact transcript.
The trained system was able to produce text that matched the intended meaning of the participant’s words about half of the time. For instance, when a participant heard the words “I don’t have my driver’s license yet,” the thoughts were translated to “She has not even started to learn to drive yet.”
The researchers noted that this is a significant improvement over previous methods that typically only translate single words or short sentences. The AI system was also able to accurately describe “certain events” from videos watched by participants without audio.
Currently, the system requires an fMRI scanner, limiting its use to laboratory settings. However, the researchers believe that it could eventually be used via more portable brain-imaging systems.
The technology has the potential to aid people who have lost their ability to communicate due to degenerative diseases or other neurological conditions. The researchers have filed a PCT patent application for the technology.
“This is a real leap forward,” said Alexander Huth, one of the study’s leaders. “We’re getting the model to decode continuous language for extended periods of time with complicated ideas. For a non-invasive method, this is a real leap forward compared to what’s been done before, which is typically single words or short sentences.”
