A man with a severe speech disability finally spoke with expression and sang with a brain implant that instantly converts neural activity into words. The device translates thought into speech, allowing tone changes when he asks questions, emphasizes the words the speaker wants, and hums a string of notes in three different pitches.
The system is called a brain-computer interface (BCI), and it uses AI to translate the participant’s electrical brain activity as he attempts to speak. The device is the first not just to translate the intended words, but also features parts of natural speech like tone, pitch, and emphasis, which helps understand emotion and meaning.
In the study, a voice mechanism yielded a synthetic voice that mimicked that of the user in just 10 milliseconds of the neural activity that indicated his intention to speak. This system marks a significant improvement over the previous BCI models, which produced speech within three seconds or produced it only after users finished miming an entire sentence. According to Christian Herff, a computational neuroscientist at Maastricht University, the Netherlands, who was not involved in the study, this is the holy grail of speech BCIs because it allows fluent, continuous speech.
The study participant, a 45-year-old man, lost his ability to speak clearly and fluently after having been diagnosed with amyotrophic lateral sclerosis, a form of motor neuron disease. This disease damages the nerves that control muscle movement, involving movements necessary for speech. His speech was unclear and slow; however, he could mouth words and make sounds.
The participant underwent surgery to insert 256 silicon electrodes, five years after his symptoms began, each 1.5 mm long, in a brain region that controls movement. Study co-author Maitreyee Wairagkar, a neuroscientist at the University of California, Davis, and her colleagues trained deep learning algorithms to record the brain signals every 10 milliseconds. Their system decodes the sounds the man wants to produce in real time, rather than his intended words or the constituent phonemes — the speech subunits that form spoken words.
“We don’t always use words to communicate what we want. We have interjections. We have other expressive vocalizations that are not in the vocabulary,” explains Wairagkar. “In order to do that, we have adopted this approach, which is completely unrestricted.”
The synthetic voice of the device was also personalized to match that of the participant through training the AI model with a voice taken from previous interviews with the man before the onset of the disease. The team asked the participant to produce made-up words and interjections such as ‘aah’, ‘ooh’, and ‘hmm’, the sounds of which the BCI successfully produced, without needing a fixed vocabulary.
Using the device, the participant responded to open-ended questions, spelled out words, and said whatever he wanted. Listening to the synthetic voice made the participant feel happy, and the voice felt like his own.
The BCI identified whether a sentence was spoken as a question or a statement. The system could also determine which words were being stressed by the user and adjust the tonality using the input. Previous BCIs could only produce a monotonous voice with a flat tone.
According to Silvia Marchesotti, a neuroengineer at the University of Geneva in Switzerland, the device can become crucial for adoption for daily usage by patients in the future.
Copyright @smorescience. All rights reserved. Do not copy, cite, publish, or distribute this content without permission.
