They create an implant so that people with paralysis can communicate through an avatar

BarcelonaWaving with your hand, shrugging your shoulders as a sign of doubt, or nodding your head are some of the body gestures that underpin human communication, which by nature combines speech with these types of movements. People who suffer a cardiovascular accident, such as a stroke, or a neurodegenerative disease, such as amyotrophic lateral sclerosis (ALS), have less ability to make themselves understood, as the resulting paralysis alters both speech and body movement. Now a team of researchers at the University of California, San Francisco (UCSF) has created a brain implant that is capable of decoding the speech and gestures of people with severe paralysis and channels them through an animated digital avatar.

The journal Nature Neuroscience publishes the results of this study on a new system that connects the brain to the computer. The researchers' goal is to move towards more natural communication for people with paralysis who have difficulty making themselves understood. Until now, technologies that seek to restore communication for people with severe motor paralysis were limited to decoding speech or limb movement in isolation. This new system manages to integrate both modalities at the same time, which allows for the recovery of non-verbal communication that is "more expressive and natural," according to the authors.

The system uses a single high-density electrode implant (ECoG) which they have tested on three people with severe paralysis who could no longer make themselves understood. These are two people affected by a stroke and a third with ALS. The authors first demonstrated that with a single implant they could capture signals linked to various types of movement, including movements of the upper extremities and movements of the mouth and face related to speech, such as shrugging the shoulders, raising a fist in a sign of victory, or nodding the head.

Then the authors used parallel speech and gesture decoders in two of the participants to animate a full-body avatar: the participants were asked to try to make specific gestures, to respond to conversational questions, or both. In the conversation tasks, the system accurately decoded speech and gestures in both participants. One, in fact, achieved an average accuracy of 100% in both speech and gesture decoding throughout three conversation blocks.

AI Model

The researchers found that, although the brain areas responsible for speech and hand movement have a separate general organization, there is a partial overlap in neural activity. To avoid interference between the two signals, the team designed an artificial intelligence model capable of processing both activities independently and simultaneously. According to the authors, this work represents a decisive step towards the development of neurotechnology tools that restore not only the voice, but also the capacity for gestural and emotional expression to patients deprived of mobility and speech.