Tapping into the brain to help a paralysed man speak
He has not been able to speak since 2003, when he was paralysed aged 20 by a severe stroke after a terrible car crash.
Now, in a scientific milestone, researchers have tapped into the speech areas of his brain — allowing him to produce comprehensible words and sentences simply by trying to say them. When the man — known as Pancho — tries to speak, electrodes implanted in his brain transmit signals to a computer that displays them on the screen.
Dr Eddie Chang, a neurosurgeon at the University of California, San Francisco Medical School helps Pancho, paralysed since age 20, speak through an implant in his brain that connects to a computer program.Credit:New York Times
His first recognisable sentence, researchers said, was “my family is outside”.
The achievement, published on Wednesday in the New England Journal of Medicine, could eventually help many patients with conditions that steal their ability to talk.
“This is farther than we’ve ever imagined we could go,” said Melanie Fried-Oken, a professor of neurology and paediatrics at Oregon Health and Science University, who was not involved in the project.
Three years ago, when Pancho, now 38, agreed to work with neuroscience researchers, they were unsure if his brain had even retained the mechanisms for speech.
“That part of his brain might have been dormant, and we just didn’t know if it would ever really wake up in order for him to speak again,” said Dr Edward Chang, chairman of neurological surgery at University of California, San Francisco, who led the research.
Dr Chang prepares to connect Pancho’s brain to a computer, which uses artificial intelligence to recognise the words he intends to say.Credit:New York Times
The team implanted a rectangular sheet of 128 electrodes, designed to detect signals from speech-related sensory and motor processes linked to the mouth, lips, jaw, tongue and larynx. In 50 sessions over 81 weeks, they connected the implant to a computer by a cable attached to a port in Pancho’s head and asked him to try to say words from a list of 50 common ones he helped suggest, including “hungry”, “music” and “computer”.
As he did, electrodes transmitted signals through a form of artificial intelligence that tried to recognise the intended words.
“Our system translates the brain activity that would have normally controlled his vocal tract directly into words and sentences,” said David Moses, a postdoctoral engineer who developed the system with graduate students Sean Metzger and Jessie Liu. The three are lead authors of the study.
Pancho (who asked to be identified only by his nickname to protect his privacy) also tried to say the 50 words in 50 distinct sentences like “My nurse is right outside” and “Bring my glasses, please” and to respond to questions like “How are you today?”
His answer, displayed on a screen: “I am very good.”
In nearly half of the 9000 times Pancho tried to say single words, the algorithm got it right. When he tried saying sentences written on the screen, it did even better.
By funnelling algorithm results through a kind of autocorrect language-prediction system, the computer correctly recognised individual words in the sentences nearly three-quarters of the time and perfectly decoded entire sentences more than half the time.
“To prove that you can decipher speech from the electrical signals in the speech motor area of your brain is groundbreaking,” said Fried-Oken, whose own research involves trying to detect signals using electrodes in a cap placed on the head, not implanted.
After a recent session, Pancho, wearing a black fedora over a white knit hat to cover the port, smiled and tilted his head slightly with the limited movement he has. In bursts of gravelly sound, he demonstrated a sentence composed of words in the study: “No, I am not thirsty.”
In interviews over several weeks for this article, he communicated through email exchanges using a head-controlled mouse to painstakingly type key-by-key, the method he usually relies on.
On a video call, Pancho communicates using a painstaking method involving a head-controlled mouse that he currently uses to type out letters one by one.Credit:New York Times
The brain implant’s recognition of his spoken words is “a life-changing experience”, he said.
“I just want to, I don’t know, get something good, because I always was told by doctors that I had zero chance to get better,” Pancho typed during a video chat from the northern California nursing home where he lives.
Later, he emailed: “Not to be able to communicate with anyone, to have a normal conversation and express yourself in any way, it’s devastating, very hard to live with.”
During research sessions with the electrodes, he wrote: “It’s very much like getting a second chance to talk again.”
Pancho had been a healthy field worker in California’s vineyards until a car crash after a soccer game one summer Sunday, he said. After surgery for serious damage to his stomach, he was discharged from the hospital, walking, talking and thinking he was on the road to recovery.
But the next morning, he was “throwing up and unable to hold myself up”, he wrote. Doctors said he experienced a brainstem stroke, apparently caused by a post-surgery blood clot.
A week later, he woke up from a coma in a small, dark room. “I tried to move, but I couldn’t lift a finger, and I tried to talk, but I couldn’t spit out a word,” he wrote. “So I started to cry, but as I couldn’t make any sound, all I made were some ugly gestures.”
It was terrifying. “I wished I didn’t ever come back from the coma I was in,” he wrote.
The new approach, called a “speech neuroprosthesis”, is part of a surge of innovation aimed at helping tens of thousands of people who lack the ability to talk but whose brains contain neural pathways for speech, said Dr Leigh Hochberg, a neurologist with Massachusetts General Hospital, Brown University and the Department of Veterans Affairs, who was not involved in the study but co-wrote an editorial about it.
That could include people with brain injuries or conditions like amyotrophic lateral sclerosis (also known as Lou Gehrig’s disease) or cerebral palsy, in which patients have insufficient muscle control to speak.
“The urgency can’t be overstated,” said Hochberg, who directs a project called BrainGate that implants tinier electrodes to read signals from individual neurons; it recently decoded a paralysed patient’s attempted handwriting motions.
Researchers emphasise that the electrodes are not reading Pancho’s mind but detecting brain signals corresponding to each word.
“It’s now only a matter of years,” he said, “before there will be a clinically useful system that will allow for the restoration of communication”.
Pancho’s buoyant personality has helped the researchers navigate challenges, but also occasionally makes speech recognition uneven.
“I sometimes can’t control my emotions and laugh a lot and don’t do too good with the experiment,” he emailed.
Chang recalled times when, after the algorithm successfully identified a sentence, “you could see him visibly shaking and it looked like he was kind of giggling”. When that happened or when, during the repetitive tasks, he’d yawn or get distracted, “it didn’t work very well because he wasn’t really focused on getting those words. So we’ve got some things to work on because we obviously want it to work all the time.”
The algorithm sometimes confused words with similar phonetics, identifying “going” as “bring”, “do” as “you” and words beginning with “F” — “faith”, “family”, “feel” — as a V-word, “very”.
Longer sentences needed more help from the language-prediction system. Without it, “How do you like my music?” was decoded as “How do you like bad bring?” and “Hello how are you?” became “Hungry how am you?”
But in sessions that the pandemic interrupted for months, accuracy improved, Chang said, both because the algorithm learned from Pancho’s efforts and because “there’s definitely things that are changing in his brain”, helping it “light up and show us the signals that we needed to get these words out”.
Pancho shops at a farmers’ market in Sonoma County. Credit:New York Times
Before his stroke, Pancho had attended school only up to sixth grade in his native Mexico. With remarkable determination, he has since earned a high school diploma, taken college classes, received a web developer certificate and begun studying French.
“I think the car wreck got me to be a better person, and smarter too,” he emailed.
With his restricted wrist movement, Pancho can manoeuvre an electric wheelchair, pressing the joystick with a stuffed sock tied around his hand with rubber bands. At stores, he’ll hover near something until cashiers decipher what he wants, like a cup of coffee.
“They place it in my wheelchair and I bring it back to my home so I can get help drinking it,” he said.
The speech study is the culmination of more than a decade of research, in which Chang’s team mapped brain activity for all vowel and consonant sounds and tapped into the brains of healthy people to produce computerised speech.
Researchers emphasise that the electrodes are not reading Pancho’s mind but detecting brain signals corresponding to each word he tries to say.
“He is thinking the word,” Fried-Oken said. “It’s not random thoughts that the computer is picking up.”
Chang said: “In the future, we might be able to do what people are thinking [which raises] some really important questions about the ethics of this kind of technology.” But this, he said, “is really just about restoring the individual’s voice”.
In newer tasks, Pancho mimes words silently and spells out less common words using the military alphabet: delta for D, foxtrot for F.
“He is truly a pioneer,” Moses said.
The team also wants to engineer implants with more sensitivity and make it wireless for complete implantation to avoid infection, said Chang.
As more patients participate, scientists might find individual brain variations, Fried-Oken said, adding that if patients were tired or ill, the intensity or timing of their brain signals might change.
“I just wanted to somehow be able to do something for myself, even a tiny bit,” Pancho said, “But now I know I’m not doing it just for myself.”
The New York Times
Most Viewed in World
From our partners
Source: Read Full Article