Illustration by Adobe Stock
Adults with autism are working with engineers, computer scientists and psychologists at Vanderbilt University and Vanderbilt Health to design an augmented reality platform, populated by virtual avatars, which can strengthen their connection to each other and the world.
The participant-informed research, which was featured in April at an international Conference on Human Factors in Computing Systems in Barcelona, Spain, is at the forefront of efforts to help people with autism lead more fulfilling, happier lives.
Many adults with autism become socially isolated, lonely and depressed. In-person interactions can be stressful, in part because of differences in communication styles or sensory sensitivities.
But playing games using life-size avatars, or virtual representations of themselves, in an environment designed to be friendly to the needs of people with autism can make social interaction more comfortable. Augmented reality also can overcome distance and transportation challenges, such as difficulty with driving, that otherwise would prevent autistic people from meeting others in person.
The goal is to design a platform from which — like science fiction — adults with autism can transport realistic, three-dimensional avatars of themselves to another person’s space, “as if they were sitting in front of each other,” said Nilanjan Sarkar, PhD, who led the research.
Sarkar, Professor of Mechanical Engineering, holds the David K. Wilson Chair and is Director of the Robotics and Autonomous Systems Laboratory in the Vanderbilt University School of Engineering.
Leveraging a pilot grant from the Vanderbilt Kennedy Center, since 2010 Sarkar has received $4.8 million in grants from the National Institute of Mental Health to develop adaptive response and co-robotic technologies, and virtual intelligent systems for people with autism.
He also has designed adaptive technologies for people with schizophrenia, dementia, and those recovering from a stroke to help them learn new skills, function better and connect with others.
“This ability to transport somebody to a different place, to be together … has implications beyond autism,” Sarkar said during a recent Conversation on Research in Intellectual and Developmental Disabilities hosted by the Kennedy Center.
In the future, avatars could enable colleagues in remote locations to work together on a project in a real conference room, in real time.
“The whole future of work can be influenced by the ability to do that,” he said. “Of course, it’s not that simple.”
Designing useful adaptive technology is especially challenging for the more than 5.4 million adults in the United States who identify as being on the autism spectrum.
“Many autistic people have a strong autistic identity,” said Amy Weitlauf, PhD, Associate Professor of Pediatrics at Vanderbilt Health, who co-authored the paper. “It’s how they experience the world, but the world is not always set up in accessible ways for them.”
Communication and social skills among this group vary widely. Some people with autism have difficulty understanding or responding to the cues of their neurotypical peers (those without autism). Even when people understand those cues, masking or hiding their autistic selves can be tiring and isolating.
Building and maintaining social relationships can also be an important part of finding fulfilling work. Fortunately, as the understanding of individuals with autism has advanced rapidly, so has the development of adaptive technology that can facilitate communication and connection.
“Autistic people want social connections in spaces and ways that are comfortable and affirming.”
Amy Weitlauf, PhD
Two-dimensional, video-based platforms like Zoom and FaceTime do not allow autistic individuals to filter out the stress-inducing environmental stimuli around them or achieve a realistic sense of presence. More appropriate may be augmented reality headsets and glasses, which use sensors, motion-sensing cameras and a processing unit to layer virtual imagery over real-world settings.
Another technological innovation, the MetaHuman software developed by Epic Games, merges virtual reality, human animation and video analysis techniques to create very realistic avatars.
A realistic setting
Sarkar has found that augmented reality telepresence is more acceptable to individuals with autism than virtual reality. “With virtual reality, everything is artificial,” he said. “Augmented reality blends the real with the virtual … It’s much closer to real interaction.”
As an example, aspiring chefs can retain headset-projected instructions better if they are set against a realistic setting.
“Imagine trying to learn to cook a new dish,” added Weitlauf. “Having augmented support to cook that dish in my own kitchen instead of a virtual one … I’m more likely to be able to do it again.”
The study reported in April recruited eight adults with autism through the Kennedy Center’s Treatment and Research Institute for Autism Spectrum Disorders (TRIAD). Weitlauf, who received her PhD in Clinical Psychology from Vanderbilt, is the institute’s Associate Director for Research.
Participants were invited to design their own avatars, which brought its own revelations, explained Mahrukh Tauseef, a graduate student in Electrical and Computer Engineering and the paper’s first author.
Not only did they want their avatar to look younger (no surprise), but they went beyond a mere photographic resemblance to build in features they felt were more “authentic” and representative of their true selves. Participants then helped design two interactive scenarios: a two-person game of checkers and a game of Apples to Apples, a popular, matching card game played by four avatars seated in a circle.
“It’s not about playing cards,” Tauseef said. “This is their medium. They engage like this with each other … We wanted to create an environment where it’s light; it’s playful. That’s a friendly way of connecting with others.”
Playing checkers proved to be difficult for the avatars. So, the engineers and computer scientists programmed the checker pieces to move when the avatar’s hand hovered above them.
The researchers found that participants were more engaged when playing checkers one-on-one. “There were too many people (in the card-playing group) to look at in one space,” Tauseef said. “The headsets and glasses we were using had a very narrow field of view.”
Anxiety and sensory overload also may have played a role. “Many autistic people are still very lonely in a group if there are aspects of the social interaction that are stressful,” Weitlauf said. “This system is an important step toward creating autism-friendly virtual spaces that promote comfort and inclusion.”
“Our findings,” the researchers concluded, “underscore the potential of augmented reality telepresence as an … affirming communication medium that can support authentic self-expression and foster meaningful social presence, particularly in contexts where in-person interactions are limited.”
Vanderbilt co-authors included Zachary Warren, PhD, who holds the Donna and Jeffrey Eskind Family Chair in Autism Spectrum Disorder and who has worked with Sarkar for more than 20 years; Alexandra Watkins, PhD, postdoctoral fellow in Mechanical Engineering; software engineer Zalen Ingram; and graduate students Rendong Zhang and Ritam Ghosh.
The technology continues to advance. Sarkar’s lab is investigating affective computing, which combines cameras and microphones to detect changes in facial expression and tone of voice, eye movement trackers, and wearable heart rate monitors — all to help people with autism and their avatars read the emotions of others.
Also under development: haptic devices that apply force feedback on the hand, enabling users to “feel” the three-dimensional virtual objects their avatars are manipulating.
None of these developments are meant to replace real physical connection. Avatars will be most effective, Weitlauf said, when they work hand in hand with the humans who designed them and the humans who use them.