An accessible, voice-controlled digital hand has been developed by researchers at the University of Michigan, designed to improve the convenience and accessibility of VR systems.
The software, called HandProxy, allows VR and AR users to interact with digital spaces by commanding a disembodied hand. This hand can grab and move virtual objects, drag and resize windows, and perform gestures such as a thumbs up.
The system has been designed to tackle complex problems in VR, with tasks such as “clear the table” completed without precise instructions for each object on the virtual table by using GPT-4o, the AI model behind ChatGPT.
The researchers claim that the hand’s ability to independently complete tasks on the fly which the researchers say makes it more flexible than current VR voice command features.
Anhong Guo, Morris Wellman faculty development assistant professor of computer science and engineering, University of Michigan, commented: “Mobile devices have supported assistive technologies that enable alternative input modes and automated user-interface control, including AI-powered task assistants like Siri. But such capabilities are largely absent in VR and AR hand interactions.
“HandProxy is our attempt to enable users to fluidly transition between multiple modes of interaction in virtual and augmented reality, including controllers, hand gestures, and speech,”