Disney is developing a mixed reality (MR) prototype called the Magic Bench, which is headset-free. The system, created by Disney Research, allows people to sit on a bench and interact with animated characters with haptic feedback.
The system works when a user sits on a bench opposite a large screen with an RGB camera and Microsoft Kinect system in place. The camera and Kinect work together to create a real-time, HD, 3D reconstruction of the bench, surroundings, and participants.
Underneath the bench are hidden haptic sensors which play noises/sounds in conjunction with actions on the screen ie a character snoring.
To eliminate shadows that occur in areas where the depth sensor has no corresponding line of sight with the camera, a modified algorithm creates a 2D backdrop. The 3D and 2D reconstructions are positioned in virtual space and populated with 3D characters and effects in such a way that the resulting real-time rendering is a seamless composite, fully capable of interacting with virtual physics, light, and shadows.
"This platform creates a multi-sensory immersive experience in which a group can interact directly with an animated character," says Moshe Mahler, principal digital artist at Disney Research. "Our mantra for this project was: hear a character coming, see them enter the space, and feel them sit next to you."
"The bench itself plays a critical role. Not only does it contain haptic actuators, but it constrains several issues for us in an elegant way. We know the location and the number of participants, and can infer their gaze. It creates a stage with a foreground and a background, with the seated participants in the middle ground. It even serves as a controller; the mixed reality experience doesn't begin until someone sits down and different formations of people seated create different types of experiences."
In a post on the Disney Research website, it adds; 'Our approach is to instrument the environment leaving the user unburdened of any equipment, creating a seamless walk-up-and-play experience. We demonstrate this technology in a series of vignettes featuring humanoid animals. Participants can not only see and hear these characters, they can also feel them on the bench through haptic feedback.'