UI grad student's video system merges virtual, reality
URBANA — It's "snowing" in your living room, and the flakes have started to settle on your coffee table.
And there goes a "grenade" rolling under the couch.
In IllumiRoom, a new technology developed at Microsoft Research by a four-person team that included a University of Illinois graduate student, the virtual and real world merge.
"It takes the game out of the TV and into your living room," said Brett Jones, a UI computer science Ph.D. student who earned bachelor's and master's degrees from the UI.
Jones recently returned to campus after presenting a paper with his fellow researchers in Paris at a conference about the intersection between humans and computers called the Association for Computer Machinery Special Interest Group on Computer-Human Interaction Conference. His team garnered a best paper award for their work.
While interning at Microsoft Research in Washington last summer, Jones and Microsoft employees Hrvoje Benko, Eyal Ofek and Andy Wilson asked the question: Can we bring a video game into the room?
The area of research called "augmented reality" is about taking the physical world and making it appear virtual, Jones explained. For example, making a bookshelf in the room appear to look like a cartoon version of the shelf.
In a span of three months, the group conceived, designed and implemented the project.
It works with a standard projector and a Kinect sensor (a motion-sensing device that works with Mircosoft's Xbox game console). A color picture and a 3-D scan of the room are taken, allowing the device to know colors and recognize distances and shapes. That allows for lots of cool things to happen.
For example, "We can then write the math for a ball bouncing on the table," Jones said.
IllumiRoom can perform several different tasks. On a basic level, it can take the game and make it bigger, essentially expanding what viewers or players see in the game or TV to beyond the TV. While watching a hockey game, for example, the viewer can display the scoreboard on the wall behind the television. The device can produce what Jones called a "radial wobble" which can make it seem as if the room is shaking after the gamer shoots a gun. And the device can make items in the room appear different, such as in a cartoon.
"The real magic is in the software," Jones said.
As opposed to two players each wearing their own pair of 3-D glasses, Jones said playing a video game or watching television with IllumiRoom is more of a shared experience.
For their demonstration, the researchers primarily focused on using the device in video games because they were able to access open-sourced software for some games. But the concept could also be applied to watching shows or playing interactive educational video games, Jones said.
"We just kind of scratched the surface," Jones said.
One of the next steps is to work directly with game designers and cinematographers on improving the product, he said.
A graduate of Homewood-Flossmoor High School in the south suburbs of Chicago, Jones said he has always been interested in the intersection of art and design and computers, and at the UI, he has had the opportunity to work with professors — David Forsyth and Brian Bailey — and other students in the field of human and computer interaction.
Earlier this spring, he and fellow UI doctoral candidate Rajinder Sodhi won an Illinois Innovation Prize for their work in extending or amplifying virtual experiences. Both will return to Microsoft Research this summer to continue their internships there.
A video of how IllumiRoom works can be viewed at http://www.brettrjones.com.