This image of the reconstructed model of the Kalabsha temple uses perceptually accurate simulation of natural lighting. The scalable lightcuts algorithm uses knowledge of human perception to render this complex scene in a few minutes while evaluating only 1/10,000 of the lighting evaluations.  Model: Veronica Sudstedt, Patrick Ledda Render: Kavita Bala, Bruce Walter

Scientists know where the courtyard was, where columns once stood, the location of the hypostyle hall. They can point you to the three chambers that form the temple’s sanctuary.

But to see what that sanctuary was like when ancient Nubians walked its halls, how firelight washed over its walls, or how moonlight cast shadows over the temple’s massive stone sculptures, the curious are left wholly to their imaginations.

That may change.

Exciting research under way at Cornell is redefining how computer graphics and virtual reality can test the limits of imagination, and ferry the curious to places never before seen—like Kalabsha.

Kavita Bala University Photo

She hopes her work may lead to the construction of a virtual Kalabsha, where archaeologists would have a chance to visit the temple in all its glory, two millennia ago. For that to succeed, though, it has to look real, right down to the lighting.

Most important, she says, is remembering the virtual temple will be seen by real people, a factor sometimes overlooked.

"People who have looked at lighting have tended to not look at how we perceive it," Bala says. "The hardcore engineering perspective is measuring the light energy and trying only to evaluate that very accurately. Being accurate requires very time-consuming computations. But the algorithms that have been developed don’t exploit the fact that there’s a human observer at the end."

The challenge is knowing when the observer has enough information to perceive what’s before them as genuine, even if some processor-hungry data is missing.

"My goal has been to produce graphics images that faithfully represent

The kitchen includes complex illumination from lighting fixtures in the ceiling and walls and sun streaming in through the windows. Model: Jeremiah Fairbanks; Render: Kavita Bala and Bruce Walter

To build a virtual temple by harnessing today’s most accepted methods of complex computation and graphics would not only be unacceptably time consuming, but costly. Even for Hollywood, a virtual Kalabsha of the scale and accuracy Bala envisions would break the bank.

So how do you achieve virtual reality, with an emphasis on reality, on a budget?

A little sleight-of-hand.

Rendering the forest for the trees

Much of Bala’s work focuses on stripping some of the geometric and lighting complexity from graphics already too bogged down by their own computations. Such intensive graphics are a drain on a computer’s processor and can be painstakingly slow to render, which is impractical. A lot of what the observer sees, Bala explains, doesn’t even register.

"The complexity of the scene overwhelms our visual system and our ability to process all the information," she says. "Then the question is, ‘What shortcuts can we take?’"

The top image shows the Kalabsha temple but rendered at night, in foggy lighting conditions, and with complex effects like motion blur and depth-of-field. The kitchen (bottom), now smoke-filled, includes more complex shadowing and lighting effects. The multidimensional lightcuts algorithm computes these computationally expensive effects with a surprisingly small additional cost by exploiting the insight that "more is less": more complexity is less visually salient. Model: Jeremiah Fairbanks; Render: Kavita Bala and Bruce Walter

"If you have this better understanding of how we perceive images, you can improve both the modeling and the rendering," she says. "In the modeling you can take shortcuts by not representing all kinds of details that would overwhelm any graphic system trying to handle all of them. They don’t matter because there’s so much information going around that the eye is tuning out."

When you determine what can go—how many trees in a forest, for instance—you can start to determine how many fewer polygons are needed to produce realistic images. Bala says her less-taxed graphics are subjected to a psychophysical test-run with viewers, the best judge of whether something seems off. These tests indicate viewers are convinced images with less graphical information are no different than more complex counterparts. Viewers see both images, Bala says, and simply can’t tell which one is the original.

Much of Bala’s work aims to realistically mimic light, while reducing resources needed to compute complex environments.

"You’ve heard the phrase ‘Less is more,’ right? I say, ‘More is less,’" she says. "What I mean by that is, the more complex the lighting is, as human beings, the less we’re less able to perceive all of its complexities. We throw away all of that information."

Bala’s group has developed scalable rendering algorithms—lightcuts and multidimensional lightcuts—that exploit this insight to enable rendering of extremely complex lighting

Bala says the science has only scratched the surface so far, but is leading to better and more efficient algorithms that will help speed up the rendering of reality and will help graphics systems scale to handle the complexity of the real world.

Bala’s work on reducing the computational heavy lifting of complex aggregates of objects and analyzing how viewers pay less attention to individual objects was presented recently at the 2008 conference for the Association for Computing Machinery’s Special Interest Group on Graphics and Interactive Techniques (ACM SIGGRAPH). The published paper, which was co-authored by Ganesh Ramanarayanan, a Ph.D. candidate advised by Bala at Cornell, and Professor James A. Ferwerda, a perception psychologist from the Rochester Institute of Technology, explains the "more is less" concept: "Aggregates of individual objects, such as forests, crowds, and piles of fruit, are a common source of complexity in computer graphics scenes," according to the paper. "When viewing an aggregate, observers attend less to individual objects and focus more on overall properties such as numerosity, variety, and arrangement. Paradoxically, rendering and modeling costs increase with aggregate complexity, exactly when observers are attending less to individual objects."

Lord of the pixels

Many of the advances in computer graphics have incubated under the bright lights of the entertainment industry, and Cornell’s research has played a leading role.

Hollywood has already tapped Steve Marschner, a Cornell associate professor of computer science, whose improvements to the rendering of skin were used to breathe life into Gollum in director Peter Jackson’s film trilogy, "The Lord of the Rings." That work earned Marschner a technical achievement award from the Academy of Motion Pictures Arts and Sciences.

Marschner is also tackling a better algorithm for hair, which aims for a more natural look while reducing the rendering time to boot.

Hollywood isn’t the only entertainment industry to profit from special effects advances. Computer gaming has long mined computer graphics research to lend a little more realism to games like Doom, and massive multiplayer online role-playing games like Second Life, where virtual interaction still depends more on imagination than good graphics. 

Although computer graphics research will continue to notch up the gee-whiz factor in movies and games, that same research may one day take root in a host of other applications, including rapid prototyping, virtual archaeology and e-commerce.

Sounds like ...

Doug James Karen James

Surgical simulation is one example. What would it feel like to puncture a membrane during surgery? Being able to feel effects with haptic feedback, as well as see them, would be a big boost to education.

Which is why James, like others, recognizes that "virtual" without the "reality" can bring down the whole house of cards. In the same way Bala is looking to bring complex images to life, bathed in convincing virtual light, James understands that without authentic sound and feeling, even the best images can only tell part of the story.

"One whole area that’s just coming out of the dark ages, so to speak, is realistic sound synthesis," he says. "There’s been a lot of work on getting sounds to play back in some rich environment that sound like they could be real, but in the end they are essentially, just canned sounds. ... We fundamentally lack the ability to simulate physical sound sources for all sorts of different things, like even just crumpling a piece of paper, or making some tires skid across the ground, or crashing a truck into a stack of TV sets. Any of these things are just way beyond what we actually know how to do, for various reasons."

Even sounds we take for granted, like water pouring

This still image is from an animation that visualizes the estimated rate (in 2005) at which plastic water bottles were not recycled in the U.S.: approximately 845 bottles/second. The bottle dynamics and sound vibrations are computer generated.

"Nobody’s really ever done that," he says. "Even to get started and think, ‘How does fluid make sound?’ and when we know that, ‘What kind of algorithms should we create to simulate it?’"

People’s instinctual ability to detect false sounds adds to the challenge.

"For graphics, making physics look real isn’t so hard, but actually making sounds seem real can be tricky," he says. "People have a very good ear for these things. Our brains have evolved to detect unnatural rustling in the bushes because if you didn’t, you might not make it."

The challenge of creating realistic haptic feedback is compounded by the "chicken and egg" problem, James says. He and his associates don’t have many existing applications or devices to work with because hardware isn’t created without the technology to drive it.

"Building devices that can convincingly depict contact interactions is just hard, from a mechanical design perspective," he says. "Another reason is algorithmic. We fundamentally lack algorithms to display the multitude of contact interactions you take for granted. So how do you actually display, using a robotic device in a convincing way, picking up a piece of paper? We don’t have devices that can reflect forces from your hand in a way that will make you think the subtle interaction is true."

"Even if you could," James jokes, "it would still be some sort of poor silent movie, right? Most of the graphics we have now are silent movies."

James’ research into realistic simulated sound and haptic feedback has taken him and his associates down a path of interesting exploration, he says. That research has, like Bala’s, pointed to a need to consolidate what data is processed during the first steps of any digital journey.

These images are from a simulation of 3,601 falling, colliding, and vibrating plastic chairs. Hundreds of millions of collision events were processed between deformable chairs in a matter of hours using Bounded Deformation Trees.

It’s like a drag race, says James. You could wait until the flag is dropped to step on the gas, but you’ll go a lot faster if you rev up your engine first.

From big screen to living room

Bala sees a time, perhaps in the not-too-distant future, when people will be able to scan their living rooms electronically and rearrange furniture virtually.

More importantly, Bala points out that same technology can also be used to preserve sacred and historic objects that may otherwise be lost to the ravages of time.

"We need digital representations of this kind of cultural information that is dying out," she says. "We need to archive and keep it. That’s one of the applications that really excite me."

Both Bala and James say with so much still to learn, the research remains exciting as much as it is creative.

That’s appealing to students, James points out.

"Aesthetically speaking, it’s really exciting for students who want to have a little of everything," James says. "To do computer science stuff, you need to know programming languages and how to design algorithms, but for these new research areas you also need physics and you need to be creative, and have various other skills."

Not to mention fun.

"I just love my job," James says. "It’s great. You come in and try to figure out how to solve all these puzzles. And not only to solve the puzzles but to even figure out what the puzzles are, which direction you should go."