AR: a researcher’s perspective

It’s that time of year again – SIGGRAPH Asia 2015 is fast approaching. The conference will take place in Kobe, Japan on 2-5 November. Among the fantastic tech papers, presentations, courses, computer animation festival and other CG-related events is a special panel on pioneers. One of the panelists is augmented reality (AR) researcher Steven Feiner from Columbia University. At SIGGRAPH Asia, Feiner will be discussing human computer interaction and VR/AR tech – we caught up with the Professor to find out more.

Prof. Steven Feiner.
Prof. Steven Feiner.

fxg: When we hear about augmented reality today, what does that term mean to you as a researcher? What are some of the areas right now or recently that you’ve been looking at in AR?

Feiner: To most researchers in the field, augmented reality (AR) refers to interactively merging virtual information with our perception of the physical world, so that the virtual and real are geometrically aligned with each other. This can make it possible for important information that we wouldn’t normally be aware of, or that would require additional effort to obtain, to be immediately perceptible and tied into the exact context to which it relates. And, that can provide crucial advantages over the status quo in many domains. Although work on AR uses graphics to appeal to our sense of sight, AR can also involve sound, touch, smell, and taste.

My lab investigates the design of AR user interfaces, and software infrastructure for AR. We develop and evaluate interaction and visualization techniques, determining what virtual objects to present, how to lay them out in the surrounding world, and how to interact with them. And, we also develop prototype applications that address task performance in domains such as maintenance and repair, navigation both inside and outdoors, and games and entertainment. We are especially interested in how AR can be used to assist collaborating users, whether they are working together in the same place or at distant locations.

fxg: People might be surprised to hear how long AR research has been going on – what do you see as some of the biggest breakthroughs in the area over time?

Feiner: The first AR (and VR) research was done by Ivan Sutherland, who published a paper in 1968 on his group’s development of a see-through, head-tracked, head-worn stereoscopic display. So, the field is about 50 years old at this point. Breakthroughs beyond Sutherland’s pioneering work have included significant improvements in fast low-latency tracking and capturing of the surrounding world, high-performance modelling and rendering, high-quality displays, and effective user interfaces. That said, there is still much research to be done to improve our ability to display virtual objects that look, sound, and feel (to pick just three of our senses) exactly as if they are actually present in the real world. And, to use that ability to develop user interfaces that present the right virtual objects at the right time and the right place to help make us better at what we do.

Above: a video from Feiner’s research group.

fxg: Where do you see the main areas that AR and VR are and will be used in, perhaps in ways that aren’t just for entertainment?

Feiner: Entertainment is going to be a crucial driving area, as long as there are people to be entertained. However, there are many other high payoff domains. For AR, these are areas that can benefit from having information that is otherwise invisible be made visible directly in the real world context to which it applies. (I’m using terminology specific to our visual sense here, but this really refers to all our senses.) Plainly put, AR is relevant to task performance in general. Examples include manufacturing things, maintaining them to make sure that they don’t break, and fixing them when they do. This means everything from equipment maintenance in a factory to surgery in an operating room, playing a sport, or watching it being played. And, whether or not we use AR while performing a task (e.g., it might be considered an unfair advantage for a professional athlete), it can also be an extremely effective way to learn how to perform a task. And, to learn about complex concepts in general. Imagine being able to learn physics by changing constants such as gravitational acceleration, and seeing how objects respond. (And, would it be easier to learn how to juggle with virtual balls in simulated Martian gravity?) How about representing more abstract concepts such as mathematical algorithms or sociopolitical systems as 3D visualizations that we can literally walk around to explore in detail?

Insofar as VR tries to replace our perception of the physical world with a wholly virtual world, it can be useful not just in games, but in simulations of other tasks that take place somewhere real or virtual that we are not. For example, consider exploring some distant place for which we have a detailed physical model. Imagine a geologist being able to walk around the scanned surface of another planet and then being able to direct a rover on that planet to take a sample of something that looks interesting.

Above: another video from Feiner’s research group.

fxg: There is a lot being done in the VR and AR worlds, but what are some of the particular challenges in bringing AR to life from a technical and experiential viewpoint, including UI design?

Feiner: Depending on the domain and task, AR and VR can get really hard, really fast. For example, virtually labelling a small number of well-known objects at well-known locations, viewed by a well-tracked observer is much easier than when we have a larger number of objects, raising issues of how to label them clearly and, perhaps, which labels should be suppressed for clarity. Recognizing and tracking those objects can be extremely difficult; for example, if they’re outside at night in bad weather. If we’re trying to create virtual objects that look and act like real ones, we need to model in real time the subtle interplay of light and sound between real and virtual objects in all combinations. And, we have to figure out exactly how to use these capabilities to make user interfaces that effectively address our tasks. Because we’re dealing with people, this means we need to understand how people perceive and interact with their world—the domain of perceptual and cognitive science. That’s made even harder when what we perceive and interact with can be diminished, augmented, and mediated, potentially making it quite different from the physical world. And, how can we use VR and AR to tell stories, whether real or fanciful? Or to make and experience art?

This is going to keep SIGGRAPH researchers and practitioners, and those in other domains, busy for a long time!

SIGGRAPH Asia takes place this year in Kobe, Japan.
SIGGRAPH Asia takes place this year in Kobe, Japan.

fxg: What considerations have to be made in terms of the implementation of human-computer interaction and AR/VR for there to be a pleasant viewer experience, do you think? What new advancements (ie such as light fields) do you think might help?

Feiner: Light field displays (and, for that matter, sound field displays) and the simulations that drive them are going to be important to make things that look and sound right, stimulating our senses the way that real phenomena do. For example, we would like virtual objects at different distances from us to be focused differently by our eyes, just like real objects. And, even if we can make virtual things look and sound like real ones, we’d like them to feel real, too, which is arguably even harder.

Above: a demo video of Meta’s AR tech. Feiner is their lead advisor.

fxg: Do you have any views on the efforts of some of the planned commercial AR releases, such as Microsoft’s HoloLens and Magic Leap?

Feiner: I’ve been a great believer for a long time in the potential for AR, and have directed much of my lab’s research in that direction. So, it’s exciting and gratifying to see Microsoft make a major commitment to AR with HoloLens. And to look forward to the light-field displays that Magic Leap is developing. I’m serving as lead advisor to Meta, a startup that is developing AR eyewear, and has already released a dev kit that developers can use to experiment with head-worn AR. Beginning with Ivan Sutherland, researchers have been actively exploring VR and AR for a half century. Smartphone AR apps have been in app stores for years, commodity consumer VR head-worn displays from Oculus, Valve/HTC, and Sony are promised for 2016, and head-worn AR dev kits are here and in the works. So, we are at a very exciting time in the history of these fields, as promises are turning into reality!

fxguide is the official podcast of SIGGRAPH Asia. Find out more about the conference and register here.