ILM has moved into VR - AR and MR with the announcement of their Advanced Development Group setting up a new unit the XLab headed by Lucasfilm VP and ILM VFX supervisor Rob Bredow and Creative Director John Gaeta.  The group is the embodiment of several years of intensive research and immediately it has unveiled a range of new technology and filmmaking tools from VR experiences to new virtual cinematography and even a director's holodesk - ILM style.

“The people who work here have been investing in achieving the unachievable for more than 40 years. Creative storytelling was something that George Lucas instilled in each of the companies from their earliest days and out of that came the incredible innovation that continues to this day. We are currently exploring the fictional universes of Star Wars, and I think a lot of people would like to be immersed in them. The challenge of ILMxLAB will be to find out what storytelling looks like in this new space” says Lucasfilm President Kathleen Kennedy

In a remarkable commitment to moving ILM further into areas outside just their core filmmaking and visual effects work, Kathleen Kennedy and ILM President Lynwen Brennan have moved cutting edge technology that the company has been researching for several years into a new xLab unit. Brennan thinks of ILM as a creative partner rather than an effects house, after all the have produced fully animated features such as Rango . "I believe entertainment has gotten blended and we view ourselves as an Entertainment company, and we can help any director achieve their vision no matter whether it is in live action visual effects, or animation or if it is in the new spaces - the more immersive experiences. You can take the amazing technology we have developed and use it for real time in an immersive space - by that goggles, or a location experience."

ILMxLab is headed by Rob Bredow who was hired initially to just be a visual effects supervisor, but his hiring worked in very well with Kim Libreri leaving for EPIC Games. Rob, very much with Kim's blessing, was able to take over the work being done by the Advanced Development Group. With ten years of Star Wars films planned the company is extremely well positioned with Disney's long term vision to build and develop new technologies not just inside the feature film sandbox, but beyond, in ways that can help film directors but also take these world to audiences in completely new ways. John Gaeta, on the other hand, was brought in by Kennedy and Brennan a couple of years earlier with the express intent to explore immersive entertainment and how the technology that had been developed by ILM and Lucasfilm’s Advanced Development Group could best put to use in creating premium immersive experiences.

John Gaeta with the Ipad view incorporating the Structure Sensor which is scanning the room John is in.
John Gaeta with the Ipad view incorporating the Structure Sensor which is scanning the room John is in.

"What we're aiming for is to open the two-dimensional world of the movies and allow fans to walk into those worlds with the same visual fidelity," says John Gaeta, ILMxLab's Creative Director. Gaeta is himself very well known for exploring and developing dramatic and creative uses of new technology perhaps most notably from his Matrix work that helped revolutionize the industry when it was released. Today he is a key figuring in extending the work George Lucas started even earlier with Star Wars 40 years ago at ILM.  "All that George begat caused a reassessment of innovation from movies to video games. The next 40 years of ILM is about exploding that universe with tech once again. xLAB is as attuned to Silicon Valley as it is to Hollywood."

In an exclusive interview with fxguide above, Rob Bredow discussed the new xLAB and how the huge slate of Star Wars films at ILM presents a real opportunity for the company. One immediate example of this move into VR is the work ILM has done with VR studio Felix & Paul for the new Jurassic World film that opened this week. In the VR Jurassic Experience users can stand next to actual props from the film and see in stereo VR what it would be like to observe a dinosaur up close.

A key example of possibilities beyond VR using unreleased technology is an iPad Star Wars-inspired scene where R2D2 and C-3PO are hiding from Stormtroopers in Mos Eisley, on Tatooine. As you are watching the scene on a screen, you can orientate the iPad and turn 360-degrees and see the world from any angle - including from above or flying overhead in an X-Wing fighter jet. The remarkable rendering was first previewed at a conference in Europe, and works via rendering in real time, but not on the iPad itself. As the rendering is real time but not limited by the computer power of the iPad, its render realism is remarkable, and vastly better than any similar game, that would be locally rendering on the iPad's GPU. In this area of really pushing hard core real time rendering the Xlab team is working with companies such as EA and their Frostbite Engine and Epic Games and their Unreal Game Engine. Just see the ILM Xlab video below which includes realtime renders which appear at almost  to be at a feature level quality.

Rob Bredow 'on' Tatooine with some old friends. Note Rob sees Stereo himself, but the camera he holds offers correct 2D at the same time.
Rob Bredow 'on' Tatooine with some old friends. Note Rob sees Stereo himself, but the fake camera unit he is holding offers correct 2D at the same time - wirelessly. The tracking markers on the 'virtual camera' inform the graphics engine of its position separate to Rob's head worn 3D glasses (which are separately tracked).

iPad and VR gear are not all that ILM is exploring. In these images above a series of projectors are projecting a stereo 'holodeck' onto the walls in front of the ILMxLab team. Each member person in the space sees the image in 3D but it is only correct from their point of view. The projectors warp the image so it falls looking correct from the unique perspective of the director or user. The camera seen being held by Rob Bredow above is tracked in real time and this informs the computers which warp the projected images accordingly. The net result is that in Rob Bredow's point of view there is no distortion and the world is in 3D and all around him. The effect is remarkable as the walls and their corners appear to melt away leaving the director to view the 'set' as if they were on Tatooine - and then point the virtual camera in any direction to film it (virtually).

Below, John Gaeta can be seen experiencing a dinosaur (remarkably like one that might appear in a Jurassic theme park), on the sound stage/motion capture space at ILM in San Francisco. One could imagine the system might be used to allow actors to perform to virtual characters which themselves are being driven by live action motion capture artists (of course without the actor wearing stereo glasses - and with either a green screen panel projected behind them or separated via roto).

Note John is being head tracked for the correct projection, and the dyno is being rendered from a real time Vicon motion capture (left)
John Gaeta (Right) is being head tracked for the correct projection, and the dyno is being rendered in projected stereo all feed from a real time Vicon motion capture artist live(left).

The ILM immersive environment is not dissimilar to the Disney imagineering 'Dish' system, but that is a separately developed curved space installation test system. As one might hope, there is now a very strong working relationship now between the Xlab ILM team and the Imagineers at Disney, now that Lucasfilm is part of the Disney family. This builds on year's of friendly professional collaboration from before the merger, dating back to team working together on to the original Star Tours ride at Disneyland.

Says Kennedy: "With image-quality rivaling film, you'll be able to literally step into an alternate reality."

"Apple, Google, Intel, they all have moonshot projects around AR and VR, but they're still just figuring it out," Gaeta says. "Our standards for world-building are very high, and we intend to use our technology to give people experiences that they haven't yet dreamed of."

The complete uncut in depth interview with Rob Bredow is available exclusively to fxphd.com members in the current VR course - class 10.


Thanks so much for reading our article.

We've been a free service since 1999 and now rely on the generous contributions of readers like you. If you'd like to help support our work, please join the hundreds of others and become an fxinsider member.