Once again this year there is enormous interest and activity around VR at SIGGRAPH, although with VRLA following closely after the SIGGRAPH event there are some companies that are focused on this more specific VR conference and thus not here at SIGGRAPH.
VR can be found peppered now throughout the technical papers, talks, seminars and of course at the SIGGRAPH VR village which has had long queues since it opened on Sunday.
The Foundry’s CARA
The Foundry’s new CARA tool is at the forefront of how many vfx and compositing artists are transitioning to VR and addressing the production issues of stitching, tracking and grading in 360 video.
BMW’s 360 YouTube experience features the new M2 and super model Gigi Hadid in an adrenalin-fueled motorsport interpretation of the famous ‘shell game”. The clip has had over 5 million views with less than four weeks on YouTube. Michael Ralla presented BMW M2 – How CARA VR helped Framestore to keep all eyes on Gigi Hadid, as part of the Foundry’s 2016 SIGGRAPH presentations on the trade show floor today.
Michael Ralla, is compositing supervisor, at Framestore focused primarily on TVC and advertising work.
Ralla has worked as a Digital Artist within advertising and feature film VFX for over a decade. He’s previously worked at Animal Logic in Sydney, MPC in Vancouver, Digital Domain in Los Angeles and Industrial Light & Magic in San Francisco. His feature film credits include Avatar, Avengers, Transformers: The Dark of the Moon, and Iron Man 3.
Since joining Framestore, Ralla has overseen major advertising projects including the recent 360° ‘M2 Eyes On’ campaign for BMW, and Samsung’s hypnotic eight part ‘ANTIdiaRy’ film series featuring Rihanna.
The BMW 360 project was directed by Marc Forster, primarily known for his work on Quantum of Solace and World War Z.
“When Marc Forster and Tool of North America approached our team at Framestore to help realize KBS’s adrenalin-fueled interpretation of the classic shell game, we quickly realized that we were buckling up for way more than traditional post-production work” – Michael Ralla
The campaign featured five identical high-powered BMW M2’s and Gigi Hadid driving sophisticated, elegant and hard-to-follow choreography, all seen through 360 with the goal to immerse the end user so they felt like they were part of the experience. By giving the viewer a defined motivation to spin around their own axis trying to follow Gigi’s car, the experience is extremely immersive.
Framestore relied heavily on Nuke and CARA VR for the project, and since 2014 has been helping shape The Foundry’s VR toolkit that was used on the project in its entirety. CARA VR has recently been released to the general community, (fxguide’s coverage here)
Ralla’s talk discussed all stages, from agency brief, to conception and previz, shoot and post-production with an emphasis on making the resulting 360 video the most immersive live-action experience to date with all available tools – on time and on budget.
For 360-video production, the VFX house becomes more or less the central production hub, especially from a technical perspective, but in this case also for most of the creative problem solving:
- Car choreography: From matchbox cars over previz to the final execution in the Mojave desert
- 360 camera rig and camera platform consulting: Rig layout, cameras, lenses, camera vehicle
- On-set workflow: Realtime-stitching, 360 dailies, monitoring Pre-pro and production
- Data handling and sync-check across the footage from all cameras
- 3d camera tracking and the challenges 360 tracking provides.
- Rig removals in UV space
- Comparison between different stitches: Rough stitch, first pass stitch, final stitch
- The use of photogrammetry: camera rig calibration, car rotomation
- CARA VR use for Gigis’ stitch
- RayRender; full CARA VR toolset in conjunction with custom in-house tools for full environment rebuild and potential stereo conversion
- 360 render review and artist feedback: Before and after videos plus how-to demos
- “Bigger picture”- mindset: Modularisation of tasks
Ralla explained that by the end of the job they would end up comping in Gigi, replacing the environment, making a new road, replacing the sky and the only aspect that was from the original photography are the cars, but even here the car footage was projected over 3D Geometry, “we suggested more than once – why not do the whole thing in CG, but client always wanted the cars to be real” he explained. Of course the other thing the live action gave the Framestore team was the staging and real dynamics of 5 cars and a camera car racing down a track.
This is how they did it:
The team used a multi-camera system to capture the footage and seamlessly stitch it together. The team ended up using 8-15mm lenses, set to 8mm. A wider lens increases the field of view on each camera, which meant that the rig would need fewer cameras to capture the full environment. Ralla commented that various seasoned professionals on set were concerned that filming with 8mm lenses the cars wouldn’t seem “fast” enough. In a normal TVC a DOP would shoot with longer lens to make the cars seem faster and sexier. With 360 capture there is no ‘fast’ visual language to rely on, no fast cuts, no tight frames – if fast action is required, there is no better solution than simply have the stunt BMWs driven really quickly.
The lenses were pro-consumer lens costing a few thousand dollars each and at least 3 were destroyed during filming from small rock or dirt flying up. “With fisheye lenses you ca’nt put anything in front of the lens to protect them”, Ralla explained. If he was do the same spot again he suggested he might explore using more cameras with either the 14mm Rectilinear f2.8 Canon lenses or even Ultra Prime 8R/T2.8 professional lenses (+$31,000 each). The 8mm gave 135 degrees field of view and the R8 would give closer to 122. As the 8-15mm were not professional lenses each had slightly different optical characteristics, and being fisheye lenses it is nearly impossible to shoot a full grid alignment chart. Framestore found the lens correct for each lens was key and very specific to that lens.
Designing a shell game
When developing 360 content, storyboarding presents an additional challenge. In VR there is no framing and cutting in the traditional sense, and for this project in particular the shot design would play a key role in the illusion of the shell game.
For “Eyes On Gigi”, the team aimed to previz the project as much as possible – both in terms of what the cars needed to do, to inform the stunt drivers and for what the audience would see. To solve this the team bought 5 BMW toy cars and printed out a scale-accurate paper racetrack on the floor of the Framestore. Director, Marc Forester, stunt coordinator Eric Norris (Chuck Norris’s son (!)) and the Framestore team mapped out the choreography and even filmed it with a miniature video camera.
Previz as foundation of everything
After the sequencing was worked out on the floor, the team moved to a bird’s eye view.
Assembling the spot in toy-cam team gave the a good idea of the choreography and the subsequent step of the detailed CG pre-visualisation. This also allowed Framestore to put a 360 degree virtual camera in the middle of the scene. It also allowed them to test various approaches for different camera setups, rigs and camera car platforms.
Technically, the previsualization showed that that the camera car would have to become part of the choreography, and revealed some expected problems, such as parallax errors and camera car occlusions due to proximity issues of the BMW’s. As a result, the team were able to adjust for these problems before going on set.
The “Five-Eyed Beast” & “The Shaved Corvette”
The 360 multiple camera rig was mounted on a camera car, nicknamed the “Shaved Corvette”. Framestore was able to test different cameras, lens and rig permutations virtually – using the previz scene – to a high level of accuracy.
They settled on five RED EPIC cameras shooting in 5K through 8mm fisheye lenses – , (supplied by BigLook360).
“The Shaved Corvette” is so nicknamed as the car needed to drive faster than 60mph smoothly, and have a minimal footprint, so that the BMWs could come as close as possible – the only vehicle the team could find that suited their needs was a highly modified Corvette by Alan Padelfort. One key aspect of using the Shaved Corvette was to keep the car as small as possible as the base would need to be removed in post. The final car removal solution involved an entirely new road which was done with 16 top down re-projected road in NUKE via the master camera track – to make 1.6km of virtual road.
The real deal – on set
There was less than three weeks between the first meeting and the shoot, and the previz became the blueprint for the onset execution, a clear visual reference for everybody, from stunt drivers to director, that successfully demystified the 360 video workflow.
Above is the polar projection from the stitched camera set. This was key for being able to keep track of which car was which. For the spot to work the team re-projected the cars onto car geometry built from the photogrammetry and it was specific to each car.
On set the car was photographed for tracking, camera rig calibration, car rotomation and photogrammetry.
From the Photogrammetry above, the cars then rebuild and reprojected in NUKE (below).
The Agency wanted a perfectly smooth camera movement in the final spot, so camera tracking and stablising was key and the team relied on CAVA tool set. The spot is one actual take, there were no cheats on the cars but Gigi was shot on a separate day on the same multicamera gig and then composited onto the new road and over the cars. As she walks in from the side and close to camera she is seen on three cameras and cross two seams. While the stitch is good, she needed to be rotoed and comped to fix her image and avoid it breaking up and being misaligned on the seams.
Post-production
Back after the shoot, the Framestore team spent six weeks on the post-production of the spot.
The team used the Foundry products:
- NUKE
- CARA VR use for the Gigis’ stitch which was then repaired in NUKE.
- RayRender; full CARA VR toolset in conjunction with custom in-house tools for full environment rebuild and potential stereo conversion
- 3d camera tracking and the challenges 360 tracking provides.
- Rig removals in UV space.
The entire environment was replaced with a 24K new environment but while the team would normally fly a drone to produce imagery for rebuilding the environment, this was impossible as the next airstrip over had actual F18 Jets flying in and out, so the team had to use overhead satellite footage. To avoid the ground looking flat, the team then used their stadium crowd software to populate the enormous environment with static plants and shrubs. The sky was a 16K dome, made from different second unit photography but with animated clouds.
The Ray render node in NUKE’s tool set was key to getting the project output to the quality the team wanted Ralla explained.
During post the team used an Oculus DK2 VR rig to review and examine the shot as work in progress. While the BMW spot clearly can be viewed as a VR project, Ralla feels there is a major difference between a 360 video and a VR project. The BMW spot is finished in Mono (but with the extent of the post it could be easily rendered as stereo), it also works well as an iPad experience where the user can spin around, but for Ralla unless the spot could have head movement and the ability to walk around in the space it should not be thought of a VR project. There is even a special app for see the spot on the iPhone or iPad (click here).
The planning during the pre-production stage paid off since despite the massive data requirement of a 360 project, all the stitching work, the extensive environmental cleanup and rebuilding, the project was completed on schedule. “Eyes on Gigi” was designed for a YouTube release, and mass audiences eventually proved this assumption right: It has just recently become YouTube’s No. 1 most viewed 360 video commercials of all times, with more than 5 million views.
http://youtu.be/OwDtPixqJLc
To view 360° videos, you must have a current operating system and the current version of Chrome, Opera, Firefox or Internet Explorer installed on your computer. A current Android or iOS version with a recent browser must be installed on your mobile device. To view on your mobile iOS device, a current version of the YouTube app must be installed. Older operating systems, browsers and hardware are not supported.
This presentation will be repeated,
Wednesday 10:45 – 11:15am: and Thursday 12:35 – 1:05pm: Booth 641 @ SIGGRAPH.
Digital Domain VR
Digital Domain has had multiple VR announcements and moves this week, namely: cameras, partnership deals and Comic-Con promos and conventions.
They’re with Her
For the first time in history, the US Democratic National Convention will be available to view in virtual reality (VR) through a livestream provided by Digital Domain. A free curated video stream will be provided with four different 360° camera views of the national political event, streamed live from the Wells Fargo Center in Philadelphia July 25th-28th. Additionally, 360° highlight videos will be available for on-demand viewing throughout the convention.
This exclusive footage will be broadcast to the DNC mobile apps on the iOS App Store and Google Play as well as the official DNC website and YouTube channel.
Digital Domain also provided select immersive footage from the Republican National Convention
Zeus 8k
Digital Domain Holdings Limited, the parent company of the visual effects company Digital Domain, revealed a new “Zeus” super high definition 360° VR camera at a global press conference in Beijing late last week. DD is entering the hardware market for the first time, the Zeus Camera will be capable of both recording 8K and livestreaming immersive footage at resolutions of 4k. The specs on the camera are that it can record footage at a greater than 8k resolution at 10 bit, shooting 75 frames per second. The camera is also capable of seamless 360° livestreaming at levels up to 4k resolution. The Zeus will be deployed immediately at a number of high profile global events, but at this stage there are no images of the camera or from the camera, so a production deployment schedule is unknown.
Digital Domain also announced a partnership with DreamWorks Animations to produce virtual reality content and experiences, beginning with Shrek and Kung Fu Panda. Digital Domain is working on content currently aimed at VR experiences viewed on the Gear VR and Vive platforms.
Comic-Con
Also DD and Syfy were at San Diego Comic-Con. Attendees were able to view a new VR experience for the forthcoming science fiction TV series Incorporated from producers Matt Damon and Ben Affleck.
The experience, “Welcome to the Quiet Room,” was shown all weekend exclusively on the HTC Vive headsets inside the Hard Rock Hotel in San Diego. The VR experience was inspired by the show’s debut trailer, (Below) the VR experience recreates a key setting within the world of Incorporated, a so-called Quiet Room where those who have been disloyal to the Spiga Corporation are taken for questioning and eventually ‘retired.’ Viewers were interrogated by Julian Morse (played by Dennis Haysbert of “24”), the Head of Security and most feared man in Spiga’s US Headquarters, who also provided clues about the mysterious world of Incorporated.
https://www.youtube.com/watch?v=AXj08jqquwQ