VR has become split somewhat between either the virtual world that you can explore and uses head tracking and alternatively, a filmed VR space where the camera position is fairly dictated to you. The first of these worlds is rich and interactive but tends to have no humans, no actors and is either based on stills or CG. The second has actors running around but you generally cannot interact with them, or use advanced VR features.
This all comes from the nature of the VR technology. While it is changing, for now the more interactive experience means no live action actors. By contrast, a passive but immersive experience allows you just to look at live action actors, but not even move your head, other than to just pivot around on the spot. The problem is that while real filmed sets can be stereo and 360, they are filmed from just one place, and not withstanding some very promising lightfield technology, the original camera placement is fixed all around one nodal point with a bunch of small cameras looking out in every direction.
With this in mind, the team at 8i wondered ‘what would happen if we turned that rig inside out?’
While everyone else has been building rigs that look like giant donuts, 8i, based originally in New Zealand, decided to build a rig that surrounded a real person. This allows the VR viewer to see real actors filmed live but experienced from an interactive variable position. The performance is still linear in that you can’t talk to the actor or shake their hand, but at least you can move around them a bit, get closer and have all the perspective correct and experience something most VR demos never have – an emotional performance.
Have you ever met someone at a party and for whatever reason they have stood too close and you felt almost awkward or uncomfortable? Have you ever walked in on a private conversation and felt uneasy sticking your nose into a private moment? These are all emotional reactions. While they may have some evolutionary biology logic, the dominant thing is that you just ‘feel’ the interaction – you emotionally react – being a little uncomfortable being that up close and personal.
If you’ve had those experiences, then ask yourself this, do you feel that on a closeup in the cinema? I don’t. I am happy for the cinematgrapher to cut in close for Tom Cruise’s reaction to Jack Nicholson in the court room of A Few Good Men. The moment is powerful and emotional but I don’t feel like I am intruding. But in a demo at SIGGRAPH from 8i where a mother talks to her young baby in her arms and tries to think what message she would like to tell her daughter years from now, I felt seriously intrusive, and I was still not in a ‘closeup shot’. There is a significant difference between seeing Sigourney Weaver dying in Avatar in stereo and seeing a woman choking up talking to her baby in VR – when the VR is fully immersive. It is not a trick or an illusion unique to this one clip, it is the power of experiencing something when you feel as if you are there, versis experiencing a great window into a compelling drama.
Interestingly, the actress featured with her baby came in to record a scripted piece as if she was recording a 3D message to a husband off at war. The piece was OK, but then one of the key 8i team Eugene d’Eon (ex-Weta) suggested she record a piece to her daughter in her arms. When the actress went to record a real message she broke down mid sentence with emotion – and apologized to the crew. Of course, it is this real moment of emotion that is so uncomfortable to experience right next to her, it is real, and it feels real when watching it weeks later, and hundreds of miles away on an Oculus DK2.
It just should not make such a difference that you can move your head side to side and yet it does. There is a significant difference that seems so much more than the technology should provide.
So how is all this done?
8I’s approach is to effectively create a point cloud from multiple cameras and then play the point cloud back in real-time but from a virtual camera anywhere in the virtualized capture space. This then is lightfield technology, but not the same lightfield that the term normally implies. What is even more remarkable about this experience is that the image one sees in the Oculus Rift is not perfect. It is easy to point to artifacts. But as the image is real there is no Uncanny Valley, and the immediacy of the experience means that one just forgives the technical imperfections.
There is no doubt the 8i team will be able to refine the process and decrease the edge artifacts but one just sees past them currently in a VR environment. The human mind is an amazing device. Why is it that a 4:3 episode of The West Wing stretched on a 16:9 set is watchable and avoids the Uncanny Valley when a high res 4K version of Beowulf with all the correct proportions can feel so wrong? It is possible to intellectualize the process but the results are visceral. 8i is producing some of the most significant VR not because the pixels are more authentic. They are capturing real emotion from real people and you can feel it when you stand (it seems) right next to them – often too close to them for comfort.
If you are reading this without judging the images in a VR headset, you will have to believe me when I say you can’t ‘imagine’ what it would be like – you just can’t guess from a window image on a web page on a screen two feet from your face the difference from being 18″ from a crying mother and her baby. It has to be witnessed. When you do you are in the moment and the tech falls away. It is remarkable.
To promote the technology the company has commissioned a project of 100 faces called 3D Video with #100humans Portrait Series. You can watch many of these portraits in the web interface on their web site – but don’t. The difference between a VR experience and a web experience is vastly different. What seems OK on the web is moving and emotional in a high end VR head rig. Don’t even watch it on Google Cardboard – make the effort to see it at least in a DK2 but if at all possible a Crescent Bay. Being able to look around the people is linked to having a full head rig with head motion tracking. A Cardboard VR rig will not provide this, the 8i rig is the strongest example yet of how head tracking makes a serious difference. The 8i demo joins a set of key demos that now offer a glimpse of why VR could be so significant. 8i’s demo is as significant as Epic Games’ Showdown or Smaug (Thief in the Shadows) demos or OTOY’s latest Lightstage Batcave experience But unlike these other brilliant pieces, 8i has real actors and real people filmed live, even through the 8i experiences are a fraction of the cost to produce and lack the polish of the main stream demos.
The reality is that we go to films to see actors act. While timelapse can be beautiful and awe inspiring, it is narrative drama or emotional stories that are the most compelling. It is no accident that the shot we want the most as an audience is the closeup on Meryl Streep or Robert DeNiro or Robert Downey Jr. We love the flying Iron Man suits but we need Downey to be in the story or it is soulless and humorless. Strange then that in the new language of VR 8i may have just adjusted the language of cinematography by having me able to invade the personal space of Downey – “sorry Robert, didn’t mean to interrupt you, excuse me, sorry I’ll be over here. keep going…”
