VR on the Lot

VR on the Lot.

paramountThere are now many VR conferences and events.  In the early days fxguide attended VRLA.  At those early events the scene was very much card tables and indie productions.

Today there are millions, or rather billions of dollars being invested in VR and AR/MR. VR on the lot is a new conference looking at VR with a decidedly industry focus. Held on the Paramount Lot in LA, and hosted by AMD along with The VR Society, this conference aims to take a narrative view of VR while trying to balance production issues with commercial concerns.

I was moderating a panel on characters in VR, which was both an honour and a great chance to connect with the incredible panel of experts (see below).  Just before the VR on the Lot conference, Facebook demonstrated a huge range of their work in this area that influenced much of the discussion.


VR and Social.

VR can be a lonely experience, which is why introducing ‘social’ has been a key part of the road map for companies such as Facebook.  If there were other people in your VR experience – what should they look like? What should you look like?

The issue of characters in VR is not limited to social interactions. If you see a film you want to see actor’s act. We may all love a good wide establishing shot, but we need to have the filmmakers show us close ups of the actors faces, that is what drives stories. People care about people and their stories, not wide vista shots alone.

The issue of characters in VR is not as simple as it would first appear and there are numerous aspects to be considered:

  • Is it possible for anyone to make an acceptable avatar that avoids the Uncanny Valley in VR ?
  • Who will make these avatars?
  • Is the character in VR an avatar representation of you or an independent AI avatar?

Facebook’s team defines social presence as the sense that other people’s social representations are real. If presence captures the sense of being there in VR, then social presence is defined as being there with others. Characters in VR can combine high degrees of VR presence and social presence and they can reinforce each other.

Uncanny Valley : Facebook’s Social VR solution at the Oculus Conn3ct conference.

Can a normal user make an Avatar that other people will like. If a user can’t make an avatar that crosses the Uncanny Valley, how far back, in terms of it looking like a caricature, should the system go? (In exploring this, we assume that the user is making their own avatar to represent themselves).

Who could make this Avatar?
Who could make this Avatar? The question of user generated Avatars in VR.
The Facebook solution released just prior to VR on the Lot.
The Facebook solution released just prior to VR on the Lot.

Without having one’s own Lightstage, could a camera phone waved around in front of your face provide enough data to build a realistic avatar?  As VR on the Lot was about to be staged Facebook via their Oculus Conn3ct conference addressed this very point. They showed a series of Avatar experiments resulting in their final semi-cartoon version.

A huge problem in producing a digital VR version of yourself to show to another VR user is providing facial expressions. While one can program any face to make an avatar look towards another avatar in terms of eye line, how does the system know what your facial expression is?  Even if there was an external camera, for head tracking, most of your face is covered by the VR head set. The solution Facebook suggested is to use gestures to trigger expressions. The system guesses your face mood based on your hand and body movements. The team call this VR emojis.

The evolution of the FaceBook VR avatar


Michael Booth, a 20 year video game veteran, now at Oculus, outlined the team’s journey and neatly addressed some of the issues the VR on the Lot panel would later pick up on.

The Facebook team started with ‘Blockhead’, this was just a simple face made of geometry. It was never intended to be seen by the public, and was made as an early proof of concept.



eyes-too-bigThe first real test was produced by artists as an ‘animation’ style face. Booth explained that this looked alright on a computer but the over sized eyes really became overpowering in VR. The approach also had the very real problem that an artist is needed to make each face.



toomuchThe team then tried a 2D identikit approach but the faces were thought to have too much detail and thus quickly entered the Uncanny Valley.

The 2D artworks had an additional problem that they lacked presence in VR when they were viewed by stereo 3D VR head rigs.



pill-headThe next approach saw the needle swing hard the other way with this very stylized 3D head implementation. These heads were nicknamed  ‘Pillhead’ avatars.

Unfortunately, most people would like their avatar appearance to be pleasing, if not flattering, and these pillheads are less than attractive.

At this stage the Avatars also had no arms. Some earlier versions had just hands but with no arms. While this can work for one’s view of one’s self, it is not as effective when viewing other Avatars.

3d-headThe team thought they were getting closer with these 3D stylized heads. These heads are stylised and can be combinatorially designed.

This version was nicknamed ‘toyhead’. The issue with these particular faces related to how much mouth animation could be produced to match the audio spoken. Since the only reference for lip sync is derived from the spoken audio being converted to text and then driving a pre-solved set of mouth shapes.


facebookThe final solution that was implemented.

Interestingly, one of the key aspects that makes an Avatar seem human is their movement. The only actual direct input is a combination of the head movement and the hands from the controllers. The team discovered it was key to not magnify or smooth this movement too much during processing. Producing fidelity of movement contributes greatly to identity and identification by others.

Facebook’s team decided that the final solution needed to be:

  • Something you should be comfortable with (representing you)
  • and a face that your close friends could recognize as you.

It seems from Booth’s exploration that the Uncanny Valley can be more pronounced in VR than just on screen, and that 1:1 movement can mitigate some of the Uncanny Valley and make an avatar seem more ‘human’ by unfiltered movement.  Facebook’s social VR team stated that maintaining the believability of presence in VR is more fragile than in non-immersive mediums.

The key points Booth’s team identified as being elements of believable avatars were:

  • speech
  • 1:1 tracking
  • the importance of the hands
  • eye contact and having blinking to avoid ‘dead stare’
  • gaze following
  • lipsync
  • emoting and
  • arms and body,(including adding secondary motion to items where possible)

Watch the full session below :

“A Conversation with Mike Booth: from the Oculus developer conference conn3ct”


VR Character Panel:

The Panel on day one of the conference, which I hosted, was composed of

Kiran Bhat,               Co-Founder, LoomAI
Chris Edwards,        CEO, The Third Floor
Andy Jones,            Animation Director, Wevr
Kim Libreri,              CTO, Epic Games
Maxwell Planck,     Technical Founder, Oculus Story Studio


If just anyone wants to produce a realistic avatar then Kiran Bhat ‘s company LoomAI, is at the forefront of providing a much more automated solution than just the Identikit approach that Facebook ended up recommending. His company is still in stealth mode, but as the short video test above shows, his company is able to produce plausible 3D avatars from just one single front facing jpeg. The clip above produced avatars for each of the panelist using just the small jpegs seen above each of the digital characters. It uses AI to provide the head shape and 3D form.

As the panel discussed, first person narratives are just one aspect of VR.  Max Planck from Oculus Studios pointed out that having a human form avatar is only one option.  Planck’s VR film Henry achieves great storytelling with Pixar-ese stylized characters. The film is just one of many that the Oculus Studio is doing. Henry was awarded an Emmy in September. The award, for ‘Outstanding Original Interactive Program’, is not the first Emmy to be given to a VR project (that was the Sleepy Hollow VR experience)—but it is the first original VR short to be given such an Emmy award.


If Facebook addressed the issue of Avatar believability, it did so with the assumption that

  1. The character had to be made by you
  2. Driven by you
  3. and driven in real time (like a puppet).

This is only one small part of the Character in the VR discussion. For example, people are making VR narrative stories with actors captured in a professional environment using expensive specialist hardware. Chris Edwards, CEO of the Third Floor, has had a lot of experience in using characters in virtual productions as part of previz and now this has led to the Virtual Reality Company (VRC). The VRC is behind The Martian VR experience, which was developed at the Fox Innovation Lab in conjunction with RSA Films, VRC and runs using the UE4 engine. Ridley Scott served as Executive Producer on the experience, while Maleficent Director Robert Stromberg helmed the project. Stromberg is co-founder + Chief Creative Officer of VRC.


The VR experience offers the opportunity to become Mark Watney for 25 minutes – Matt Damon’s stranded astronaut in Ridley Scott’s Oscar-nominated film, The Martian VR Experience puts players at the center of a condensed version of the film’s story, adding a variety of interactive elements for users to explore using the Oculus Rift or the HTC Vive. Users can drive a rover across Mars, pilot a ship through space, and even play basketball with potatoes through the experience, which offers a 360-degree perspective of the environments.


Andy Jones of WeVR has experienced both worlds. He is currently Animation Director at WeVR in Santa Monica, where they produced the incredible Gnomes and Goblins VR experience for Jon Favreau. The Jungle Book Director discovered WeVR when Andy, who worked on the Jungle Book, had to leave editorial early one afternoon. “Jon asked where I was off to and I said I was going to a meeting at this VR company I work with”, Jones explained, “And Jon asked if he could come along.. he was blown away”, and that is how the idea of G&G was born. (More in an upcoming fxguide article on WeVR).

What is significant about the characters in G&G is that they engage with the viewer interactively. This highlights the next issue for characters in VR, avatars that are not represented another person (such as with the Facebook Avatars) and are they not just replaying pre-recorded sampled performance. For EPIC’s Kim Libreri, the really interesting ‘undiscovered country’ is autonomous characters driven by AI. Something beyond the state machine models that inhabit say a modern video game. These would be real conversational agents driven by next generation AI technology and able to converse with using natural language understanding. AI is developing fast with Neural nets and deep learning –  allowing for very fast AI solutions (fast at play time).

Hellblade realtime EPIC UE4 performance
Hellblade realtime EPIC UE4 performance

Agreeing that simple filmed spherical 360 video (and thus no head tracking or interaction) was not the future of VR, the panel discussed complex capture volumes that allow live action ‘point cloud’ type moving captures. Fxguide has written earlier about systems such as 8i, which do this very effectively. Such systems capture real volumes so user tracking can allow you to walk around. As effective as they are, theses systems are not interactive, their characters can not look at the user for example, as they are just a recorded sequence. Libreri and the panel wondered if there could be a future in a hybrid solution where the captured data file is dynamically modified by AI to allow characters to look at the user and make eye contact. The overall story would still be linear but the character could be aware of the viewer.

Why does SIRI have a blank screen?
Why does SIRI have a blank screen?

In the image above I am pointing to SIRI on my iphone as part of my own research and asking the question… “why can’t SIRI have a face?” As humans have evolved special neurological abilities to read human faces, and faces are so expressive:  why does a company as graphically evolved as Apple just show one a blank screen? Given on screen real estate is so valuable, is a blank screen the best that Apple could come up with? In movies such as Her, 2001, Star Trek, and many others, advanced computer systems have no face. Yet as people we crave human faces, we seek face to face contact. If so much of interpersonal communication is non-verbal – surely as a society we will seek to put a face on all these disembodied conversational agents and intelligent computer interfaces?

As VR moves to AR or MR and the technology matures it will be fascinating to see how we solve characters, and what skills from the effects and film industry are deployed.

VR as theme park.

People are not yet buying complex VR sets on masse. The actual number of full headset units is really yet to take off. The common logic is that this is a chicken and egg problem, or rather a content problem. If there was a great reason to buy something then sales will come. Many investments therefore are as much about building interest as they are about making any return on investment.

Dark Corner’s very ‘dark’ Mule was a VR Experience that attracted a lot of attention.

New Scientist reported last week that at the low end of the market, simple ‘cardboard’ style VR viewer sales are on the up. “In the second quarter of 2016 over a million Cardboard units shipped in China,” says Tom Mainelli at IDC, a market research company in Framingham, Massachusetts.

Samsung’s Gear VR is bundled with the company’s latest smartphone, and Google also plans to bundle its Daydream viewer with every new Pixel phone it sells,…

in contrast, sales of dedicated headsets are flatlining. Neither HTC nor Facebook are open about the number they’ve sold, although Zuckerberg did admit to a “slow start” for the Oculus Rift.

That’s in line with data from game marketplace Steam, which conducts a monthly survey of its users’ hardware. Figures for VR headsets have remained effectively flat for months, with 0.31 per cent of Steam users reporting the hardware in September. It’s perhaps no surprise, given that VR lacks a “killer app” to drive hardware sales.

This makes the current massive push by content suppliers so interesting. The real money in the internet was not companies putting brochures on line or just providing repacked content. It was something new, and social. User generated content drove the vast wealth of companies like Facebook. It is assumed that VR will work, but what will make it work? Surely, not just 360 video content ? Many people I spoke to at VR On the Lot  do not even consider 360 video in a headset to be real VR. It is certainly the easiest content to make, but it is almost completely lacking interaction and I would suggest VR is just not a great way to tell a narrative story, it is too first person to be engaging as a normal film.  So what is the VR killer application?


Leaving aside games (see below) and social meetups with avatars (see above), the business presentations of of what was popular, especially in China, were VR systems as Mall theme parks. Combining VR with a moving platform base to create a ‘ride’ experience in an arcade. There was speaker after speaker commenting on Malls in China, Korea and Japan that had successfully set up VR experiences that kids were flocking to enjoy.


Such VR experiences are great, but a dedicated expensive machine can only work if it leans on a wider content base.


asiaThat being said, the revenue from a series of VR experiences is enough to command attention. AMD, who co-hosted the event had a series of brief case studies that seem to be compelling. One example in Asia highlighted 5000 locations, each with 5 to 10 stations, (25-50,000 stations) with 10 hours of retail time a day at $30 a user = $$$.

AMD rightly focused on the business cases of such examples and not the current and rapidly changing offerings in these centres.

creedThis is not to say VR does not also have value as a promotional or adjunct medium – for example one of the speakers highlighted the upcoming Assassin’s Creed VR experience. This new VR experience will be released at US Thanksgiving and will no doubt be very popular. From the sound of things AMD has really amped up the tech in the game and helped to produce a very strong marketing tool. But VR still needs to find it’s Killer Ap… playing second fiddle to Cinema or even popular games is not it.

Best thing at VR on the Lot (to play).

vr_lot-3There is no doubt that the best thing I played or experienced at the event was EPIC’s Robo Recall – we have already covered the game on fxguide. When I first tried it I assumed that I would not do the full demo, I knew other people wanted to play and frankly I knew so much about it… heck, I am not even a huge fan of first person shooter games on my Xbox or PS4. But hey, I was there to ‘evaluate the textures and interactivity’… I knew from the friendly booth girl at EPIC to allow 10 minutes to test Robo Recall but I assumed after 3 minutes I would have seen enough    (WRONG).

After a few minutes I never once thought of other people waiting for a go, I did not care about textures (they were great) – I wanted to get on top of the killer robot and blast everything that moved and most everything that didn’t!  It is awesome!

Unfortunately Robo Recall is not a game yet, this was just a demo, we will have to wait until early next year to use it, but Oculus should fast track this game, any user with hand controls will want it.


There were a few technical points I vaguely managed to notice while I was having so much fun. Firstly, the game demo is on Oculus and the hand controllers seem the best design right now for first person shooting games, the hand controllers are light and notice if your finger is off the button. Robo Recall only really has three weapons, but the controllers were great even when reaching out and pulling robots apart with my ‘bare hands’.

vr_lot-4The interactivity, the teleporting jump ahead function (that avoided the problem of walking out of the capture volume) and the completely addictive game play, made this the best VR experience. I have often tried VR and literally thought to myself – how long is it until I can take off the head gear without appearing rude. I just don’t need to passively stand around and watch a pole dancing robot spin on an imaginary pole. It is boring… no matter how well done or arty the sexy robot is.

I do enjoy VR  narratives, and Oculus’ Henry is a delightful animation. I am just not sure that I would not have enjoyed it just as much as a crisp rendering on a retina display iPad as a AR (turn the ipad to turn the view) experience. I guess it was a close second, but Robo Recall only worked in VR they way it worked. It was significantly better as a VR game than it would have been on any current console – even on a giant OLED television. And I know I was not alone – by the end of the second day the trade show area of the conference was very quiet, but there was still a queue for Robo Recall (and almost nothing else) – and some of those punters were there for a second or third go on the game.

A close second as best VR at the conference
Henry: A close second as best VR at the conference

(Note: away from VR on the Lot – and not included here in this article, was my visit to WeVR and playing their Gnomes & Goblins — which I will cover in an upcoming article).