Virtual Set Design

James Connelly is a Primetime Emmy Award-winner, production designer, art director, and set decorator and founder of JP Connelly. The company specializes in pre-vis and digital set design for television and special events. The company prides its self on blending their art directed architectural solutions focused on storytelling with both physical and virtual sets. Their notable projects include NBC’s The Voice and Bravo’s Top Chef, as well as Celebrity Show Off and The Masked Singer. The company uses real-time game engines to produce live sets primarily built around greenscreen stages.

JP Connelly has recently been facing a series of complex design issues accommodating shooting in a pandemic environment where studio audiences are not allowed and social distancing is required on set. The use of large scale complex virtual production allowed programs such as Mayim Bialik’s Celebrity Show-Off on TBS to film in an almost empty stage while presenting an interactive game show.

Fxguide’s Mike Seymour walked through James Connelly’s personal process in this exclusive clip below from the first visualizations of a set to the final real-time UE4 rendering, and some of the process behind the sets for Masked Singer and their other Virtual Productions.

For many people, Virtual Production, especially on otherwise empty sound stages, suggests LED volumes. While JP Connelly has closely looked at LED stages, and done several Music Videos on LED stages, Connelly points out that LED stages can be problematic for multi-camera shoots. “For Celebrity Show Off, it was really efficient to use Green screen during COVID. We needed to make sure Mayim was completely isolated, and we had a very seasoned actress. It wasn’t necessarily for her to necessarily see the world around her (on LED Screens),” Connelly explains. ” I think LEDs will be an efficient way of shooting and it will lend itself to a lot of creativity. But there are still some bugs to work out. The nature of my television is all multi-camera. And so with a large LED set, you have to be very careful about which camera is recording and what, and you can remove the option to edit later as you have to decide the shots from all the cameras Live. So it can actually be a little bit tricky and the solution is an algorithm, I can’t quite figure out yet.”  The production dilemma is that an LED screens driven by a game engine such as UE4 adjusts the image to be correct for the lens. Away from the exact position of the camera, the image on the LED walls can appear warped or incorrect, which means the one LED screen can not do live perspective rendering of the viewing frustums for two cameras at once. The only solution would be to live switch the LED screens as the director switches cameras, which would then both pop the background for the actors on set constantly, and deny the ability to change the edit later as the edit would be locked in.  But Connelly points out with 160 new LED screen stages coming in the US by the end of the year, “there’ll be certainly an incredible amount of inventory for people to keep exploring.” The option JP Connelly had currently exploring is maintaining a master set and just tracking green keying patches on the LED screens for each of the actors, so the environment lighting is correct, but the actual image can be adjusted for each camera in post. “Allowing the LED screen to go green. you got a really nice key, but also for cross shots you can get into the virtual world that they are keyed into,” he adds. “My prediction is if we are going to go to a fully virtual space, we’ll be looking at a lot of mixed media to get that to work, and that’s will mean adding quite a bit of tech.”

An LED set does also not capture quite the same illusion of space virtually that many of JP Connelly’s projects have been experimenting with. The virtual designs are no longer assuming the set is visually built on the ground level of the studio. Celebrity Show Off is a prime example and one of the second or third fully virtual productions the team has done recently. “I’m excited about what we can do,” he explains. “If you were to walk through Warner Brothers or Paramount stages, and you looked inside a sound stage, you can tell what is a film set versus what is a television set. Film sets are typically elevated six or seven feet because the DP wants to get his camera down into the floor and we give them creative freedom. In television, you build right on the floor. You just don’t have that kind of budget.” As a Designer, Connelly is excited about what he can do with an ‘infinite floor’. “We effectively have an endless budget, so let’s make our own city… I have a passion for exploring those limits of depth, negative space, and that definitely dictated the design and the concept for Show Off“.

The company is not only focused on set design, with the increasing melding of cinematography, art direction, and visual effects, especially during COVID, the team is addressing a whole new raft of issues, such as virtual audiences. “We are working on a project right now for Facebook, which has a virtual audience, and it is very important for the host of that project to engage with them as needed and see the action and respond” Connelly explains. Pre COVID many shows gained production value and a sense of being an ‘event’ by having a huge studio audience. Shows would often show wide establishing shots from the back of the studio with seemingly hundreds in not thousands of people.  “In the last six months, that has been our constant discussion, – something that I have talked about every day,” he says. “There is a negative space to fill in a theatre and for thousands of years, even in Greek amphitheaters, you would fill that space with an audience. You feel their energy even more than you even them for scenery. And so suddenly with the pandemic, you don’t have that, so there is a lean into tech.” With the Masked Singer when Connelly first designed the season one set the audience was a ‘character’ in the conceptualization of the program. “They were three-quarters of the space. And when we approached it this season, -, Oh boy, I couldn’t even tell you how many versions we went through to explore what it looked like without audience…, it’s was a challenge. It’s a real design challenge.”

JP Connelly uses the Unreal Engine for its virtual sets and dynamic lighting. The company is keen to explore recent new hardware GPU cards and especially UE5 when it is released. Mainly for its ability to push the limits of what can be done in the real-time render budgets the team has. With regards to real-time raytracing, Connelly thinks it is about a year off being used by his team. “It is still a little early for us, we’re still using some dynamic lights but we’re baking our lights and playing it a little safe, to make sure it works perfectly in real-time.”

Interestingly, when asked for his advice to the VFX community when dealing with Virtual Art Departments and Digital Production Design, Connelly points out that VFX often thinks in terms of shots and keyframes, while production designers think in terms of creating behavior. “We set up productions to create behavior and the shots come out of that. And so it is only in the last steps of our process when we get a chance to really influence the ‘shots’ …but we’re creating the worlds first, to get to those shots, so don’t piss us off !” he jokes.  “It is really exciting when you marry the two worlds, it’s really exciting because I think with VFX, there’s a real lean into some of the technical aspects but there is an artistry of world-building that I think production design and art department bring to the table”.