In this episode of the fxpodcast, we speak with David Schneider from Union VFX about one of the most interesting new production workflows to emerge around Gaussian splatting: using splats not simply as a visualisation technique, but as a practical crowd solution for feature film production.
Union VFX has been developing the approach in partnership with Clear Angle Studios, building on Clear Angle’s new Volumetric Capture Rig, or VCR. The rig uses 40 synchronised machine-learning cameras in a half-dome configuration, with integrated LED lighting, allowing performers to be captured under highly controlled, neutral lighting conditions. From those captures, Union can generate animated 4D Gaussian splats that retain a strong sense of photographic realism while being deployable inside a conventional VFX pipeline.
The key breakthrough is not just capture quality, but control. Gaussian splats are often admired for their realism, but they also present major production challenges, especially around temporal stability, segmentation, mattes, and relighting. In the podcast, Schneider explains how Union and Clear Angle have built a pipeline that allows the splats to be relit inside CG scenes using production lighting, cast and receive shadows, and be treated by compositors much more like traditional CG assets.
For crowd work, this is especially significant. Traditional CG crowds require modelling, rigging, animation, agent systems, and extensive variation work. Card-based crowds can be efficient, but they quickly fall apart when lighting or camera movement changes. Union’s approach sits somewhere new: captured real human performances, processed as splats, instanced into large-scale environments, and then relit and art-directed shot by shot.
Schneider discusses how machine-learning-based image segmentation allows Union to identify and isolate elements such as shirts, jackets, trousers, hair, and skin. Those regions can then be turned into Cryptomattes, giving compositors familiar controls for colour variation and shot finishing. That means a crowd can be populated with captured humans, while still allowing the team to vary clothing, adjust balance, and prevent obvious repetition.
The conversation also explores how close these splat-based characters can come to camera. Rather than being limited to distant background use, Union has tested characters at close to three-quarters screen height and found that, with correct relighting, they can hold up far closer than many traditional CG crowd solutions. For supervisors used to hiding crowd work behind foreground extras, smoke, atmosphere, or layers of photographic cards, this opens up a very different set of creative possibilities.
A major part of our discussion focuses on Clear Angle’s capture system (we have more on this in an upcoming story). The VCR is designed for high-quality 4D capture in a volume of roughly three metres by three metres by three metres. Depending on the requirement, Union can capture individuals separately for later instancing, or capture small groups interacting with each other, to add more natural crowd behaviours such as cheering, dancing, pushing, or laughing. These interactive group captures can then be sprinkled through larger crowds to avoid the mechanical feel that sometimes affects digital crowd systems.
Schneider also escribes a test involving a cyclist captured on static rollers, then moved through a CG environment as if traveling at speed. The same logic could apply to other kinds of performance capture: flying superheroes, wire work, stunt actions, or other cases where a production may not need a fully traditional digi-double, but still wants a highly realistic, controllable, relightable human performance.
While generative AI can create impressive short clips, it remains difficult to direct, repeat, extend, and control with the precision required for production. By contrast, Union’s splat workflow is based on captured performance and deterministic processing. The director and VFX supervisor can make specific decisions about where people go, how they behave, how they are lit, and how they are varied. In production terms, that control is everything.
Schneider also explains Union’s Nuke-based workflow for fast editorial turnover. After capture, Clear Angle can deliver sprite-like views from the rig, which Union can place into work-in-progress CG environments using a USD-based workflow. This allows supervisors and directors to make early layout and behaviour decisions using elements that correspond directly to the final splat captures. Once approved, that metadata can be passed downstream for full processing.
Rather than treating splats as a research demo or a visual novelty, Union VFX and Clear Angle are showing how the technology can be engineered into a practical, controllable, film-ready pipeline. Listen to the full fxpodcast as David Schneider explains how this new approach may reshape the way productions think about digital humans and crowd work, with captured GS performances.
fxpodcast transcript
We’re testing some ideas here at fxguide and one is publishing a transcript of our podcasts. We use Mistral’s Voxtral Mini Transcribe 2, which is quite accurate, with a string of vfx context bias keywords. We’ll then do a spot check to make sure it all seems OK. Over time, we’ll work to improve accuracy if needed but we’ve found the results to be quite excellent. It is not an exact transcription, as there are cleanups that are done — but it’s certainly in the spirit of what was recorded.
So here you go….
John Montgomery: Hi, welcome to the fxpodcast. I’m John Montgomery. In this episode, we’re joined by David Schneider, VFX and Technical Supervisor at Union VFX, where they’ve been doing some really interesting things with Gaussian splats. Mike and I actually met with David in person while we were at FMX in Germany and found what they were working on to be really impressive, so we’re having him on the fxpodcast to share with you all.
Working in partnership with Clear Angle Studios, Union’s developed a pipeline that captures performers as relightable 4D splats, enabling large-scale crowds without traditional modeling, rigging, and animation workflow. David discusses with Mike how the team solves key challenges, including relighting, segmentation, compositing, and scalability — all while exploring what this technology can mean not only for crowds, but for the future of digital doubles and virtual production workflows. It’s really cool stuff, so let’s go ahead and get to the conversation now.
Mike Seymour: David, thanks so much for joining us. Really appreciate it.
David Schneider: Nice speaking to you again, Mike.
Mike Seymour: Yeah, we recently saw you in Germany at FMX.
David Schneider: That’s right. I was giving a talk on our work on Black Mirror, but we also had the opportunity to talk about some new developments at Union.
Mike Seymour: We’re keen to talk about that today — though of course we’ve talked about Black Mirror in the past and I love that stuff. But what I was really enthusiastic to discuss is the work you guys have been doing with splats. Do you want to outline what you’ve been doing and why?
David Schneider: Yeah, absolutely. So we are currently in production on a major feature. It’s being supervised on the client side by our co-founder and executive VFX supervisor, Adam Gascoigne. As they were in prep for the project, Adam knew that there was going to be a fairly heavy crowd requirement, and so he tasked a few of us in the office with looking into new technologies and seeing how they could be applied to create more realistic crowds. A technology we’ve always been interested in since it came on the scene is Gaussian splatting. Adam was particularly interested to know if there was a way that we could use it to construct crowds — so that’s something we’ve been looking into for the better part of about a year now.
Mike Seymour: So when we’re talking about doing a crowd, to be clear, you’re not talking about just running a drone over a normally huge crowd. You’re talking about basically crowd generation.
David Schneider: That’s right. We wanted to go from the ground up. Normally, you would begin a CG crowd by building your models, then rigging them, then animating them, and then you have your agents that you use to make a large crowd. We wanted to see if it was possible to take captures of, say, one, two, or three individuals doing something — walking, dancing, or just standing idle — and use those captures to form a crowd without going down the traditional route of modeling, rigging, and animating.
Mike Seymour: Okay, so most people probably know this, but just to make sure everyone’s on the same page — we’ve got basically a point cloud when we do a Gaussian splat, in the sense that it’s a bunch of points, but we can think of them as having almost soft splats at each point so that the points kind of connect up with each other. The trouble is, while it looks photoreal and it really does look amazing and is incredibly robust as you’re moving a camera around, it isn’t polygonalized and it isn’t like a temporally coherent thing that a polygonalized model would be. So I guess that begs the question: if you’ve captured a few people, what can you actually do with them?
David Schneider: Yeah, so this really did form the base of the challenge that we knew we had to get around if we were going to make this work for our crowds. As you say, there are well-known issues with Gaussian splats — particularly 4D Gaussian splats, the animated kind — where you don’t have temporal stability from frame to frame, which can present itself as a bit of noise or jitter. You don’t have things like consistent point counts, which can make it difficult to do traditional workflows like UVing. We knew all of this getting into it — that’s a very well-known limitation of the tech. But we wanted to take a holistic view and see if, throughout the VFX pipeline, there were ways to mitigate these issues using a mixture of well-known and newer techniques.
Fundamentally, Gaussian splatting is a machine learning technology, so we also looked at ways in which other machine learning models could help us. We came up with a variety of solutions. There are really good denoising solutions that are fairly mature now, as well as some new ones on the market. What we’re doing is really a blend of CG and 2D techniques to mitigate the issues we’ve been discussing.
In addition to that, because you are essentially capturing data with no information about what is actually contained in the capture, we’re applying new machine learning models that can do image segmentation. What that allows us to do is estimate where the various parts of the models are in the splats — we can isolate things like clothing: jacket, trousers, shirt, hair, skin. We can get all of that from various angles and then project it back into the splats, at which point those pieces of information can be rendered out as cry. That takes us back to a very normal CG/2D workflow, whereby our compositing team can take those mats and do color correction, color variation, and really treat these like standard CG assets.
Mike Seymour: Okay, so let’s use the example of a football stadium. A football stadium has crowds on north, south, east, and west. The sun is coming from one angle, so the crowd on one side is being hit on the left-hand side of their face, and the crowd on the other side facing them has the sun hitting their other side. If you were to do cards — which is an old-fashioned approach for crowds — that doesn’t work, right? Because they were filmed with the light hitting them all on one side, and when you put them on the other side of the stadium, the light’s wrong. How do you get around that problem with splats?
David Schneider: Absolutely. That was one of the fundamental challenges we knew we had to overcome, or there was no way we were going to be able to put this technology into production. It did take us a while. In concert with Clear Angle — who we have a very close partnership with, and with whom we have co-developed a training pipeline — we’ve figured out exactly how to train these splats and how to process them in CG and 2D to enable true relighting. This is something we have proven within the studio, proven to our clients, and demonstrated in some of the material you’re going to be showing.
We do have the ability to put these splats, captured in completely neutral lighting conditions, into any CG lighting environment. To use your example, they can be scattered throughout the stadium — we can put, say, 20,000 or 30,000 of them into stadium seats. We would then apply our CG lighting to the CG stadium environment, and those same lights in the same render scenes, using the exact same production renderer, will also render out the splats with the correct illumination and shadowing. The splats will shadow each other, shadow the stadium environment, and the stadium environment will shadow them.
Mike Seymour: Okay, and then if I’m doing replication, I’m using these image segmentation techniques to say, let’s vary the T-shirts a bit, vary the trousers a bit so they don’t all look the same — I presume?
David Schneider: Correct. At the point where we ingest the captured material — once it’s been delivered from Clear Angle — we perform that image segmentation and reproject it back into the Gaussian splatting data. So once we produce our final CG renders, our compositing team has the ability to go into the shots and decide if they need more variation. They can use the cryptomattes to quickly and easily vary the colors throughout the scene to make sure that people don’t look like they’re just copies of each other sitting right next to each other.
Mike Seymour: So tell me about the Clear Angle side of this in terms of what you were using to capture these splats. It’s 4D, right? So you can’t just be running around them with a camera — you have to be putting them into some kind of large rig.
David Schneider: Exactly. Around the time we were doing our investigations at Union, Clear Angle had also been looking into Gaussian splatting. They’d done a fair amount of research and development, some partnership with NVIDIA, and they were starting to produce some really impressive results. They knew that in order to get the best quality possible, they were going to have to construct a new rig. So they’ve built what they’re calling the Volumetric Capture Rig, or VCR. It’s a very impressive half-dome rig that uses machine learning cameras, which are gen-locked so they run completely in sync with each other. The rig also has an integrated LED lighting solution, so you can dial in whichever lighting you want — or in our case, just have completely flat and perfectly neutral lighting.
Mike Seymour: So basically it’s giving you the ability to capture one or two people at a time — how many could you get at once?
David Schneider: That depends on what your objective is. The capture volume is about three by three by three meters to get pin-sharp focus on everybody you’re capturing. We have a couple of different scenarios. There is one where you may want to capture multiple individuals and have the ability to process them as individuals — in which case you’d probably have a maximum of two or three standing with a bit of space between them. The reason for this is that occlusion is not your friend. You want as many cameras in the rig to see as much of your subjects as possible, so you need to leave space between them.
The other scenario we’ve been using is where you want that interaction — a small crowd of people pushing and shoving, dancing, laughing, arms around each other. You can absolutely do that too. You wouldn’t be able to separate them and use them as individuals after the fact, but it really does help to make a believable crowd when you have those kinds of interactions sprinkled throughout.
Mike Seymour: So let’s say I captured my crowd and I wanted to give them pennants or hats or something to wave — like a flag — just to break it up even more beyond just the people themselves?
David Schneider: Yep, absolutely. During capture, you can have those on stage if you want to — you could hand out props at capture time. But after the fact as well, it would be very simple for us to identify where the hand is, where the head is, and augment the captures with things like putting caps on or putting flags in their hands. That’s something we can absolutely do.
Mike Seymour: Because once you’ve done a Gaussian splat as a three-dimensional thing, you can put a 2D element with it. And these days you can do relighting from 2D, but you can also do segmentation — and once you do the segmentation or the relighting, that technology gives you surface normals. Maybe they’re not 100% accurate, but they’re pretty close. Once you know a surface normal for a hand or whatever, you really know the orientation that any prop would be at. It’s like a jigsaw puzzle — one plus one is equaling three here because you’re building on multiple techniques.
David Schneider: Yeah, absolutely. And one of the benefits of the training pipeline we’ve established with Clear Angle is that our splats do have very high-definition surface normals. We don’t actually need to go through an intermediate step of trying to estimate them after the fact. Once we bring our splats in, we have those normals available to use.
Mike Seymour: So you guys did a bike rider — a guy on a bike. Why was that a good idea? What were you trying to prove or test?
David Schneider: Well, Clear Angle has a member of staff who is an avid cyclist, so he was ready to go and was totally up for doing a test capture. We brought him in, placed him in the middle of the rig, and put his bike onto static rollers, which let him pedal as fast as he wanted without moving anywhere. What we were trying to achieve with this demonstration was to make it look like somebody was convincingly moving through a CG environment, even though they were captured in one place inside the VCR rig. I think that’s something we did achieve looking at the demo material — we were able to estimate the speed at which he would have been moving had he not been on rollers, and then apply that speed to transform the splat through space in the CG environment.
Mike Seymour: Are you getting a reasonable mat from the splat? If you’re putting them in a CG environment, a bike rider is probably pretty clean in terms of shapes — but you could have somebody with hair. Is the splat going to have any artifacts or ringing because the mat isn’t coming from a green screen, it’s coming from the capture process?
David Schneider: Yeah, correct. Because you’re essentially doing something like a structure-from-motion process — starting off building a point cloud by looking at all these different cameras simultaneously and estimating points in space — the whole process inherently has depth baked into it. That does make it easier to isolate your subjects from anything happening in the background.
As far as very fine detail like hair goes, you can absolutely dial up the number of splats and the size of those primitives to capture finer detail if you need to. And then on top of that, it’s a very holistic approach — we also have our traditional 2D techniques to do a bit of cleanup and make sure we have as much detail as possible.
Mike Seymour: So let’s get back to my example of the crowd. The numbers you’re throwing around — 30,000, 40,000 — that’s the sort of thing you might have in a major film with a big crowd scene. How performance-friendly is it to have that many replicated splats? Is this going to weigh down a shot and make it really long to render?
David Schneider: No, it’s actually lighter and more efficient than you would think. By the nature of splats, they’re essentially primitive shapes in a certain arrangement. You can treat a crowd as instances of instances — a single character would be instances of a single primitive building up that character’s shape, and then a crowd would just be instances of that. So the memory usage is a lot lower than you’d expect.
For render times, the material you’re going to be showing was rendered on Karma XPU, which gives us the ability to spread renders between machines with CPUs and machines with GPUs. When we run on graphics cards, these renders are very efficient — we’re seeing render times well below what you’d expect. On a GPU-enabled machine, a 2K quality frame would probably be done in perhaps 20 minutes.
Mike Seymour: And how close could the camera get to the crowd? Like, do these things only work if you’re sort of 10 meters away, or can you get a little closer?
David Schneider: You can get a little closer than that. The distance we’ve tested — I probably couldn’t tell you precisely in metric, but I would say we’ve had our characters about three-quarters of the screen height away, which in camera terms means the camera was maybe a couple of meters from them. And I think they held up very convincingly at that distance.
Mike Seymour: Yeah, I mean, if you’ve got somebody on the other side of a fence and you’ve got a principal actor coming onto the field, you can really have them in the shot without having to worry. Once you get the lighting right they’re going to sit in the shot so much better — and the inherent realism of the splat gives you just an enormous push as well.
David Schneider: Yeah, absolutely. Traditionally when we’ve looked at doing CG crowds for productions, everybody’s had an assumption baked in that you’re not going to be able to bring this crowd too close to camera — you’d probably want to bury it behind a few layers of photographic elements or on-set extras. We think that requirement has now gone out the window, because the extra realism you get from this, especially with our relighting technique, means they hold up very well — completely unobstructed and near camera.
Mike Seymour: So are you seeing this as a solution to crowds, or are you actually seeing it as a step towards doing a whole bunch more with splats? What’s your personal opinion on the breadth of where Union could take this?
David Schneider: Absolutely for crowds — that’s something we have in production right now and it’s working beautifully. Extending on from the example of the cyclist, I’m thinking about things like a superhero film where you have a lot of shots of a superhero flying. Think about the way they used to do that — the original Superman, where it was just somebody lying on a blue screen. I’m thinking there’s now a modern version of that where you could have somebody suspended on wires in the middle of Clear Angle’s rig, with a fan on them to emulate the wind blowing. You do a long capture with a lot of performance of flight, then process that into a splat, relight it for whichever environment it needs to be in, and drop it directly into your shots. That could actually mean you don’t need a digi-double anymore for sequences like that — you’d have a much more convincing character that could hold up with whichever cameras you wanted.
Mike Seymour: The interesting thing is splats are sort of machine learning, but it’s not like hardcore neural stuff the way a NeRF is. It’s interesting to think about how it fits into a pipeline — you’re very much taking the real world as it is, and it looks photographic, but you’re freeing up the camera’s point of view. That sort of begs the question — you’re doing neutral lighting, but some splats have view-dependent characteristics. Have you worked with view-dependency, where, for instance, a specular highlight would change relative to where the camera is?
David Schneider: Yeah. That’s something that doesn’t really come into play with the relighting framework, because obviously it makes the captures look fantastic when you’re looking at them in the conditions they were captured. As part of that capture, you’re getting reflections and specular highlights from whichever lights were around them — but that’s something you specifically want to avoid in a relighting workflow. You don’t want any evidence of the lights that were on stage when these people were captured.
If you were doing a shot where you knew exactly what the lighting was going to be and there was no relighting required, you could capture under those lighting conditions and absolutely use that data to give view-dependent effects in the finished product — and I think that would look fantastic. But I think the real move forward in what we’ve been doing is just that ability to free people up from the lighting and make that decision later.
Mike Seymour: Yeah. In my stadium example, you don’t get a lot of view dependency on a crowd — you get those on cars, but not really on people. And as you say, for a crowd facing either side of a stadium, they need to be lit consistently anyway. So what sort of reaction have you had from supervisors and people you’ve shown this to? I think you had a demo in London?
David Schneider: That’s right. We held a small event at our office in London in partnership with Clear Angle. Clear Angle brought a section of the rig and set it up to show people how it works. We gave a short presentation and showed our demo reel of the work we’ve done so far, and we were very, very pleased with the reaction — it was everything we were hoping for.
Obviously this has been under wraps inside Union and Clear Angle for quite some time, and we had a fair bit of trepidation putting it out into the world and opening ourselves up to feedback. But I was very pleased. We’ve devoted a lot of effort and resources into developing this workflow, and it was very gratifying to see supervisors and producers with the wheels starting to turn — thinking, these are new possibilities, this is something I haven’t seen before, I’ve got a production coming up, maybe this is the solution to what was causing me problems. I had a lot of people approaching me afterwards, kicking ideas around and saying we have to do this. It’s not every day that you feel like you’ve been able to show experts something they haven’t seen before.
Mike Seymour: So for somebody thinking about this and wondering — what about just doing this in Gen AI? I think we need to unpack that a little. We were talking before about how splats aren’t temporally consistent in the sense that it’s not like the same hand appears in every frame moving forward. But in a Gen AI scenario, most people generating footage are limited to relatively short runs before it goes off the rails — and even then you get weird artifacts where hands come and go. Even though in a sense you have no temporal consistency, you could run a Gaussian splat as long as you wanted and you’re not going to have any of those problems of deviating, collapsing, or drifting off model — because the process doesn’t have that restriction Gen AI does.
David Schneider: Correct. You can take your captures for as long as you want — we’re currently doing 60, 90, 120-second captures, no problem. You can really run it as long as you want until you run out of storage.
With generative AI, one of the issues people tend to encounter is that the system doesn’t really have a great understanding of what it’s generating, so you start to see strange artifacts: cars that suddenly reverse direction, hands that pop in. What we’re doing is much closer to what I would call a deterministic process — you always know what the outcome is going to be. It’s very well tuned, very well dialed in, and there are no surprises. You can keep capturing as long as you want and you will get that splat at the other end. There really isn’t much risk there.
Mike Seymour: Yeah, directable and controllable are the Achilles heels of Gen AI. While it kind of looks good for a random demo piece, it doesn’t give you precise control. With your methodology, if the director wants to move a few people because one corner looks a little too crowded, you can just move anything you want — it’s very art-directable and dramatically directable.
David Schneider: Yeah. And on that note, to facilitate that process, we’ve created another workflow inside of Nuke. After a capture session, Clear Angle essentially delivers sprites to us — if you think about it, they have the perfect sprite machine: 40 cameras running, no motion blur, well-lit from every different angle you could possibly want. We can select some of those angles straight after the capture and have them sent over with a rough roto pass just to get rid of the background and give us just the character by themselves.
We’ve then built a USD-based workflow in the latest version of Nuke, where our compositors are able to take those sprites and lay them out into our work-in-progress CG environments. This means temps can get into editorial very quickly, and the VFX supervisor and the director can make decisions about the crowd using the exact elements they will end up having once the splat processing is done. They can get used to seeing those characters right at the beginning, decide on what sort of behavior they want in different sections of the crowd, and then we have the ability to export all of that metadata and send it over to Clear Angle for processing.
Mike Seymour: Brilliant. I can really see how that works in a production flow. Hey Dave, thank you so much for taking time to walk us through that. It’s really interesting tech.
David Schneider: It’s always great speaking to you. Thanks for taking the time.
John Montgomery: That’s some really interesting work, David. Thanks so much for taking the time to fill us in on what you’ve been doing with Gaussian splats — this is the kind of stuff that’s going to make the tech really useful moving forward. Well, that’s it for this episode. For Mike Seymour, I’m John Montgomery. Thanks for listening to or watching this fxpodcast.