Rhythm and crowds: Moneyball

Crowd replication is a staple of the invisible effects category, for almost obvious reasons that the shots are designed to be in the background but still give life to scenes in the film. More than 120 crowd shots feature in Moneyball, which tells the story of baseball club the Oakland A’s, and how its general manager Bill Beane (played by Brad Pitt) turned the team around. Rhythm & Hues devised a procedural technique with real extras and a Houdini pipeline to fill out the Oakland A’s stadium.

Rhythm’s traditional crowd generation and replication approach would have been to shoot multiple 360 degree tiles of the stadium with extras sitting in different parts, and then patch them together. A different technique, however, was adopted for Moneyball. “Given the amount of time we had to shoot them,” says Rhythm visual effects supervisor Edwin Rivera, “and the fact that the stadium itself was a big bowl – which meant that every seat was at a different angle – I went for something different.”

For wide views of the stadium’s crowds – the main shots – 50 different people at four different angles (90, 66, 45 and 22 degrees) were filmed. Using Rhythm’s compositing environment, each person was isolated from each of the angles on a 1K card and could then be placed procedurally using Houdini into the stadium based upon the angle of the seat and the angle of the tracked camera.

Visual effects supervisor Edwin Rivera filmed a proof of concept test for the technique, using Rhythm & Hues employees as extras jumping around in A’s uniforms, combined with Canon 5D footage of the stadium. “It was like a worst-case scenario,” says Rivera, “in that it was hand-held and rotated about 270 degrees…”

“We used that to help us figure out how many people we needed to make it feel random in the crowd and how close can we get to these two-and-a-half D people before they start to give up the gag.”

“If you use the traditional method of shooting different sections, you can quickly see a pattern forming when there are 65,000 seats in the stadium,” notes Rivera. “Our new method allowed us to randomize the people so that there was no procedural pattern, and also populate it fairly quickly. The other advantage was that when we shot it with each pass, we took them through seven kinds of emotions which would start on a bloop light. We knew that frames 1000 to 1500 had them jumping up and down and screaming, say. And in 1750 to 2250 they were sitting down pretty sedate. We had 20 seconds for each emotion. So we have a pretty reliable library now of any person at any angle – we could have them do any different number of emotions.”

– Above: a progression of the crowd work from original plate to final shot.

Crowd plates for wider views were shot on digital Genesis cameras, which helped increase the speed of processing, although extreme close-up views were captured on film to match the principal photography. A LIDAR scan of the stadium helped Rhythm to track the camera movements and know where each seat was.

In terms of the Houdini set-up, artists could dial in the size or density of the crowd, and also the various emotions being expressed by the extras. “For example,” explains Rivera, “at any given time, an entire stadium of people aren’t doing the exact same thing – so we could say there were 50 per cent of people sitting down bored, and 15% up booing, then 10% cheering and so forth. That allows the director a greater amount of flexibility that we wouldn’t have had typically with an entire section of extras.”

– Above: Original plate through to final shot.

The crowd system had been used in part for a bullfighting ring crowd on Knight & Day, although that relied mostly on proprietary tools. For Moneyball, artists used Houdini mostly out-of-the-box, although a number of tools were created to manage the large amount of data created from shooting multiple extras angles. Says Rivera: “Houdini actually had a lot of the tools we needed, but we did write a fair amount of tools internally to keep track of the tools themselves. With fifty times four different angles per person, you wind up with 600,000 frames and then when you make the individual 1Ks for each of those scans, there are another 1.3 million frames. It was about 2 million frames online before we even got any plates!”

Another important part of creating the photoreal crowds was adding details, not just random movements and emotions from the extras’ performances. “There are so many extra things layered into our scenes like people walking around, waving flags, just sitting down,” comments Rivera. “Initially, Houdini would be able to populate the stadium very quickly after tracking. It would only take a day or two to come back with a render of a bunch of fans, but it became a matter of making it look like a believable crowd and not a bunch of extras acting on directions. So we shot a bunch of people walking around, milling about. We created some digital signs to pepper into the crowds randomly. The people at some points become so small that you don’t really need them holding the signs per se. There’s a lot of things about the size of the shots and the scope that allow for some cheats, as it were.”

Lighting the baseball crowds also proved a challenge, since DOP Wally Pfister had created a unique look, especially for the night game scenes, that Rhythm had to match. “Typically you would have six or seven huge bays of lights with a bunch of other auxiliary lights at a game,” notes Rivera, “which means any given person is lit pretty much the same in any given spot in the stadium. But Wally created a much more stylistic lighting scenario. Let’s say the actor was in the foreground and the bank of lights to the right were on – typically he’d turn off just about all the lights on the left hand side. That sets up a challenge for us, because not only do we have to match the mood of the crowd, but also the lighting.”

In Houdini, artists also developed a procedural lighting ‘cheat’ for all the crowd shots. “Firstly, we knew exactly which lights were on for any given shot,” says Rivera. “So we used Houdini to set the 3D lights position and distance from any particular person on any particular card, and got an off-set of the person’s 2D matte ever so slightly. The compositors could then use that off-set to create faux lighting on the human. We were also able to use that lighting pipeline for any of the day shots as well, since we had shot the extras with fairly diffuse lighting.” The same procedural matte off-set trick could be used to sell shadows from the stadium overhangs, again with the help of the LIDAR scan.

“I know every shot they we worked on, but having seen the film I have the sense that we worked on only five shots,” laughs Rivera. “You get so caught up in the film and the principal actors that you forget the screaming crowds in the background weren’t there when we shot it. I much more enjoy the challenge of the photoreal visual effects than the fantastical, partly because they’re harder.”

All images copyright  © 2011 Columbia Pictures. Courtesy of Rhythm & Hues.