Framestore operates two worldwide divisions, the commercials unit and the feature film division – this is perhaps a cultural aspect of the company being built on the merger of the original Framestore and the Computer Film Company. In fact, for many years the company was called Framestore CFC, reflecting these two rather distinct focuses.
The requirements for commercials work may appear at first sight to be very much the same as film, but this is not so beyond the same fundamental tools that both teams use. When it comes to project management in particular, short-term small teams doing commercials is really quite different than large film teams working for months if not years. But…working on commercials in New York, London or LA (Framestore offices) is actually very similar and until recently the various offices doing commercials work each operated very differently. Today Framestore’s commercials unit globally manages over 250 staff, most of which (200+) are artists doing commercials work via the same shared database.
The nature of the work varies across the advertising spectrum, from simple short turn around projects to fully-CG immersive VR projects. Each of the advertising divisions, particularly London and New York, grew up independently and worked on their own projects without much cross-pollination between facilities. However, this began to change as client needs expanded and they were driven to share commercial work across remotely-distributed teams.
In 2010, Jesse Lucas, now Gobal Head of Pipeline for Framestore’s commercials division joined the company and saw that there would be much to be gained by unifying the company’s various pipelines, scheduling and asset management. “Several years ago,” he says, “we were increasingly sharing work across facilities in an effort to reduce our company-wide dependency on freelancers which can be very expensive. Part of the strategy behind establishing a uniform pipeline across all offices was making it easy to share work and to utilize the best artist for a given task wherever they may be.”
As Lucas described, he was pinpointing very specific business challenges when evaluating how to deploy a new cross-site pipeline. Those included:
● Get established teams at multiple facilities to align on a single pipeline
● Maximize use of internal resources, reducing the need for costly freelancers
● Reduce time/space inefficiencies of archiving and unarchiving commercial projects
● Automate dependency tracking of projects in key applications including Maya, Houdini, Nuke and now Hiero, with enough flexibility to support everything from traditional TV spot work to VR immersive experiences.
The initiative
Framestore’s original office in London had an internal production management process largely driven by Excel spreadsheets, while an entirely separate pipeline had been developed in New York. More recently, in 2013, as the new Los Angeles office was coming online, Lucas was tasked with aligning all three teams via an efficient, unified pipeline.
“Part of this strategy was making sure it would be easy to share work between the offices so we could always leverage our best internal talent no matter which office a project was booked out of,” explains Lucas. “If we have our best riggers in New York and we have a big rigging challenge on a project, then it shouldn’t matter that the other phases of the project are being done in Los Angeles. Our goal was that the pipeline tooling should make it easy for that rigger in New York to participate in LA’s project.”
Framestore had several long-standing teams with well-entrenched processes in place that were concerned with any potential slow-down that deploying a new pipeline might introduce into their deadline-driven advertising pace. While the film visual effects teams at Framestore had been using Shotgun for some time, the short turnaround demands of commercial work made it more challenging to impose new pipeline demands on the commercial artists.
“After evaluating and coming to consensus on the pipeline needs of each facility,” continues Lucas, “we looked at a couple of other off-the-shelf systems, but they didn’t have the application integration that Shotgun had. Shotgun already had engines for our primary applications, Maya, Houdini, Nuke and Hiero, so it wasn’t a very hard decision to make.”
Unlike the feature film division that approached their Shotgun solution based on scheduling first, the commercials division producers were already using CETA scheduling but it was the asset transfers, shot publishing and archiving that drove the commercial’s team interest.
Bids are done in CETA and when they are awarded, the Shotgun project is setup and the name contains the CETA job ID. The actual tasking uses tasks but for many commercials there is not always line producers (some projects are just too small for multiple levels of producers), so a commercials team members may not be individually tasked the way they are in the features world.
“So it was set up such that instead of having to strictly schedule things in Shotgun, anybody can go into any task and begin working and as soon as you publish from that task, you become assigned to it,” explains Lucas. “The turnarounds are sometimes really quick so we don’t have time always to have someone sit there and fill out the database before anything can be done.”
The implementation
Working from the back forward in the commercials pipeline, they decided to target specific workflow bottlenecks and to approach pipeline deployment in a results-driven way that they could be easily quantified. The first problem was the time and storage space inefficiencies of archiving and restoring commercial projects.
“After finishing each commercial project we were going through a time and labor-intensive process of archiving all the data in the project,” says Lucas. “Even though we knew we didn’t actually need every asset we were archiving, we didn’t know what we did need, so the only safe route was to archive everything. In the purely file system-based workflow, it’s rarely clear on a project of scale exactly which files you really need to keep. You could try to write some clever scripts to go into the work files and draw the connections after the fact, but that approach is complicated and error prone. Our new Toolkit-based pipeline and having the relationships of all our assets encoded in the Shotgun database has really helped us tackle this problem head on.”
Having Shotgun in place allowed Framestore to maintain a dependency graph going all the way back to the beginning of a project. At the end of each project they are able to isolate the final assets delivered to the client and know precisely which renders, models, textures and rigs they need to keep for archiving, and what they can discard. This also dramatically simplifies things on the back end when clients ask for additional assets after a project is complete. These requests require Framestore’s technical teams to restore project files—a process that is significantly faster and easier when pulling from 125GB rather than 3TB of data.
The system is still being developed, but for example in Nuke, which is clearly a hub for so many of the assets coming together, an artist can’t even render “without everything upstream of your output node being published,” states Lucas. “The Shotgun toolkit is like an abstraction layer to help you deal with the concept of publishing files – it is basically an asset management system – but not a traditional approach – it tracks things and allows you to create database entries that are flagged as published files.”
The primary components are now working, and throughout the pipeline, via metadata, all the relations are tracked and recorded for publishing. Says Lucas: “So if everyone stays on pipeline at the end in Nuke, when you make that final comp – you can look at say what lighting renders constructed it, what animation caches were in those lighting passes, and what shader packs were used, and go all the way back down the tree to know entirely what constituted that render.”
The team is continuing to work on fully implementing their Hiero solution. Lucas describes it as working on “getting that last 3% right now.” Basically, the whole pipeline down to Nuke is fully working, and ”the tools we are working on right now are going to export all the latest published clips – bring them into the timeline for all the shots and then export the timeline from Hiero and all the metadata about which comp shots were in that timeline, and then the team will track those Hiero exports as they go to the client.”
Next on on their plan was to tackle aligning artists on a single pipeline to maximize the use of internal resources and reduce dependency on freelancers.
Using internal talent more effectively meant being able to seamlessly move work between Framestore’s London, New York and Los Angeles offices. Lucas and his team built multi-site functionality on top of Shotgun using the dependency tracking that Shotgun’s Pipeline Toolkit provides. “During the publish process we lock files down so they can’t be modified, and that means they’re then safe to be localized to different sites if people want to use them,” outlines Lucas. “The permissions lockdown is key to guaranteeing consistency of those files across sites. When one of our offices makes a localization request for a file, any other files it depends on are automatically localized along with it. Making this work was fundamentally tied to the dependency resolution logic in Shotgun.”
Framestore has a dedicated a team of six developers who have spent a year building out the pipeline and replacing legacy systems that had been in place for 10 or more years. As the project has proven so successful the current two staff in London, three in New York and one in LA is about to be joined by a second developer in LA.
Unlike the Feature film division that has their own Shotgun server in-house, Lucas and his team decided to have a single remotely-hosted Shotgun server for the commercials solution. “When we first started, we were just dabbling with Shotgun,” he says, “and that is what we set up, and it works so that is what we have carried on with. We like that we stay up to date with all the updates, we don’t have to pay some systems guys to manage it, and we don’t have to buy hardware, so it works well.”
The remote hosting and accessibility of Shotgun via its web interface and the Shotgun Review mobile app for iOS also frees artists and supervisors to leave the office as soon as they’ve submitted their renders for the evening, as they know they can easily log in from home later to check out the results. They’re also able to securely make notes and provide feedback on their shots from set, and don’t have to be connected to the Internet in any special way, which also encourages further collaboration.
The implications
Now all of the creative advertising teams across Framestore are running their projects in Shotgun. Every render has a record of its creation, and the artists can review those renders in context using Shotgun Screening Room or RV. As the commercial industry evolves out of the traditional 30/60-second broadcast spot, and into VR, mobile and digital, Pipeline Toolkit has also been flexible enough to support all of those very different workflows. The system integrates from the producers after they have bid the job to the passing over to Framestore’s in house front2 software for client review.
Explains Lucas, “One of the first jobs we finished in New York on the new pipeline was a VR project promoting the film ‘Interstellar’. We had a team of about 20 modelers working together to create hundreds of assets from LIDAR data. It was probably one of the biggest asset build jobs we’ve done in New York, and our Head of CG told me emphatically that he did not believe we would have been able to manage that job on our old pipeline. There were too many assets to handle without a proper database. Having everything tracked in a database and being able to change between asset contexts in a clear way was very helpful.”
More recently Framestore completed a Marvel “Avengers” project for the Samsung VR viewer with more assets than any other job completed in the past. “Rather than having scenes and shots in these VR jobs, you just have tons and tons of assets and environments so it’s great that the Pipeline Toolkit and Shotgun in general can cater to that less-linear situation as well,” says Lucas.
Shotgun is a project or product group that works using what is known as Agile software development, although they did not know that’s what it was called when they discovered their model or rapid prototyping, functional teams and constant rollout actually had a name and was ‘a thing’. Lucas similarly runs his own team in much the same way, “although perhaps less formally”, he suggests. “I run a meeting once a week and we agree on what needs to be done, protocols and naming conventions, etc.” Lucas finds his development style very similar to that of the Shotgun developers – “it is the completely natural way to do things with dynamic languages – I can hear “oh there is a bug ? – I can fix it in 10 to 20 seconds and get a package out to LA from NY in another 20 seconds…”
“None of this would have been possible without the systems team at Framestore,” adds Lucas. About a year before Lucas started, systems got every machine in the company on a change management system called Puppet. Lucas credits the hard work of the systems team with giving him the stable environment to very rapidly roll out the whole new Shotgun system. “Everyone has a VNC server running so we can instantly connect to anybody’s machine and instantly debug things.” This allows the guy who is responsible for say the Houdini Shotgun implementation to service all Houdini users worldwide inside Framestore “instantaneously from his desk in London…I’d say it is very Agile – if I did not know that was a loaded word!” he jokes.
Projects are now flowing smoothly across facilities, and the team via Shotgun is keeping pace with Framestore’s incredible volume of projects that typically ranges from 12-25 commercials, music videos, VR projects and more per month. “I am surprised,” comments Lucas, “at just how well it has gone off to be honest, it was in September 2014 we started NY on the new pipeline, and it was November and December of 2014 that we added LA and London – everybody just jumped on and took off.”
In addition to eliminating production bottlenecks and enabling overall workflow improvements, Framestore is looking towards even deeper levels of integration with some other tools such as Autodesk’s Flame, which Lucas was literally working on as we spoke to him in New York. Lucas will be at SIGGRAPH presenting a short talk on the Wednesday of SIGGRAPH, hoping to share his personal insights “about asset transfers, things to watch out for, keeping things simple, and see if we can’t help people out with an example of how we did it,” he adds.
Finally, Lucas would like to wanted to give a shout-out to the members of his team – Ian Thompson, Karsten Wagenknecht, Jesse Spielman, Dane Bettis, Beau Casey and Christian Dugenio – for all their hard work and dedication over the last two years. “Without them none of this would have been possible,” he says.