Scrum in VFX

At FMX, the Cinnamon VFX team recently decided to speak about their continuing project of growing and perfecting their company. For the independent Eastern European based Cinnamon – beside all the normal technical things, software tools, etc – the most important parts that they needed to solve was project management. At the German conference they presented their move to Agile to a packed room of curious VFX artists and TDs.

Background

Many people will assume that there is a ‘correct’ way to schedule a project, and in most cases that would align with a model of planning in pre-production, and then scheduling, revising and adjusting to the delivery deadline. In reality, this model is just one of many ways to run a project . Some would argue it is not the most healthy approach. At fxguide we did a lengthy story with Oscar winner Ben Grossman discussing the problems with VFX production, some of which pivoted on how projects were scheduled. Many people think that project management is the same as project scheduling. If you are forced into lengthy overtime it often seems like the overages must be a symptom of bad management, but it may not be. Management is like anything else, given the wrong tools, the task can be nearly impossible to pull off. The software industry has been moving to a new model for sometime, but at fxguide we have seen little evidence of these new ‘agile’ approaches being widely adopted in VFX.

In Cinnamon VFX one finds a company actively exploring new approaches and advocating a very different way to run, even a short run TVC project.

Cinnamon VFX is trying to find a scheduling and planning solution that

1. involves their employees,
2. reduces waste/over time and the ‘panic – crisis’ mentality,
3. allows for planning that is more accurate and in turn more profitable,
4. is respectful of staff and clients, and
5. works for short term and long term projects.

Agile

Agile is an approach to project management that has proven enormously successful in software development. In fact many large VFX houses use the Agile approach for their internal software development, but until now the approach has had little traction in actual production. This is surprising, given how closely and similarly software development tracks to VFX production. In both cases the final goal is only defined to a point, there is a lot of exploration to find the details of what is really wanted. VFX and software both suffer from not being able to specify the finer details before production starts. VFX involves vast amounts of money and huge amounts of creative staff, working to strict delivery deadlines. Like software engineering, the workforce in VFX is highly skilled in producing a digital product at the end of the day. They do this with varying levels of user/client clarity of direction and in an almost entirely workstation based environment. Teams in both Agile and VFX are cross functional with a criss cross of accountability and reporting lines into management, customers and teams. An environment artist answers to their department, but also their producer – as well as to the VFX supervisor and the Director. While they may work in the Environments area, they could be on an entirely different film than another in the same department.

Perhaps most significantly – VFX  has always struggled with deadlines, workloads, burnout and the stress of delivering something that “no one has ever seen before” and to a deadline that is normally immovable. VFX deadlines are often worse than software deadlines as they are fixed in a calendar by millions of dollars of promotion and marketing activity. In the VFX world, missing deadlines is rarely an option.

At the other end of the production spectrum, commercial work faces irregular work volumes, fast turnaround and aspirational quality hurdles. It would seem VFX would have turned to the success that Agile project management has produced in other areas to see if a version of Agile could help in VFX.

One company has: Cinnamon VFX was founded in 2006, by Vadim Konov and Alex Prihodko. One day the two were talking about finding the ideal studio to work at, and in the end they decided they should just build one themselves. The company is based in Ukraine and has been very successful. They mainly work on TVCs but also on some major international feature films such as Where the Wild Things Are, Diana, Zero Theorem and Brighton Rock. The company has worked with major facilities such as The Mill, Framestore, and Liga01. 

Their venture was so successful they got busy –  very busy. 

“We were in serious need of some help.” explained co-founder Alex Prihodko. “We all had tasks, there were a lot of small projects and you’re invited to meeting after meeting – everyone was busy – the clients loved us and so you end up working 24/7! It was chaos.”

Looking ahead, the team decided there had to be a better way to run a VFX company, especially a smaller company during relatively short run TVC style work. This is their story moving from chaos to a world of increased billings, happier staff, freer weekends and less overtime.

The problem: What was the best way to run a modern VFX company?

They decided to start with the theoretical aspect of project management, before shifting to the real world issues of implementing any models in the harsh light of competitive post-production.

In particular the issues of production planning and scheduling, while dealing with uneven workloads. 

As the team loved their work, and they were always looking at visual ways to solve problems, they turned their attention to the problem of project management. Up to this point, the focus was just on creative but clearly if they were going to continue and even expand into new areas such as VR and immersive storytelling – then they needed to be more focused on the business or run the risk of burning out their staff.

Options or approaches the team could take…

The founders explored a wide range of models such as traditional “Plan-and-Document” methodologies like Waterfall, Spiral, or Rational Unified Process (RUP). RUP dates from the late 90s and in some ways was a precursor to Agile. It relies on:

• a tailorable process that guides development/production,
• custom tools that automated the application of that process,
• services that accelerate adoption of both the process and the tools (such as visualisation tools).

The most common approach in our industry is really a version of the Waterfall approach that assumes if we could just properly plan before we start, and then define the problems clearly before we start work, etc, then all the chaos of a modern production would be solved. The spiral model is a superset of the Waterfall – that incorporates a ‘risk-driven process model generator’. Based on the unique risk patterns of a given project, the spiral model guides a team to adopt elements of one or more process models, such as waterfall, but equally it could promote incremental or evolutionary ‘rapid’ prototyping. In this world the post house would adapt and change based on risk and the type of job, sometimes constantly temp-ing shots to clients – but sometimes rigorously defining specification via technical previz first, for example.

The typical approach to these styles of scheduling are a Gantt Chart where the various tasks are estimated by type and they often overlap. Dependencies are not immediately evident and complexity is hidden in what seems like an ordered graphical representation. Gantt Charts are perhaps the most basic scheduling visualization tool used in our industry today worldwide.

Problems with current tools

For all their widespread use, there are many issues with Gantt charts,

• the dependencies are not clear until the system fails.
• it assumes accurate estimating can be done over the life of the project.
• it tends to be part of a top down process where both management and the very nature of the project management is a waterfall with decisions and actions flowing down from one layer to the next. It therefore can lack buy-in from team members.

Just estimating duration is difficult when viewed from a higher level task point of view. Each major task is assigned a gross duration, but a better approach may be to start with even units of time and see what can be done in each. The diagram above shows this reversal of thinking. Instead of seeing these tasks as 20,22, or 27 day tasks… the team could see which of these tasks can be done in a day or less. Prihodko explains “You cannot make a prediction of 1 scheduling task, but you can predict
 1000 of tasks! Yes, it could sound strange, but anyone can make a prediction. You cannot make a prediction of one large task in a schedule (accurately), but you can predict 1000 smaller tasks, one by one. For sure it’s harder to make them, when they only last 2 hours or less. But if you take a look at longer distance – it’s possible. Isn’t that how the Navi in your car works?”

There is no sense of progress inside a Gantt chart item or the traditional scheduling of how a task is running internally. Until the day it is meant to be completed the progress inside the large item is oblique. Some tasks take a long time to get results, others seem to get most of the work done quickly but the final polishing tasks are vastly longer proportionally. A traditional model can accurately say something is happening now –  but on a reasonable size unit of time – how accurately does anyone know how much of the task is actually completed. Estimates are normally very wrong.

The Agile option

The team at Cinnamon VFX ended up on Scrum and Agile Development. Before outlining the Cinnamon solution it is worth commenting (as we noted in the introduction) that many post houses around the world use some Agile project management in their R&D units. At Siggraph last year, fxguide spoke to Weta Digital which used a version of Agile for the product development of Manuka, Weta’s impressive new Renderer. At Eclair Labs in Paris, since December 2015, the R&D team has adopted Agile under the direction of Cédric Lejeune, VP of Technology for Post production Services at Eclair | Groupe Ymagis. He strongly favours the approach for R&D and in the six months since its introduction the team has been successful in its use. Thomas Mikota, (Avatar, District 9, King Kong) has just finished 6 months designing and implementing such a system at a Chinese VFX studio.  His model combines:

• Agile (learned from his experience developing software)
• Test Driven Development
• Scrums and Sprints (modified for film)
• Kanban (a key visual ‘job production’ board and another framework used to implement Agile)

He did this based on management techniques he learned on films such as Avatar and “a lot of ideas taken from the Toyota playbook when it comes to lean manufacturing,” he commented. “Ideas that come from the literature around lean startup theory.  I come at it from the systems and software design point of view because people like magic more than they like meetings about management techniques.  The biggest problem is people really don’t understand just how broken things are. There’s no one evangelizing these ideas, because the people (like me) who understand them are working 60 hours per week getting productions completed”. he explained.

In the world of software development, if one goes to any major equipment supplier from Autodesk to The Foundry, you will almost certainly find some kind of Agile implementation. The issue then is how to utilise Agile for creative production, not just software development. Is it possible to transfer these ideas to production, especially short run TVC style work, which seems a long way from large scale software engineering?

Cinnamon’s solution:

In the Cinnamon Agile system, there is no overlapping long and interdependent tasks, just a series of short sprints with small team managed tasks. The team meets daily and everyone tracks the work of the current Sprint normally on a whiteboard or pin board (Kanban style tool).

The whole project becomes a series of short sprints – even on a three week TVC project.

Everyone defines these smaller units.  Creating progress daily is possible as the task units are small. If something is blocked, the team is there to help, and the task is not so big that the optics on the progress are lost. It is very important to note that, as any member of a good Agile team would know, the model fosters a mutual support structure, if you hit a roadblock on any given day – the team knows and offers help if they can.

The “cnm Framework” above makes it possible to understand an exact project status in 3-4 words or a simple picture. This works for every employee, no matter how new, whether they are an artist, a producer or a client. One doesn’t need to be expert to see where they are. The framework can be different for different types of projects, for example “Full CG”, “VFX Commercial” or “Feature Film”. But the principle is the same in each case.

“Your particular lifecycle can’t appear from a textbook, a team needs to analyze and optimize their own production processes,” explains Konov.

Central to this Agile approach is the Sprint. In larger projects a sprint may be 2 to 4 weeks, less on a TVC. All that the team is concerned with is what is in the current Sprint. The product backlog defines what needs to be delivered to the ‘owner’ .

When estimating workload, if planning horizons are reduced to something approximating 4 days of work, then a project can avoid getting off track. The planning unit is small enough that by adding one more person for a day, one can normally correct the schedule (…assuming the planning unit you start with is only a nominal 120 hours). Of course, the team does not assume 100% efficiency. Rather something closer to 60% is practical once all the other demands of work are factored in – from meetings, to backups, phone calls, someone being sick, etc.

Rather than looking at Gantt charts, the team look at Burndown Charts. This is a graph of a fixed number of tasks in a Sprint, As the Sprint runs, the chart is burnt down as the work is done. Thus, even inside a Sprint, one has an idea as to how well it is tracking and why it might be not on schedule.

The planning is done as a team with people voting on complexity (Sprint poker). The small tasks in the Sprint are managed daily and in the open with a daily Standup(normally in front of the whiteboard/kanban) and there is both a Sprint review where tasks or “stories” are discussed and a retrospective about what was good or what did not work that feeds into the next Sprint. Note that retrospective is not at the end of the project but at the end of each Sprint. Some post teams struggle to even have a retrospective to learn from their experiences after an entire project. Here the retrospective is done regularly during production.

The estimating process is a good window into the practical nature of the system Cinnamon has adopted. Using what is called Sprint Poker, people in a team estimate how long a small task will take. The only options are 0,1,2,3,5,8,13,21,34 Infinity… the idea is that as a task grows so too does estimation error. It is therefore impossible on anything over a day or two to be very accurate, saying a task will take 19.5 days implies a level of accuracy that just does not exist. This process has many advantages:

• estimates are not implying accuracy they do not have
• team members buy-in – they ‘own’ their own estimates
• if one person estimates wildly differently – then perhaps they know something others don’t
• as the process is repeated and reviewed per Sprint – the team quickly learns and improves estimates

Once a sprint starts the team enforce the golden rule that there are “no changes when sprint starts”. This avoids the team having moving targets and the project being constantly in flux, a very common problem with many TVC projects the world over. 

Any uncompleted tasks are known as the ‘backlog’.

“We develop our normal team with normal workload for main typical projects”. comments Konov. “We also defined and locked the amount of people in the team working on a sprint and aim to set the same workload per sprint.”

All aspects of Cinnamon VFX now use Agile, they even structure their outsourcing work to fit in. When the process was being rolled out, there was naturally some doubt amongst some artists, but as deadlines were met, and overtime collapsed under the weight of good planning – returning weekends and time to be creative – team members begged to be included.

The Holy Grail of planning is connecting multiple projects so the company now has a more even workload and every sprint it still 4 or 5 days. This is coupled with exclusive periods that contain no overlap so each project can be given the team’s full attention and the very best chance at being creatively successful. This may seem like an impossible dream to many, except Vadim Konov, Alex Prihodko and the team at Cinnamon VFX who seem to be actually living it.

The path of Agile is a reaction to the project, not a project following a prescribed path. It does not pretend there is a master plan to be followed, it assumes small steps with constant revisions and checks, it assumes everyone’s involvement in an evolving implementation.

Thomas Mikota, (ex Weta, Asylum and Pixomondo) speaking independently from Cinnamon VFX, summed the situation up regarding the industry at large. “Our industry is in great need of a change.  I’m incredibly interested in seeing how we might work together to bring that about as we find other like minded individuals to begin spreading the word.”

Judging from the tremendous reception that the Cinnamon team had to their experiences at the FMX conference and the reception their ideas got, it can only be hoped that other companies will explore Agile in a production environment.