How the hell do you manage all this stuff?

What is the average pattern in a facility doing high-end work? Increasingly, it involves moving the images into data from either film or tape and then having the images remain as data until output. Tape is a linear interchange format. From TVC to Feature films the desire is to scan or telecine the footage into data and never use video tape. This is being accelerated by such products as lustre – Discreet’s new advanced workstation grading system, and by the

Many facilities out there today are recognizing the need for centralized storage, but many are unsure of the technologies that are (a) available, and (b) the ones which apply to their needs. There are basically two shared storage models in wide use today – storage area networks (SAN) and network attached storage (NAS). In a historic context, SANs have been deployed for speed, and NAS systems have been deployed for simplicity.


The center of a new wave of solutions is SGI (www.sgi.com), and smaller companies such as Maximum Throughput (www.max-t.com).

SGI is back in fashion, while most people think of SGI in terms of Onyx and Octane workstations – their non entertainment industry business is built around servers and massive data management issues for government and the miltary. Their new focus is on graphical data pipelines. This in no way reflects a move away from graphics workstations but rather that the infrastructure needs of a major facility now completely exceed NT or simple LINUX solutions. Because of this, SGI has become very sexy once again to serious feature film post-production facilities, doing 2K and higher, feature film work.

While SGI handles all formats from SD to 35mm film resolution data, smaller companies such as Max-t are targeting HD in particular. Max-t like SGI realised that the solution is not just moving data but moving huge amounts of image data. Like SGI they are focused on not just a computer solution but a solution that delivers SD and HD I/O as an intrinsic part of the infrastructure solution.

As a direct result of the rapid shift in facility structuring, Discreet has had to modify its plans and entire product line to be cheaper, more open and integrated with these powerful third party solutions.

Discreet

Discreet has become extremely aware of the move from dedicated storage to integrated facility strategies and at NAB announced a major policy shift.

This summer Discreet will add switchable storage to its high-performance infrastructure product line-up, comprised of stone disk arrays and advanced stoneFS file system-including new hardware RAID options, wire high-speed networking, backdraft job management and video I/O, and mountstone cross-platform file sharing.

Discreet is fundamentally evolving the way it addresses facility infrastructure. Consequently, Discreet’s products will now be designed so that they are open and can talk to a wide range of different file systems and applications. To achieve this goal, Discreet plans to add support for standard file systems to all its applications. The first Discreet system to demonstrate these capabilities will be lustre, but all systems products will soon be vastly more open to third party solutions.

“The real value for customers today is the ability to effectively integrate all the different applications used in a post-production facility, not just those of a single manufacturer,” explains Marc Petit, vice president, systems product development for Discreet

Switchable storage is an advanced, high-performance storage system designed to instantaneously move large amounts of data between systems. Rather than copy the media between two systems, the stone disk arrays are switched automatically using a virtual ‘patch’ panel. This provides the advantage of allowing rapid transfers of terabytes of data while maintaining peak high-resolution (HDTV or 2K, RGB) bandwidth guarantees.

In addition to slashing by half its own storage options pricing, the company is also unveiling its infrastructure sparks program, designed to bring choice to clients via leading third party storage and networking technologies – including initial partners Maximum Throughput (Max-t) and SGI.

Max-t

“We are extremely pleased with Discreet’s new infrastructure sparks program,” says Mike Hughes, vice president of business development at Maximum Throughput. “It allows us to work closely with Discreet to deliver solutions that better suit our mutual clients’ needs for advanced collaboration and file sharing.”, he says, “having a firm like Discreet representing our products in the field is great validation for what we’ve done” he comments. “With this relationship, we’ll be able to jointly and effectively address a large number of the workflow issues that our collective customers face every day of the week.”

The Max-T product offering Is Sledgehammer HD!O, a network attached storage device (NAS) that has also Incorporated SD and HD resolution video I/O and playback capabilities. NAS systems are built specifically as file servers, and the server side focus for Sledgehammer has been speed. Sledgehammer will deliver a sustained 150-180MB/sec. of throughput onto or off of a Gigabit Ethernet network, without accounting for any cache effects (which would only increase performance). Testing has shown that on a per client basis over NFS, performance ranges anywhere from 30MB/sec. through to 100MB/sec., depending on the platform in use, and the underlying client operating system.

With Sledgehammer HD!O, for example post-production companies can have a video data network bridge. Sledgehammer HD!O enables any facility – with any system, running any OS – to centralize the input & output of HD material. And it leverages the world’s fastest NAS system, Sledgehammer HD!O allows fast facility-wide access for efficient, collaborative workflow, using uncompressed HD – at great speed ( a single Sledgehammer HD!O delivers over 240 Mbytes/s of network throughput), while scaling to 16 terrabytes in size. Acting as a centralized storage pool for all facility clients (including rendering clusters), Sledgehammer HD!O can also act as a virtual DDR for 2D and 3D artisits alike. Sequences of images (.tga, .tif, .sgi, etc.) of any resolution (up to 2K) can be immediately played back to a broadcast monitor, at full frame rate, with no format conversion. Sledgehammer also has full EDL tools for capture and compile work. Real-time HD/SD video capture, playback and recording make telecine-to-network a one-step process. One can capture HD data once and share it anywhere in the facility – without encoding or compression – all networked users can access the same data at the same time regardless of the client platforms in use, including Windows, UNIX, Linux, Mac OS9 and Mac OS X.


Max-T @ Asylum

Just over a year ago, Asylum Visual Effects of Santa Monica, CA, set out In search of a centralized storage system to support their fast-growing 3D department. The four main requirements for this system were (1) the ability to share all content amongst systems running different operating systems, (2) the ability to deal with data sets that varied wildly In size (from small setups and textures to large Images), (3) the ability to cost-effectively access the content from a rendering cluster, and (4) the ability to access the data at very high throughput rates.

Tommy Hooper, Director of Technology at Asylum, commented ” Until recently, NAS had never been an option for us,” said Tommy Hooper, Director of Technology at Asylum. “We simply needed more speed than was ever cost-effective to buy using other NAS technologies. Max-T has changed the market dynamics with Sledgehammer. We can now reap the multi-protocol benefits of NAS with no compromises. We evaluated many systems prior to making our decision, and Sledgehammer was the clear winner.”

Asylum employs a Sledgehammer to the back end of a Maya-driven 3D department, with the 3D artists working on systems of varying configurations. Artists develop setups which can be either rendered locally, or handed off to a Linux-based rendering cluster. In either case, all data resides on the Sledgehammer (which is RAID5 protected, and as such, all information is safe from disk failure). thus eliminating the historically time consuming and wasteful practice of first copying data locally and then copying back finished work.

Max-T at Jim Henson’s Creature Shop

Resolutions just seem to keep getting bigger and bigger. Jim Henson’s Creature Shop in London recently took on a project for Universal Studios Japan entitled “Sesame Street 4-D Movie Magic.” The stereoscopic film is part of a new ride attraction at the theme park.

Steve MacPherson, systems architect at The Creature Shop commented, “Since Sledgehammer HD!O was put into production at Henson’s it has remained in production. It has not been rebooted nor have any operational delays been experienced. We run a variety of other servers and at various times have had to devote attention to them. We now have 3D operators singing the praises of an NFS server. Max-T’s Sledgehammer in particular has proven to be an impressive workhorse and contributed significantly to 4K stereoscopic production at Jim Henson’s Creature Shop.”

At 4K resolution, a single minute’s worth of material is on the order of 100GB. So the need to have a huge amount of storage in support of such a project is key, and a single Sledgehammer NAS system will support up to 32TB of capacity which can be added on the fly without ever bringing the system down or reconfiguring the network.

Aurelio Campa, Technical Manager of computer graphics at Jim Henson’s Creature Shop added, “We deal with a huge variety of digital imaging projects including high-definition TV, digital film effects, stereoscopic theme park rides, and file sizes up to 54MB per frame at 30 frames per second. We needed a shared storage workhorse that provided the enormous bandwidth required to simultaneously feed multiple clients, and a render farm. Max-T has pushed NAS into a new realm, providing SAN level performance with reduced management issues and rock-solid reliability.”

Creature Shop has in fact just recently added a second Sledgehammer which doubles the file level throughput to which the artists and clusters have access. The new Sledgehammer simply appears as a new shared volume on the network.

In summary, Sledgehammer HD!O allows fast facility-wide access for efficient, collaborative workflow, using uncompressed HD. Acting as a centralized storage pool for all facility clients (including rendering clusters), Sledgehammer HD!O can also act as a virtual DDR for 2D and 3D artisits alike. Sequences of images (.tga, .tif, .sgi, .dpx etc.) of any resolution (up to 2K) can be immediately played back at full frame rate, with no format conversion. Sledgehammer also supports EDLs for batch capture. Real-time HD/SD video capture, playback and recording make telecine-to-network a one-step process. One can capture HD data once and share it anywhere in the facility – without encoding or compression – all networked users can access the same data at the same time regardless of the client platforms in use, including Windows, UNIX, Linux, Mac OS9 and Mac OS X.

SGI

IRIX operating system and the XFS file system started 21 years ago. About 6 years ago, as customers started to want to share huge amounts of visual data SGI started on it’s CXFS (Clustered eXtended FileSystem) program, which allows each application in the facility to share the same high speed data and not have to copy local copies to each workstation seat. Today CXFS works with Windows, NT & Windows 2000, Linux 32-bit, Linux 64-bit, Solaris and by year’s end OSX.

Part of SGI’s solution is their Data Migration Facility (DMF) software for Storage Management.  Briefly, DMF manages disk, tape robot and tape library.  It maintains a database of every file and has enormous capacity (40 billion file, 18000 Petabytes) and it allows the operators to “see” the files no mater where they are.  If they’re on the shelf – the operator is prompted to load tape xxxxx, if it’s in the near line tape robot – it automatically is migrated to the disk cache.  And of course if it’s on the disk – you get it immediately.  A variety of tape robots are supported as are various tape drive formats.

This new approach of total fast control and data sharing has been embraced by leading film effects and post-productions all over the world from Laboratoire Eclair in Paris to WETA in New Zealand.

SGi @ Laboratoire Eclair

Digital techniques have become an integral part of film production. A legendary film processor has become a digital leader; Laboratoire Eclair was established in 1907. Its renowned laboratories and studios in France, continue to develop films and shoot and produce 35 mm prints for release to movie theaters. In recent years, the company has expanded into digital services

In 2001, Laboratoires Eclair digitally post-produced 11 feature-length films. Laboratorie Eclair has moved from analog film postproduction to a fully digital production and postproduction workflow.


Eclair now does frame-by-frame scanning and digitization of 35 mm negatives, using SGI technology to perform restoration techniques or add special effects and to transfer the digital frames back to film. SGI workstations and servers run the Kodak Cineon system, Discreet flame, flint, inferno, and smoke systems, and Nothing Real Shake software. To handle the data load, Eclair needed an architecture capable of moving huge volumes of data among the scanners, the workstations, and the film recorders.

“We’re moving toward a completely digital system because we know that sooner or later all postproduction will be entirely digital,” says Philippe Soeiro, who manages the digital side of Eclair. “We therefore had to design and build an architecture that covers the entire process.”

Many of the films that move through Eclair’s facility require multiple digital processes that can include 3D effects, compositing, calibration, repair of damaged frames, and the removal of extraneous frame elements such as cables, poles, and scaffolding. The number of films that require this kind of treatment is increasing. To provide all these services, Eclair must move enormous quantities of data. Scanning a single frame at two kilopixels per line (2K frame) yields an uncompressed 8 to 12MB file. At that rate, a 105-minute film requires about 2TB of storage space, and the information has to be accessible at high transmission rates. Eclair initially used the NFS protocol for data storage and movement, but NFS limited available bandwidth to 14MB per second and frequently resulted in bottlenecks and delays. Eclair, with an eye to its time sensitive customers, looked for a better way to move data. The solution: a SAN featuring an SGI Origin 200 file server running SGI CXFS (clustered extended filesystem).

Eclair implemented CXFS on its SGI Origin 200 server, which means that all files stored on the RAID can be stored on the RAID can be shared at fast transfer rates over the SAN by Eclair’s 12 hosts. CXFS unifies the available disc space on the SAN. “The combination of the SAN and CXFS gives Eclair the performance of a local disc with the advantages of Network storage” says Soeiro. “It used to take several hours to import a single complete shot. We can now do it in 10 minutes. We are able to work more interactively and respond more quickly to our clients.”

The high-speed SAN architecture with its Fibre Channel technology solved many of Eclair’s technical problems. To ensure there is no break in workflow of postproduction, the workflow needs the fastest asset-sharing mechanism hence the SGI CXFS on a storage area network. With 200MB per second, the SAN is a perfect way to move large amounts of data through a workflow.

For file sharing, a SAN is superior to network-attached storage because it optimizes the size of the data-transfer packets in a range from a few bytes up to 200KB. With NFS, the maximum packet size is 1,512 bytes, which produces a higher system load and longer transfer times. At Eclair, the SAN operates as a digital assets server with superior performance, enabling Eclair technicians to work at high resolution with 2K images and at 24 images per second with minimal wait time for asset transfer.

SGI @ Weta

On The Fellowship of the Ring, Weta Digital first used SGI DMF to manage 100TB of data from approximately 10 million files, which range from small to extremely large. A file can consist of an element, a texture, one version of a shot, or a completely rendered image sequence. Adding the data from The Two Towers doubles Weta’s information storage to 20 million files. Approximately 230TB, representing the first two films’ worth of data, is now managed by SGI DMF.

“We rely heavily on DMF. DMF is running on one Origin 2000 system and we recently upgraded it to 12 400 MHz processors. The key objective is to free up as much disk space for the artists as possible. We use SGI DMF to offline the data from the online disk storage to tape storage.” comments Scott Houston, Chief Technical Officer, Weta Digital.

The StorageTek L700E robotic library now has six LTO [linear tape open] drives and four DLT drives. In June last year WETA migrated from DLT to LTO, which gave greater capacity on the tape cartridges and faster tape cartridges. “That gave us more capacity on nearline; we went from about 25TB available to about 75TB potentially available” he adds. “The 300 artists working on the second film were moving 1TB of data in and out of DMF every day. Being able to migrate the data seamlessly between online and nearline and then back again has been absolutely critical. We couldn’t do that without SGI DMF.”

Weta Ltd. has already begun repurposing its assets stored by SGI: The Two Towers video game was released a month before the film and Weta is currently working with Electronic Arts on an action game based on The Return of the King. “We need to have access to all the files, including files that are two, three, four, or almost five years old, and we need to be able to have access to these as well. We keep everything, and keeping and managing that is going to be a challenge. There are also opportunities for repurposing some of those assets, and that will be essential for Weta in the future.” he notes.

Weta Digital’s workflow for The Two Towers exemplifies the explosion of data necessary in the creative side of the film industry. Increasingly, management of complex data in the digital content creation and cinema mastering aspects of the business expand the need for more powerful tools to manage that data. The choice of SGI DMF on the highly scalable SGI Origin server family allowed Weta Digital to meet the challenges of the trilogy’s movie release schedule and deliver another Academy Award winning blockbuster film.

While infrastructure has long been on the unsexy side of the visual effects business, increasingly it is the edge in being able to bid medium to large scale projects, and with both studios and directors starting to understand how that can translate into meeting deadlines and maximising on screen visual effects, the spot light has shifted behind the scenes and will continue to do so for perhaps the next 3 to 5 years as the entire industry faces the move to full data workflow.