Burn, baby burn! Discreet ships Burn

Distributed rendering for the discreet effects products was first demonstrated at the NAB Discreet Users Group meeting several years ago. It has taken a couple of years, but version 1.0 of burn running on Red Hat 8.0 has been released. Burn gives artists the ability to render flint, flame, and inferno batch setups (including action within batch) on network-connected systems running Linux Red Hat 8.0.

Distributed rendering for the discreet effects products was first demonstrated at the NAB Discreet Users Group meeting several years ago. It has taken a couple of years, but version 1.0 of burn running on Red Hat 8.0 has been released. Burn gives artists the ability to render flint, flame, and inferno batch setups (including action within batch) on network-connected systems running Linux Red Hat 8.0. Our coverage includes an overview of the exciting new product, comments from users who have been testing the product over the last several months, and a set of benchmarks which show that burn can be an extremely effective tool to add to the mix at facilities. We take a full look at this new product in five sections: Overview, Hardware, Software, Burning, and Performance.

Overview

How effective is burn? Very. It is a version 1.0 product, and there are issues to be considered which will be highlighted later. However, after using it over several months artists at SOMERsault (my home base) quickly saw how useful it was. There were times we were fighting over getting time on the burn renderers. The ability to keep working while rendering is of particular interest during client-supervised sessions, especially those that involve working with HD imagery. The benefits of using burn are also echoed by Elad Offer, Creative Director/Partner for Money Shots in Santa Monica, who feels that “the ability to work on a shot then send it off to burn rather than watch the paint dry allowed (him) to at least double (his) throughput. With the current budgets getting smaller and smaller, it is an absolute life saver.” Testers who have used burn over the last several months have most certainly become fans.

These sentiments are echoed by Nathan Robinson, Senior Visual Effects Artist at Ntropic in San Francisco. “The ability to render things in the background and still work in the gui is a huge timesaver….this has especially been useful for HD and film work,” according to Robinson. “Clients have always complained how long it takes to do HD jobs and now I can improve throughput by utilizing burn. Simple things like setting up a bunch of color warper nodes and setting them off to burn while working on some of the more intricate shots in the foreground is a huge timesaver. I have also noticed the biggest plus is sparks.”

As artists became more and more familiar with burn, we also discovered which type of renders benefited most from submitting jobs to burn, as well as which type of renders to avoid. Burn integrated extremely well into the network topography at my facility. Our sgi systems are interconnected with a HIPPI network, but we used existing 100BaseT wiring to connect to burn. I was quite surprised that file transfer times over the network did not have a huge detrimental impact on the performance of burn and larger format files actually had the greatest benefits. Upgrading to gigabit ethernet will certainly help in the long run.

Pricing for burn has been set at $5,000US per software license (per render node, not processor), or $16,000US for 4 nodes. The licenses can be floating so they are not fixed to one specific piece of hardware — this will allow a lot more flexibility for facilities with large render farms.

There is only one recommended hardware configuration — an IBM xSeries 335 machine –, which can be purchased from discreet or from IBM directly. Hardware support for burn is not provided by Discreet, but two software support contracts for burn are available. Wire is also required when using burn, but Discreet has a discount program for facilities that purchase burn.

Discreet recommends a maximum of 8 render nodes for SD processing and up to 20 render nodes for HD/Film processing. After that point, there is a diminishing return on rendering speed (and investment).

Burn essentially consists of three parts…the discreet FFI sgi systems, the Windows 2000 PC-based backburner, and the Linux-based burn software. Jobs are submitted from the IRIX systems to the backburner manager which runs on a network connected PC. From this software, jobs are distributed to the various Linux burn systems. The Linux render node communicates directly with the sgi systems, grabbing setups and clips as needed and rendering the clips directly to the IFF framestores. The backburner software that runs on Windows is also utilized for combustion and 3dsmax rendering.


Sparks must also be purchased for each Linux burn system — Sapphire, Tinder, and SpeedSix have all announced that their sparks will be available for burn. In our benchmark testing, sparks stand the most to gain from the horsepower of burn (see below). It is critically important that burn sparks packages are affordable since facilities will need to purchase burn licenses for every node. In our view, prices for network rendering should be at the very least similar to other plug-in network rendering licenses such as Shake or AfterEffects. Better than a per-node charge, a minimal site license would be the ideal situation since sparks packages have already been purchased for the main workstations. Facilities large and small can’t afford to be continually charged higher rates for sgi/discreet solutions versus the identical plugin sets for more affordable PC workstations. Tinder will be available for $1000 per node and SpeedSix Monsters for Burn will be $1000 for the entire set (all boxes). Sapphire sparks are more expensive, with a price of $2000 for the entire set or $1600 per node if four or more burn licenses are purchased.

Hardware

When burn was first shown publicly, the units included OpenGL hardware. This appeared to make sense considering the amount of processing that is done utilizing the sgi graphics cards. However, in the final shipping product this has changed and the reliance is fully upon the processors. The processors utilize the Mesa graphics library, which handles software-based OpenGL rendering. This improvement is critical because even though there is only one officially supported hardware configuration for burn, the renderer will work on many in-house Linux render systems already installed at facilities around the world.

According to Maurice Patel, Product Marketing Manager of Discreet systems, using OpenGL cards did not prove feasible because PC cards were limited to 8-bit. “In the end,” Patel comments, “we got a better quality result using Mesa and it also meant that we could use servers for rendering without them having to be populated by expensive OpenGL cards.”

The recommended hardware is an IBM xSeries 335 machine, which can be purchased directly from discreet for around $5500US or directly from IBM. These are the systems that we have at SOMERsault, since we did not have an existing Linux render farm in-house. They work extremely well and are a very solid product.

Money Shots is running their burn software on custom-built dual 1.2GHz Pentium4 CPU Linux boxes, which is part of an existing render farm for 3D. They also use 100baseT for network interconnection, and according to Offer, “it works perfectly. Why pay a lot when you can get away with a cheaper solution?” They have also developed some custom scripting which stops the Linux backburner server daemon when a 3D render begins and restarts its once the 3D render has completed. It isn’t the perfect solution, since it always stops the burn render for a 3D job, but it is certainly a way in which the two can coexist on the same hardware.

Installing and managing Red Hat and the burn software is not as straightforward as installing effects apps, so it is important that facilities have an experienced IT person or at least know that there is a bit of a learning curve to get started. In our case, it took several days to figure everything out since our IBM x335 systems came with blank hard drives and no real instructions on how to get things up and running. Luckily, our sysadmin did have some initial Linux experience which helped out tremendously. There will be a definite learning curve for facilities that don’t have experience with Red Hat….for us it was: “Kickstart file? What the hell is a kickstart file?”.

Discreet has created improved installation instructions based upon testers feedback. They now include a Linux kickstart installation — this is a file that automatically installs the needed Red Hat components on the systems. Some of the more obscure modifications that need to be done to some of the Linux system files are also now documented in the burn users guide. In the end, users should simply be aware that it is certainly not as simple as plug and play. Because of its UNIX-base, it is similar in many ways to IRIX installation and troubleshooting.

Users will also need to download Red Hat 8.0 kernel 2.4.18 (since it is not the current release), or purchase a full copy so there is a full set of CDs and manuals. Red Hat Linux 8.0 Professional is available through Amazon as well…buying the CDs through this link will also help support fxguide.

Software

Several software items need to be installed on the various systems…..

  • FFI sgi system – burn client : This allows the sgi system to communicate with the backburner manager that runs on a Windows system. Installation on the sgi system is quite straightforward, and doesn’t even require a restart.
  • Windows 2000 system – backburner : backburner Manager is the controller of network rendering and is also used for controlling distributed rendering for combustion and 3dsmax (which use Windows systems, not the Linux systems). The install also includes the backburner Monitor, an application that allows users to monitor and set priorities for renders.
  • Linux system – burn, stone and wire : burn and the backburner server are installed, which handle the actual rendering of batch setups. stone and wire for Linux is used to grab frames from the remote framestores and then save rendered frames back to the effects system.

Since the monitoring of the burn renders is done from a single Windows 2000 workstation, it is helpful to have remote monitoring set up in the effects suite so that artists can view the status of their submitted renders. For this need we ended up installing vnc on the windows system, which allows users to view the desktop on other computers:

An alternative system is to use rdesktop, which is available at www.rdesktop.org.

Let’s Burn

Rendering a batch setup is as simple as hitting the burn button.

When this happens, FFI does several things. First off, a copy of the setup file is saved in the /usr/discreet/burn directory. It is given a name of “Burn_host_date_time” and it is this name that shows up in the backburner Monitor window. Files are kept in this directory so that the burn renderer always knows where to find the files.

Next, FFI creates a library in the current project called “_Burn_” (if it hasn’t already been created). Two reels are added to it. One is named with the name of the setup, followed by _input. This is where the source clips for the render are kept. A second reel is created in the library (named with the name of the setup, followed by _output) and a placeholder result clip is created. If you view this clip in the library before or during the render, it is filled with scrambled frames that are not representative of the render. As the setup renders these scrambled frames are replaced with the actual frames of the render. Discreet is aware that a more elegant solution is possible (such as creating a black clip or labeling frames as “unrendered” similar to the desktop edit reel) and will be potentially improving this in the future. (Click for a larger view)
Finally, the job is submitted to the backburner Manager (click on the image for a better view of the interface). As mentioned earlier, we use VNC to monitor backburner. The monitor allows you to view the various jobs that have been submitted, check the progress of renders, see the average time per frame, change the priority of jobs in the queue, restart burn jobs, and monitor for errors on the burn systems. (Click for a larger view)

You’ll notice when rendering that the first frame on each burn system takes a considerable time to render. This is probably due to the fact that this time includes loading the setup from the remote system as well as startup time for the burn renderer. However, after the first frame is completed, the time to render will be shortened considerably. Burn will render one FFI job at a time and not distribute frames from various jobs at the same time. If one job needs to be rendered more quickly and move up in the queue, it is easy to change the priority of the job. Rendering will stop on the current render, the new setup will be rendered, and when completed the previous job will continue rendering where it left off.

Rendering on burn is fairly robust, but the stableness of it seems to diminish as the size of the images being fed to it increase. I’ve easily done 20 or more NTSC renders and multiple HD renders in a row without any problems. However, there are definitely times when the burn systems run into problems and will unexpectedly quit. This seems to happen much more often with higher resolution images — the larger they get, the less stable burn becomes. It also seems as though when several effects systems are continually submitting jobs, they seem to crash the manager or the burn systems more often than a single system does. Most of the time the linux servers re register automaticallly, but sometimes the server app needs to be manually restarted.

On the positive side, the backburner Manager keeps track of the systems and will redistribute the processing of frames as needed in order to finish the task. It is also quite easy to set up a couple of UNIX aliases on the FFI systems which will go out and quickly restart the burn software (actually the backburner servers which run on the burn systems). Once they’re back up and running again, the backburner Manager recognizes this and will allocate tasks to the newly discovered systems.

Because FFI creates new libraries, reels within the libraries, and setup files on the system disk (in /usr/discreet/burn), it is necessary for users to do a bit of manual labor in cleaning up files and clips. One way to simplify this may be to create a cron job which nightly deletes files in the /usr/discreet/burn directory which are older than 3 days. It can be quite useful to have some of the files still residing on the system, because if something happens during a render you can use VNC and the backburner Monitor to resubmit a job to burn without doing so from the FFI system. Clip library management must be done manually.

Performance

In general, we’ve found performance on the burn systems to be quite outstanding. As facilities become more familiar with rendering on burn, they’ll discover what works and what doesn’t. As a starting point, here’s a general overview of keys to what we’ve discovered performance-wise:

  • CPU-based tasks stand the most to gain. While this may seem obvious, it is clear that processor-intensive tasks run the best on burn. So if you like using Sparks, the ColourWarper, and the ModularKeyer you’ll see a nice speed increase. Sub pixel rendering on Sparks (such as Sapphire RackDefocus) happen much faster on the burn systems than my inferno 4x400Mhz IP27 system.
  • Higher res images gain the most speed. While NTSC/PAL images work very well with burn and allow the artist to continue working while rendering, higher res images benefit the most. This is probably due to the fact (in our testing) that the use of Sparks and other processor intensive tasks were even speedier on the high res images. With more intense calculations, the burn systems really shined.
  • 100baseT networks can be acceptable. We were quite surprised that our 100baseT network provided great results with burn. Performance would definitely be improved with an upgrade to the discreet-recommended gigabit Ethernet, but in the meantime we’re sailing along quite well. NTSC image transfer would probably stand the most to gain, since the rendering to image transfer time ratio is so low.
  • don�t try to render optics. If you have a batch setup with optics, don’t try it. Unless you’re heading to have a liquid lunch. At HD resolution, optics takes way way way longer to render on burn than it does in batch. Enough said.
(Click to view) Fxguide has created a PDF document with several render speed tests which we completed on Onyx2 IR3 inferno vs. a 4 rack unit burn render farm. While not fully scientific, we did try to figure out some of the general guidelines for what caused burn to shine and in what instances we thought it might be better to stick with sgi batch renders.
What about image quality? Well, there certainly are differences in renders between systems. In the renders we’ve completed, the differences are really no better or worse than those between inferno, flint, and flame. Have you ever tried to render optics on a flintSE? Or compared an Onyx2IR render with a flame MXE render? If so, you’ll understand. One interesting idea to help with this is the additions of a �Burn Preview� button within batch so that users could preview one frame of a burn render to make sure hardware-based effects translate in the correct way to burn.

According to Patel, Discreet recommends:

  • If possible, render everything on burn for guaranteed continuity consistency
  • Always render entire shots on burn – don’t render part of the same shot on burn and part on IFF,
  • Don’t render one frame on one system and one frame on another and juxtapose them
  • Similar shots should be rendered on similar systems – significantly different shots should not be a problem

Again, your mileage may vary — differences have been perfectly acceptable in most situations, it is simply important that artists are aware of the possible differences.

Concluding Notes

Burn, even with its rough edges, is poised to be a hit for artists and therefore Discreet.

“For the most part I am extremely excited about the benefits of burn,” says Ntropic’s Robinson. “When looking into the future I feel this software helps us to be more competitive in our market place. Having burn is almost like having another flame in house that is just doing rendering at a fraction of the cost. Unfortunately it still gets pricey when you start adding all of your favorite sparks.”

At SOMERsault, we’ve quickly come to rely on burn as an additional tool. It really does make a huge impact on HD jobs, since I can keep working on scenes while rendering other material. I don’t think I could have gotten a recent job done when a client moved the deadline at the end of the day up 3 hours. Instead, even with a couple of restarts of the backburner server, the job was done and the clients were happy.


Particle Render – burn
Difference between burn and inferno (gain 1000)


Difference between burn and inferno (gain 1000)