The tech challenges behind the industry awards process

Creating an awards program to recognize outstanding Post, Visual Effects and CG contributions with meaningful awards is involved and challenging. These industry awards honor specific artistic and technical elements of films, TV shows, and other creative content, and need to be inclusive and accessible to inspire the whole community.

Challenges include creating a user-friendly process for submitters, addressing the issues of judging the growing multitude of source formats, transferring large files worldwide for submission and viewing, and the security of sensitive content. Developing and managing a fair and inclusive process for submitting and judging entries is primarily the responsibility of the awards committee’s. Given the often subtle or hidden nature of CG and post-production work, there are procedures for entering sample scenes and making-of clips to communicate how the work was performed. Means of judging and nominating entries varies, but includes committee vetting of the content, worldwide nomination events and wider membership voting on-line.

The technology described in this article has been used on the industry’s top awards programs including ACM SIGGRAPH’s Computer Animation Festival and the Hollywood Professionals Association (HPA) Awards, playing a key role in increasing the breadth of content and submission reach in order to engage a more diverse and worldwide audience.

Format choices

One of the challenges with creating these processes is the ever-increasing variety of formats, each having its advantages and proponents. Every artist wants their work to show its best, but allowing any codec would require an involved conversion process, as submissions ultimately need to play on a single timeline and display spec to be judged alongside one another. The main requirements for submission codec are; viewable on a theatre-sized screen, open and accessible to submitters, and capable of being transcoded to other formats.

The consensus is that submissions should be limited to one or two formats; having a single format negates any need for a transcode process for viewing at events, and two formats may balance entrants’ capabilities better without overburdening the process for the awards organizers. Currently the main formats in use are; Avid DNxHD36, which gives a good balance of quality, accessibility (DNxHD is a SMPTE standard) and ability to be transferred over lower bandwidths, and ProRes 422 HQ giving a higher quality picture and option for facilities that don’t have a DNxHD pipeline.

Utilizing web and cloud technology

Web browser advances have played an important role in improving the submission process, via web services and media upload tools. The popular Python-based web framework Django is used to deliver a web form for submitting information about an entry, and white-labeled versions of Sohonet’s file transfer and cloud storage products, FileRunner and FileStore respectively, are used to receive and store the content. This allows a no-install, no-plugin approach to uploads aimed at breaking down the barriers to submission by creating a secure (HTTPS) uploader free of requirements such as software purchases or having a sysadmin install plugins and alter firewall policies.

FileRunner is essentially a web application that facilitates upload to the object store, which is an Openstack Swift deployment. One of the key advantages is the redundancy – it creates three copies on upload and has built-in geographic replication. Prior to using Swift, having a fixed server deployment inevitably meant that uploads slowed significantly on deadline day—the dedicated infrastructure approach is not cost-effective in an environment with essentially one peak period per year. Instead, a cloud implementation gives the potential to scale to the peaks of upload demand (known as horizontal scaling) by pulling in resources from the rest of the cloud.

Quality Control and Content Vetting

One of the major problems that technology has helped to overcome is ensuring media is in spec and to an acceptable level of quality. Submitters are not necessarily aware of the involved processes and viewing environments and therefore don’t always appreciate the importance of the format and technical quality of the work they submit. Manual vetting of entries is a cumbersome process, checking each clip against a spec and requesting re-submissions against an extremely tight timeline proved to be a major problem.

To overcome these issues, browser technology is heavily utilized to help automate the QC process. By using HTML5 and JavaScript directly on the submitter’s web browser, the header of the media containing the metadata can be read prior to upload to check much of the spec. As this is done prior to upload, the process saves a lot of time for submitters, and almost all issues can now be reported in this step, including discrepancies in resolution, video and audio codecs, bitrates and the required container format (Quicktime).

Automated metadata check after uploading.
Automated metadata check after uploading.

The browser is also now used to automatically generate a hash (md5) of the file based on its content directly on the submitter’s computer. This allows automatic and verifiable version tracking, which can be retained with the media all the way from the upload process, vetting pipeline, straight through to delivery, ensuring the original remains pixel by pixel exactly as intended.

Judging the visual content of work requires an accurate representation; one of the harder challenges of the QC process has been aligning the color levels of the entries. Getting this right has been critical in avoiding crushed blacks or washed out material at events and when transcoding for the on-line view and vote. Color levels can take time to get right in a pipeline with just a few different sources, and generally requires each source be tested through the pipeline to check the end result. When you have submissions from hundreds of sources in an environment where there is little time for re-submission, this becomes very challenging.

Solving this problem was approached by creating a custom and automatable method of measuring the color levels on submitted media. Metadata was the obvious first test but was found to be misleading: Clips marked as Rec.709 often didn’t translate to color levels adhering to that spec. The reliable method that was found was decoding the actual pixel values from the essence of the media, and then feeding these through a custom filter using the Filtergraph functionality of FFMPEG. The filter outputs the percentage values of pixels above and below thresholds, giving the information required to mark clips as video range, full range, or most challenging, highly variable ranges where elements have been composited over each other in the making-of clips.

The result of the filter is then fed back to the original submitter who can choose to correct to the spec or allow automatic correction by Sohonet by using it as an input to encoding of proxy media for on-line versions, and import settings for playback systems.

Tracking Entries

Tracking the status of entries
Tracking the status of entries

Another challenge has been tracking the information associated with entrants, submitting facilities, balloting, and vetting processes involved in checking the content of the media. As examples the content must adhere to the rules of the award they are submitting for, be in the correct category, and usually presence of company logos is not permitted. API’s are heavily used to automatically sync submitted information from the Django database created by the web form and other tools used by the awards judges and committees. For instance the VES Awards Committee heavily use Shotgun to track the vetting and approval process, which is initially populated with information through the API.

Digital Screeners

Awards processes often include the requirement for digital screeners, so that a judging committee or the association’s membership can vote for the entries. Sohonet have methods to provide the playback and content delivery through the FileStore product. A typical awards workflow would have the process of transcoding way back in the process just after the original content is submitted. This gives entrants the ability to review and approve transcodes so any issue can be resolved ahead of time.

Serving media for digital screeners focuses on playback quality. Media is encoded using FFMPEG to a number of H.264 formats, from resolutions suitable for mobile devices to full HD. Transcoders are typically set with a target bitrate higher than is typical with consumer streams (15Mbps at 1080p), aiming to minimize compression artifacts that might make it difficult to judge the quality of CG and post production work.

Progressive download rather than adaptive streaming is also utilized, as a typical judge who works in the creative industries will be more patient to get the desired quality than a typical consumer. Media is played directly off FileStore by using the temporary URL middleware of Openstack. As the name suggests, this middleware creates a URL to the media on the object store, which expires after a time limit, meaning that links can’t be passed on or published. Again, the cloud nature of the infrastructure means that it can scale out along with the demand for the media.

The technical processes around holding an awards event have come a long way from having a room full of mystery tapes, DVD’s and hard drives. Technology has allowed the shift of focus to the actual content, and entering these industry awards is now much more accessible allowing worldwide participation.

Article author Martin Rushworth is Director of Technology at Sohonet, member of HPA, VES and SIGGRAPH, and serves on the VES Awards Committee.