Believe it or not, many of the principles of optical compositing apply to digital compositing and it's valuable to have a deeper understanding of what takes place behind the scenes when we hit that "process" button to composite.
Optical compositing works on exposing film to light. It is an additive process, the more exposure, the lighter the image. Digital images are similar, an additive process. In a typical scenario, 8 bit linear video, any given pixel has a numerical value, each channel of red, green and blue, 256 levels from 0 to 255. These values are expressed as three numbers separated by commas, red,green,blue. For instance, black would be 0,0,0 and white would be 255,255,255 and orange could be 190,190,10 . Combining images (compositing) is a mathematical processing of these numbers. We don't generally need to see the maths, it happens behind the scenes but here is a typical example of what our software is doing during a BLEND composite.
figure (1a.) shows the background image, (1b.) shows the foreground to be composited, figure (1c.) shows the matte, or alpha. |
When we hit the process button on our compositing software, first it mathematically multiplies the background (1a) by the inverse of the alpha (1d). Where the matte is black, pixels from (1a.) are multiplied by 0 for each channel of RG & B and therefore end up as 0,0,0. Where the alpha is white, they are multiplied by 1 and retain their original values. Grey areas are multiplied by the appropriate numbers between 0 and 1. The result of this is shown as figure (1e). You can see a black area in the image where the matte was black.
Next the foreground in our example (1b) is multiplied by the matte (1c) giving us the result (1f.) where the foreground is now on black. Lastly those two results, (1e.) and (1f.) are mathematically added to give a final composite. Normally when you add two images you will have some areas that clip out to maximum and the whole effect is very bright. Because the foreground is cut out and being added into a "hole" that has been created in the background, there are no un-naturally bright areas.
"Add" is a simple mathematical process - for example, a pixel in the lower left hand side of the background has a value of 175,100,60 and the foreground has 0,0,0, when the two are added together obviously it maintains the same value as the background image. Pixels in the foreground subject area also remain the same as they are in the foreground. Pixels in the soft edge of the matte area however are "blended" - for instance if the 1a. value is 150,50,20 and the 1b. value is 50,50,100 then the composited pixel will be those two added together - 200,100,120. This maths is carried out for every pixel in the composite and in FFI is known as a "blend". |
Another scenario is where the foreground is a 3D element rendered on black or perhaps a practical fire element shot against black as in our example 2.
(2a.) is our background, (2b.) is our foreground, and so on as labelled. In this case, you still need to multiply the background by the inverse of the matte to create the 0,0,0 value, or "hole" for the fire image to be added into, but you don't need to multiply the foreground by it's matte. This is why some compute generated elements to be comp'd are known as "pre-multiplied". The composite, (2g.), is (2a.) added to (2f.). This is known in FFI as an "ADD" comp.
Variations like blurring etc. are carried out on the relevant elements before the maths is applied but that's basically it.
Perhaps the most common compositing task is combining two images that have been acquired separately but must be combined "seamlessly", that is - so the audience believes they were captured in camera at the same moment. Here is a noted list of what I consider to be the most important factors to address in matching the images.
If you can possibly have input into pre-production it is worth "stating the obvious" to the relevant members of film crews. Best case is to get on set yourself, but failing that, if you can be diplomatic about it try making a few suggestions such as ...
- Preferably shoot on slower than 500 ASA filmstock for green or bluescreens - ie. 250 or lower. Grain structures on fast stocks are getting better but can still cause "boiling" edges on keys.
 -Shoot charts as reference. These will give you "known" values that you can measure your filmed results against. The first chart - called a macbeth chart - is for colour. The second one - a SMPTE field chart will show you how the camera was set up and how much distortion is being introduced by the lens. Both are practical printed charts that should be placed in front of the camera and filmed. Most camera departments will have a macbeth chart but it may pay you to have one of your own. |
Try and shoot the background plate first.
Take a video tape of the background plate on set for the foreground shoot and use it as reference. Failing this scour the internet for stills or movies of that which you aim to recreate.
- if possible get someone to carry out the foreground action in the actual background and shoot that to use as reference both for shooting the foreground and for the composite. Make sure that this reference take is marked for telecine transfer so you will have it with you in the comp.
- Take notes on lighting, camera details etc. to aid in matching the foreground plate. You can make up your own camera log sheets to take on set which will help prompt you on what to survey.
- It can be useful to shoot a chrome ball or grey ball in the background plate (as a separate take). The chrome ball will show you all the relevant light sources, both direct and bounce light. The grey ball will also show you lighting cues and is useful for 3D crew. They can put a 3D ball in next to it and adjust their lights until they match the real ball. . |
 It's also useful to shoot some wide shots of the general scene during a take to show the light sources and general stage setup. You may think you will remember details but weeks after the shoot it's better to have a pictorial record. |
Stress the importance of lighting match and if possible get it right "on the day". It is not just lighting direction, it is also general conditions such as sunny / overcast. Film shoots are often very schedule driven and waiting even 15 minutes for clouds to pass over is sometimes deemed not necessary. You may be called upon to argue your case but remember ... be diplomatic !
- watch for focus and depth of field. If the camera is focussed on the midground with shallow depth of field, an actor walking from behind camera into the scene should be out of focus until they reach mid-ground. This sounds obvious but it can be overlooked. For example, if the actor is being shot against greenscreen a safe option would be to shoot it with follow focus (actor always in focus) and do the defocus in the composite. It can be tough to pull a defocussed greenscreen key but if the subject is sharp all through then you will get a good key which can then be defocussed.
It's worth saying a couple of things about the telecine session - if your elements are being transferred from film to tape.
- NO grain reduction or edge detail enhancement. Both these are commonly requested by directors but will add artifacts to your images.
- NO colour grading of the blue or greenscreen areas. Some telecine colourists try to "improve" the colour of a chromakey background by making it more even and saturated. Telecine systems use a keying process to do this and will almost surely degrade the edge of the key, precisely the area that you need to pull your key so say NO to this.
- some directors are happier to colour grade in telecine than in composite suites. If this is the case you may find they will transfer the bluescreen twice - once a flat grade for you to pull your key, and a second time with the foreground graded. Due to "float" in the telecine grades the mattes pulled from the flat grade may not match up to the graded pass. You can either grade your flat pass to match (don't tell the director!) or stabilise and line up both passes before pulling key mattes.
OK, so let's assume you are in your compositing suite with elements ready to go. What should you watch out for ?
1. Match lighting ... if your foreground is fully out of wack you may be able to flip either front or back to help it match. If that isn't possible then you might be able to help the comp. by creating a roto-mattes and grading either front or back to create fake shadow areas. It is surprising how well that can work. If the foreground is supposed to be backlit but wasn't "on the day", you can again create a roto-matte of the foreground subject, shrink it in and soften it, and use it to kick an edgelight and a glow. Go on the internet and find images of similar lighting situations to help you work out what your foreground needs.
2. Match colour and contrast. ... if you do not have reference shot on the background, look for objects in the background that are a similar distance from camera as the foreground you are adding. Sample their highlight, mid and shadow areas and take note to match these areas of your foreground. Shadow colour is a depth cue and in daylight, shadows tend to get lighter the further you get from camera as they have more atmosphere in front of them. The best way is to match values to objects already in shot - that are a similar distance from camera. If there are none, then you will have to "eyeball" it.
3. Match detail sharpness (different to focus) and film grain ... different lenses record detail differently so look at objects in the background and take care to match that quality in the foreground detail. Film grain will need to be matched also ... hopefully it will already match, but if not you may need to work on that. It is not generally desirable to degrain and then regrain images but may be necessary.
4. M edge sharpness ... nothing captured on film is really sharp and depending on where the subject sits in the depth of field you may find the edges seem softer than the subject itself. It's important to hunt around within the background plate for similar edges at similar distance from camera and match them. This can be particularly helpful when comping 3D renders into filmed backplates. If you comp. one copy softened for the edges you can then comp another copy over the top with the edges shrunk in and softened to blend. This allows you to soften the edges separately from the fill.
5. Match focus / defocus ... as mentioned earlier, if there is any defocus to be applied, try and use a defocus filter rather than just a gaussian blur. Lenses don't just soften a subject when they are out of focus, there is an effect called "circle of confusion" which is a little like adding a step and repeat or scatter to a defocus. There are plugins or sparks that can correctly emulate this and I feel it is worth the trouble though you can get away with gaussian blurs if the defocus is not too extreme.
6. Watch for cast shadows . Again dependent on light direction, your added foreground may need to cast shadows on objects in the background plate. Roto-mattes are your friend here - make mattes and colour grade through them to create shadows. This is MUCH better than just comping black through the mattes. Most shadows aren't just darker they have colour cast. You will also want to avoid doubling up shadows. Your foreground subject may need to cast a shadow across both lit and shadowed areas. In most cases it should not further darken an existing shadow, it will just join on to it. It can be tricky to blend edges of added shadows and existing ones but I think it's worth the trouble. Double darkened shadows are a total giveway and compositors found guilty should be severely punished !
7. As a general approach. We would recommend minimising intermediate processes because these can subtly degrade image quality and the more processes you apply the more you will move towards this being unacceptable.
