At fxguide we are acutely aware that many of our readers are outside the USA, and working on a range of projects, which is why we are always keen to talk to specialist studios from around the world.
We recently spoke to Bogdan Mihajlovic about QLBEANS, a camera tracking specialist company in Belgrade Serbia. This small boutique studio specialises in camera and object tracking. We were keen to find out how they operated and what clients can provide to make tracking more accurate and effective.
QLBEANS actually worked on some of the GDC Blackbird project. For the plate photography The Mill NY wanted to remove the ground shadows from some of the original plate photography. QLBEANS helped clean up the road plate, remove some signage and some plants. This part of the GDC demo was done in a fairly traditional fx pipeline. To have clean plates for the live car compositing, The Mill wanted a clean open road, without the Blackbird and its shadows. These high quality clean plates were then played and comped in real time at GDC. The team also cleaned up the reflection map pass for the final render, so that the cars did not reflect the camera truck, (the camera car with the arm filming the Blackbird). Without this clean up, the cars in the GDC demo would have accurate reflected the Arri-camera that filmed the Blackbird.
Belgrade is a local hub and has good telecommunication infrastructure. The city located equally distant from several major European centers. It is wonderful place for work and live according to Mihajlovic. For QLBEANS, Belgrade in Serbia has several advantages, in terms of infrastructure they rely on a fiber network connected on 10GB link to the rest of the world. “Local network transfers of 4k and 5k files are smooth and easy” he explains. The company does 3d tracking and matchmove for production companies worldwide.
FXG: What tools do you use, I understand you have developed some of your own?
BM: All artists at QLBEANS came from different backgrounds and vfx studios. Matchmoving today is a standard segment of any cg / vfx pipeline whenever live footage and 3d are blended together. There are many software packages available on the market and the tools are getting better and more efficient all the time. Many of the tracking algorithms are implemented in packages like compositing and 3d software’s.
We are using a mixture of Syntheyes and PFTrack to get the initial track and solves. At the end of our pipeline everything is finished in a 3d package.
Shots that we are dealing with are getting more and more challenging. Basic tracking is usually done in-house by our clients on their own automatic tools. When they have issues, is what they typically pass shots on to us. The ability to resort to hand animation and a full understanding of space, camera movement and technical knowledge is crucial. We have developed in house tools and scripts to ease the production pipeline. We have automatic scene tools, animation transfer tools from object to camera and vs. versa. Our pan and zoom tools tend to be for more precise for the alignments of objects and 3d points to a pixel level of precision.
FXG: How much of your work involves working with scene or environment data with Lidar scans?
BM: Lidar is a great source for 3d tracking and survey work. Polygons and point clouds can get very dense and heavy to manipulate but they are an extremely accurate source. We love lidar scans! Survey tracking is the best approach over all for a matchmoving process. When you know and get exact position of a point in space and have that point 2d tracked in the plate, the matchmove solver usually turns out to give you super accurate results. CAD models of the cars are also often just as good as Lidar or even better. It can be crucial to have an accurate 3d object of any object you are tracking in a shot plate. Accurate object or survey data can help in establishing the right lens as well. Getting a Lidar scan is expensive and not all clients can afford it. Even big production houses don’t use it very often.
By contrast, photo scan tools are a good and affordable choice. They provide a good source for matchmoving when Lidar or CAD is not an option.
FXG: If you use Lidar scans is there any advice on how to make this useful for your use?
BM: It is really an issue of objects and space,which means the file size of the data files. Sometimes files are huge – which can be an advantage since you have such a great choice of points. We request getting both the source Lidar – before any clean up and the cleaned up version. In cleaning up a file we sometimes find that the stripped down version can miss crucial points for tracking and alignment. That’s why we ask our clients to share both.
FXG: I assume most of your work involves un-distorting the footage – do you normally get camera charts or lens info?
BM: Getting accurate script notes is not always possible. In most cases they is only broad information but even that can be a huge help in 3d tracking. Lens charts and grids are perfect to have but only if they are accurately connected to the take and plate (written in the script notes). The gate size used on camera is the most crucial information beside the lens. With Arri cameras it is normally easy, but with the RED Camera, the set up can be hard to work out. For example, what was the resolution, was it cropped? etc
Another key is to make sure that the final edit and conform process is noted and any script notes updated. Source resolution is often lost as the meta data is removed if the footage is transcoded. In this case the notes from the edit play an even more crucial roll.
If all the material is properly organized, then getting chart or lens grid shots really comes as a cherry on top of the cake to make our 3d track more precise.
FXG: A lot of your work involves object tracking – beyond camera tracking – especially cars, again are you given geo? Do you make your own or look up models?
BM: CAD models are best path for car matchmoving and object tracking. Even if we don’t track the car in some shots, a car model can help checking and setting up exact lens in the best possible way. Geometry is extremely important for matchmove. To define the space and lens in reference to the camera chip size or film gate you need minimum 3 values :
- measurements from the set.
- Height of one object in front of the camera + a measured distance from that object to the object in the back of shot and
- the height of back object.
Camera tracking or matchmove can really seem like more of a detective job some days. Having as many measurements and photos from the set can really help to recreate what happened during the shot.
FXG: Do you get clean copies of the camera source with metadata or is it normally transcoded and the metadata stripped?
BM: The metadata is usually stripped out and gone. We just don’t get it. With smaller studios sometimes we have a better chance of getting more information and seeing the original camera files including any metadata. But if on set they use old lenses then the new cameras can’t record this information, so we still need the lens values to be written down.
FXG: Is it still extremely hard to solve a zoom during shot? What makes a hard shot? For example the crowd dancing (in the club) on your reel looks almost impossible, how did you do that?
BM: A zoom during a shot is not that hard anymore; it was tricky, – back in the day. In heavy duty zoom shots it’s good to have a side recording of the lens values. There are often times small cameras that can be fitted next to the lens to get that information. A zoom is potential problem on motion control systems. When you have many passes and complex move, then a zoom can make things more complicated. I would say avoid zooming on motion control. Zooming will change the camera pivot point during the move and create a small parallax shift.
What is a huge help is to get for zooms is a recording of the lens chart distortion grid. That way all the values and lens stretch can be replicated and adjusted.
Zooming can introduce additional complexity to tracking, one of the key problems is this animated distortion.
If the directors likes to zoom, or there is no other way to get a shot, then the vfx supervision and survey team need to get as much information from the set – and really cover all aspects of the set measurements.
The crowd dancing in the club on our show reel is for a Spotify project done by The MILL.
It is an example of combining initial solves with hand animation. In this shot the physics of the camera move is crucial to know. The camera is a heavy object and so in most of the cases it can’t have big jumps in translation values, everything is small and any fast moves we see in the plate must come from rotation. This is extremely important when preparing the cameras for motion control systems following them with a matchmove setup. Speed is the next thing to pay attention too. Camera can’t move very quickly any solution that has a 6ft per second or faster move is clearly an error. You have to be realistic with replicating what happened in reality.
FXG: What can people do to make the tracks easier and more successful?
BM: Take as many photos and measurements on set, follow what is going on with the gate and lens values, make VFX script notes in addition to the regular production notes.
Make sure to get measurements between measured objects. Don’t just measure just objects alone on the set. We really need the distance between them as well. That connection measurement can help creating simple survey that will ease the process later. If you don’t have a distortion grid then see if you can point camera the camera at some parallel lines on the set, for example a wall, window or door frame – and record this. The lines should be where distortion is most visible, out to the edges. If you are shooting a character for rotomation it is always good to also film them with any witness cameras as well> It is worth spending time on witness cameras and then syncing them with main unit. It will save so much time later in the post.
FXG: How do you bid the work – is it per shot, per day ??
MG: Some clients just share plates and we track for them based on a day rate. Other clients request a quote or estimate of working days and hours before we start. We are always opened to whatever the client needs and we can talk to see if we can make the budget work for both sides.
Below Mercedes AMG 30 by Blacksmith, visual effects studio based in New York City.