What’s The Foundry buying? The Tech of ADF

The Foundry today announced that they were bringing Mischief paint into its product line, but what are they buying exactly?

There are really three things The Foundry achieve or get with Mischief:

1. Access to the tech and patents of ADF that the drawing and painting program Mischief is based on
2. Access to world class research headed by Dr Sarah Frisken (co-inventor of ADF)
3. A way to extend the technology, already inside The Foundry, out to a wider much broader design audience

That last point is a key aspect. Bill Collis, Foundry CEO, was recently quoted in the UK Telegraph saying, “we have 20,000 clients, from Disney and Pixar to Industrial Light & Magic and Double Negative that use our software.” Mischief has about 4000 clients, but, significantly, very few are the same people. Mischief retails for the equivalent of a rounding error on a facility license of NUKE, and it’s a completely creative non-technical piece of software compared to MARI or KATANA. In the same article it goes on to say that the company has grown between 20pc and 30pc each year since inception in 1996 and is “on track for a turnover of £20m. It employs 270 people across its London, Manchester, Los Angeles, Silicon Valley and Shanghai offices.” But for The Foundry to continue to grow it needs to move outside just the highend ‘pipeline’ customers of NUKE, MARI or KATANA. Mischief lets other Foundry technology get to a much wider and very different user base. Version 2.0 is just the start of the Foundry’s Mischief product line.

The Foundry started this move into new markets with the merger with Luxology. MODO brought not just a 3D application but a user base of product designers and non-film users, a group that were previously not Foundry customers. MODO also was a fraction of the price of say NUKE. Mischief is a fraction of the price of MODO: costing now only $25 – which in itself begs the question how do you make any money from a $25 application?

Volume, for one, but also from the technology behind the application. ADF is actually something that started as a 3D tool, so while Mischief is very much a 2D paint tool, it took the MODO team in California almost no time to realize that the ADF technology could be used for better modelling and sculpting tools or new forms of three dimensional paint. In fact, not content to stop there, Brad Peebler, The Foundry’s President of the Americas and co-founder of Luxology, started getting some old friends and hard core programming types to take the patented ADF technology and experiment with it. Literally, he immediately hired some incredibly talented freelancers to experiment in partnership with Frisken, and just play with the technology. The early results are pretty impressive, and went beyond what anyone expected. Of course, it is completely unknown if any of this will ever appear in an actual product or which one, but this exclusive fxguide sneak peak shows just how much more than a simple paint package is being announced today in London.

One of those engineers/artist that Pebbler got is Taron (aka Timur Baysal). If you have been researching cutting edge texturing and painting tech you may have seen his app Verve. It is remarkable and inspiringly realistic for oils and actual paints. Verve actually paints using fluid simulations and as remarkable as it is, “it’s going to get a massive update soon,” fxguide has learnt. Peebler described Taron recently to fxguide as “an incredible human, phenomenal artist and inspiring programmer.” But Verve is not based on ADF…it was just a quick side product Taron came up with…(remarkably!).

Another key Peebler consultant/hire due to ADF is ex-Sculptris heavy weight talent Tomas Pettersson. Taron has been testing ADF sculpting for fun and pushing Pettersson from an artist point of view. He also wrote some very cool test shaders for the test sculpt video we feature exclusively here just below which is all new ADF technology

A fun – ADF sculpt test application, done by the engineering team to see if ADF could work for sculpting and explore the new technology.

Some might expect ADF or paint technology to appear in MARI rather than MODO, given the texturing program is already a layered Photoshop-ese program. Bill Collis, however, does not see that as inevitable. “MARI does a very particular thing very well, and Mischief is a very different product,” he explained. For Collis the point of the Foundry’s tools is storytelling and most visual stories start with a sketch. Mischief then is the logical start but more of an expressive experimentation and design tool, than the much more prescriptive and precise end texturing tool that is MARI.

The ADF technology behind Mischief is very powerful, the product is not a dumbed down implementation but a “smarted up” clever use of technology to make the sketching process fast and intuitive, to paraphrase Peebler. “NUKE looks like the cockpit of a fighter yet and rightly so,” adds Collis, whereas the new version 2.0 of Mischief is “incredibly beautiful and direct.”

ADF tech (2D)

Adaptively Sampled Distance Fields (ADFs) are a way of representing a shape in 2D or 3D that can greatly aid in drawing, rendering, collision detection, color gamut science and sculpting. From their first appearance at SIGGRAPH at the start of the 2000 they appeared to be a significant path forward, but then the research appeared to go quiet. Actually, the reverse is true. From those early SIGGRAPH days the technology has become almost ubiquitous. So significant was ADFs that Mitsubishi Electric Research Labs (MERL) closely held onto the technology and in particular the valuable patents. Very few creative applications have ever gained access, Mischief being one exception, in large part to being built around some of the original research. But the tech was also licensed into a huge number of places behind the scenes.

Heard of Saffron Type? No? It is the scalable type built on ADFs that is at the heart of Adobe Flash and the Amazon Kindle. It is implemented today in software and hardware. Used a cel phone in Asia? Outline-based Asian fonts often require 5 to 15 MB of storage, which is problematic for memory-constrained cel phones. In contrast, uniform-width stroke-based fonts (USFs) require much less storage, but as the name implies all the strokes in the characters are the same size so they are ugly. A character set like this can be reduced in size to only 250 KB. However, since each stroke has a uniform width, USFs lack the expressiveness. Enter Saffron which allows stylized stroke fonts (SSF) which are beautiful but similar in size to USFs. The same logic applies in computer milling where ADFs allows very small footprints to facilitate remarkable 3D printing.

In the effects and animation market, however, the tight control on patents has meant the only real application has been in 2D rather than 3D. fxguide spoke to Dr Sarah Frisken, one of the early developers and researchers at MERL and now at 61 Solutions, the creators of the Mischief sketching, drawing and painting software.

Infinite canvas and future

What makes an ADF paint program any different or special? The mathematical nature of the ADFs means that the canvas is literally enormous and one can draw at any scale, at any rotation and with extremely fast (data light weight) paint strokes. Tens of thousands of paint strokes and still the image is completely free of resolution or format until you export.

ADFs have several advantages for creative applications: they provide high-quality stroke rendering; they work with hardware-based rendering so drawing is extremely responsive; they are very compact, resulting in small file sizes; they can be scaled without introducing pixelation artifacts; and they can accurately represent much richer and more complex shapes than traditional vector-based stroke representations.

At their heart, ADFs are a fundamental graphical data structure. In ADFs, distance fields are adaptively sampled according to local data and spatially stored in a hierarchy for efficient processing and rendering.

ADFs can be generated from stock distance functions, triangle models and range images. Editing can be performed for discrete edits and for tool paths swept along straight lines and Bezier curves. ADF rendering can be achieved via asynchronous adaptive ray casting, point-based rendering or rendering with standard hardware after conversion. Vector graphics using cubic Bezier curves are not new – they are used in Adobe Illustrator for example. But programs like Illustrator are not painting programs. While the principle of Bezier curves with handles and tensions is very similar to ADF, ADF – and the way it is incorporated in Mischief – is not just a Bezier curve drawing tool. Not that a Mischief artist would even imagine curves and tension defining handles. Mischief is actually one of the most artist friendly paint programs on the market. Firstly, the canvas is infinite, and the resolution is whatever you want to be viewing the canvas at.

ADF actually came from research into medical imaging for applications such as anatomy education, surgical simulation and computer-assisted surgery. “1998 was the first paper that represented shape with distance,” Dr Frisken explains. The team needed a way to represent three-dimensional medical examples such as the way a knee worked as part of a coordinated a multi-institutional, multi-disciplinary project to simulate
arthroscopic knee surgery.

Frisken served as project leader, algorithm designer, software system designer and implementor on the project. “We couldn’t represent a knee as just a mathematical equation with some distance, you needed to sample the space within which that knee sat, and the distance in that space. If you just sample at a very high rate you get a huge volume of data, and that makes it slow to process and render, so we started looking at ways to reduce the number of samples you needed and we came upon using an adaptive sampling of the space.”

The ‘adaptively’ part of ADF comes directly from that initial knee problem and it has since been implemented in a host of ways from simple Oct-trees to more complex and powerful data structures. The first research was done by Frisken and Ron Perry who continues today to work on ADFs at MERL, but who also consults to the Foundry.  He was key to establishing Mischief and coding the original the product. Perry is very much a key author and developer of the technology, but he also has other important research interests at MERL where is is a senior researcher.

The font work was actually one of the first productizations of the ADF technology.  Saffron was released in around 2004/2005 and adopted by Macromedia which would be acquired by Adobe, and thus bring the technology to an ever increasing customer base – today estimated at 500 million people. Monotype Imaging (MTI) also licensed the technology and it is via MTI that the Amazon Kindle uses the ADF technology.

After a stint at Tufts University, Frisken started consulting for Pixar and Disney’s 2D Animation Research Group for a better drawing system. In order to use ADFs to do that, Frisken needed to license back the patents on her earlier research. MERL was also keen to work with Frisken on 3D milling. Roughly speaking, in exchange for 3D help, Frisken got 2D patent licenses for the drawing program. Frisken spent several weeks with the animation team making the 2009 film The Princess and the Frog trying to understand their challenges. “By the time we finished the product that group had been disbanded,” she says. “One of the wonderful things about our contact with Disney is that we brought in all this IP, but I was always going to retain the product IP after the development was over, and it felt like we had something pretty special in terms of a drawing program.”

Frisken decided to take a couple of years off to produce a commercial painting program. The program first saw light of day in a small beta program from 61 Solutions Inc in 2012, and it was then launched in June 2013. 61 solutions was the original company founded by Perry and Frisken.

The resulting Mischief could possibly be the most simple, artist friendly and non-tech paint program on the market. It literally requires no understanding of maths and has vastly less menus than say Adobe Photoshop or Illustrator or even Corel Painter 2015.

The program does have to process the mathematical representation and then render it, but the distance fields are produced on the fly, and produce the textures stroke you see on the screen using the GPU. Each stroke in the database of strokes the user has made is not represented as a spline but as a set of mathematical equations that go into the renderer which scales them and the ADF define the strokes. (It is not as if those strokes are converted to a sampled representation as one might do in some traditional 3D pipelines). You can tell how the program works, as very heavy drawings (60,000 ~ 80,000 stroke paintings) do slow down as you scroll the image – since the whole image is redrawn every frame of that move (there is no buffering or cheats).

Of course, one of the great aspects of Illustrator is its editability. Splines are incredibly useful since you can adjust the path or curve. While there is enough data to be able to refine a path, this is not something that Mischief allows, and it is a major design decision based on speed and ease of use. You can scale, rotate or delete but you can’t edit a brush stroke in the picture (in the database).

The program does have a host of other features such as importing images to work over – a Photoshop style layers approach but there is no text functionality – despite the great Saffron pedigree. The program is Mac and PC and retails for around a twentieth of Corel Painter. (the new price for v2.0 of Mischief is $25 – and there is a free version). The Foundry are launching with the new 2.0 today.

New Features in V2.0:

• Free version (with limited functionality)
• New Panel UI
• Trackpad support – Mac for navigation
• Multi-touch
• Eight Custom Swatch Palettes 
• Editable Hex and RGB values
• Six dockable Brush and Eraser Versions 
• Control for Minimum brush size
• More Hotkeys assignments 

The program at the moment is just a 2D paint program, but could Mischief be expanded to have 3D brush strokes with volume? “The technology can work in 2D or 3D,” says Frisken. “We have even discussed internally some 4D applications. The software that is the backbone of Mischief right now is absolutely able to do 3D. All that is exposed right now is 2D but the underlying engine could do 3D. We have imagined sketching in 3D or sketching on a 2D canvas at any orientation or rotation to the camera.”

ADF vs Voxels (geek out)

Back to 3D and the roots of the technology.

You might be wondering about the differences between traditional voxel representations and ADF distance fields in 3D? It is a complex area of 3D maths because a voxel representation can be thought of as a partitioning of space into cubes, and one can store anything in each cube (including theoretically distance). But if one accepts that most voxel implementations are not storing distance then the comparison is a little easier.

A more traditional voxel representation use regularly sampled volumes and they store either binary values such as inside or outside or density values (using a density of 1 inside the shape, a density of zero outside the shape, and an average density for voxels that contain the surface).

Voxels have many uses – sometimes the value is assumed to be the average of what is in the cube/box/voxel, sometimes it is assumed the value is at one corner of a 3D grid.

Significantly, if one uses a discrete sample approach, then you really only get binary samples for shapes that have hard surfaces and you have to interpolate the density inside the cube when you need to locate the surface of the shape for rendering or other processing. (eg. 1 inside the shape, 0 outside).

There are several problems with this approach. First, traditional voxel models were regularly sampled so you needed a lot of samples to represent shapes that have complex and compound detail, even if that detail is only on the surface or limited to a very small part of the shape. If you are more than one voxel away from the surface, the density is just zero, so you have no way of knowing where the surface is. And, while one can reconstruct the surface position inside a voxel, you can’t apply higher order filtering (which is applied over larger regions of space) to get a smoother surface reconstruction. Thus, voxel models tend to have a fair amount of aliasing, or require a lot of samples… ie. data. 

Distance fields are defined throughout space and they vary continuously across smooth surfaces (unlike voxel density which jumps from zero to one as soon as you cross into a shape). Thus, they can be sampled more sparsely and they can be reconstructed with less aliasing. Importantly, they can also be reconstructed relatively far away from the surface, as the reconstructed distance field gives you useful information such as what direction you should look to find the nearest surface and how far away it is (both useful for ray tracing or for estimating forces for interpenetrating objects). Which is why Frisken initially thought of using ADFs in collision detection over a decade ago. Also key for any good renderer is that when a sample is on the surface, the direction of the distance field is the same as the surface normal, which is useful.

In the sculpting R&D test application above, Tomas Pettersson used a vector distance field for his sculpting system. This is an extension of ADFs, to do this you sample the vector distance instead of just the signed distance. The vector distance tells you both the direction to the closest surface and the distance from the surface. These are known as vector distance fields, and for the price of 3x as many values per sample, they allow The Foundry’s team to reconstruct surface normals or direction vectors more accurately and they allow artists to represent non-manifold surfaces such as points, lines, and infinitely thin sheets.

ADFs both introduced the idea of representing shape using distance fields and they proposed a way to sample distance fields sparsely or ‘adaptively’ so that you only sampled in regions where the distance field was useful to the application in order to save space and time. Back in 2000 the first paper used octrees, but the team has move on since then with several other spatial sampling methods. Mischief v2.0 uses a different sampling method today but one that is even more efficient. 

Future Foundry

The new wholely owned subsidiary company will be headquartered in Boston, with team members located in Boston, London and Shanghai. Bill Collis points out that already the number of developers has “greatly increased”, and the new team is already hard at work and have been now for some time helping to finish version 2.0.

“The Foundry has a proven record of taking exciting, innovative concepts and commercializing them for a broader market,” says Sarah Frisken. “By becoming a part of The Foundry, we now have the ability to grow our team, to be more responsive to our users, and to further our vision. I’m also very excited about the addition to the team of Chris Cheung, who will head our product agenda. Chris has invaluable experience in building a large and loyal following in the artistic and design communities so we’re very happy to have him on board. With our talent and technology, we will create new and exciting products that in turn create new possibilities and experiences for our customers.”

In the new organization, Frisken will become chief scientist, allowing her to explore the convergence of technology and art in future innovations. She will work closely with Cheung to drive the conceptualization, development and commercialization of digital artistic solutions. Christopher Kenessey, chief sales and marketing officer for The Foundry, will assume the position of president of Made With Mischief. In his new dual role, Kenessey’s executive leadership will drive adoption of Mischief among a broader market while also identifying development opportunities through technology crossover.

It seems right now the limit on new tools is one of developer bandwidth and perfecting the input device, but the technology is not limited in this regard. The application of an oil paint stroke interests Frisken. “If one had an infinite amount of time that is one of the things I have always thought that would be fun to try,” she laughingly comments.

Mischief currently has about 4000 users, and that number is expected to expand dramatically soon. But even as the program expands the team are keen to keep the UI as simple as possible. That being said, user requests such as fast responsiveness for huge paintings, more transfer modes for the layers and a more flexible lasso tool are all being explored by the researchers.

Beyond Mischief, today in addition to drawing, fonts, 3D printing, etc,  games companies use ADFs to add decals to game assets in real time, and its original design brief such as collision detection with ADFs remain remarkably unexplored.

And on an unrelated point:

We could not resist asking Bill Collis about the recent Black Magic Fusion ‘Free option’ announcement, given how significant NUKE is to the compositing community. Here’s what he said:

“NUKE has for its whole life had competition from many other compositing solutions, including Fusion. NUKE is widely chosen as a professional artist’s tool and a robust post-production platform. Our customers don’t make the jump to NUKE based on our pricing or our marketing efforts – they rely on our high investment in R&D, our aggressive development schedule, and the way we partner globally to turn complex post-production concepts into friendly applications. They also depend on our commitment to pipelineable products, the value we place on listening to our customers’ evolving needs and the emphasis we put on standing alongside them with our highly-respected support team. Our success in all these areas makes it look easy – it isn’t”.

-Bill Collis, CEO of the Foundry.