Last year at NAB conference, Light Field Labs announced their plans to make a Holographic monitor. If it had not been for the names of the founders and their work at Lytro, such a press release would be easy to dismiss. The word hologram gets tossed around quite a lot, but this team seemed incredibly qualified to actually deliver. Light Field Labs was founded in 2017 by Jon Karafin, Brendan Bevensee, and Ed Ibe. Three exceptionally qualified serious light field experts. Karafin came from Digital Domain, Bevensee has a PhD in Particle physics (he worked on the Hadron Collider) and Ibe’s track record goes back to IBM and is marked by a strong mechanical engineering background. All three men were in the leadership team behind the Lytro Cinema Technology, and in particular the professional high end Cinema camera and capture system. The Lytro Cinema camera has 755 RAW Megapixels, runs up to 300 fps with up to an incredible 16 stops of Dynamic Range and records a whopping 400 gigabytes per second. The Light Field Lab team in the past has made both consumer and professional products and successfully delivered them to the market.
This week Light Field Lab secured $7 million in seed round funding led by Khosla Ventures and Sherpa Capital with participation from R7 Partners. Light Field Lab will use the funding to complete a prototype of their revolutionary light field display monitors, enabling perhaps for the first time real holographic objects to appear as if they are floating in space without the aid of glasses or head-mounted gear. $7 million may not sound like a lot, but this is just ‘seed’ captial, and as such $7 million is more than respectable. The investment is a statement of confidence. Khosla Ventures, for example, has been involved in some pretty big ‘moonshot technologies’ across cleantech, energy and information technology.
There are a few ways for an audience to believe they are seeing a 3D object. The most wide spread is Stereo projection. Here the content is actually two copies of any imagery, one for each of the left and right eyes. The problem is that you need to wear glasses to control what is seen by each eye. Importantly, no matter where you sit in the theatre, you see the same two images in stereo projection. If there is a car on the screen – no matter how much of an illusion there is of it being ‘solid’, there is no difference in what you see if you move seats. You can never see the other side of the car, even if you walked across to the other side of the cinema.
On marketing posters this is sometimes addressed by lenticular posters. Effectively as you walk past the poster, and the film covering on the poster switches which part of the poster you see, so someone on the left sees one image and someone on the right sees a different image. It is crude and but you do not need glasses. The problem is that it tends to be incredibly clunky and work in normally just one direction horizontally.
The other new import Holographic display device is the RED Camera company’s forthcoming Hydrogen One ‘Holographic’ smartphone/camera. This technology is different again. The Red Holographic 5.5″ (2D: 2560×1440) ‘holographic display’ is believed to be a diffracted backlight, autostereoscopic display which is very high quality but different from the Light Field Lab Monitor. In theory, a true hologram is capable of focusing light to pixels at a higher resolution than is possible with autostereoscopic pixelation. But the Hydrogen One is yet to be released or its monitor technology fully published. The screen comes from Leia Inc, who do say their their screen uses ‘diffractive lightfield backlighting’ ( trademarked as DLB), so it will be interesting to see the phone when it is released later this year.
Strictly speaking the definition of a holographic display allows all the degrees of freedom to move and look at the displays from any angle. Some researchers have called the projection of laser light onto smoke or dust holographic, but this is not the same thing and very limited in range, colour space and resolution. The closest example to a holographic monitor previous has probably been the Light Field work in VR.
The holy grail, that Light Field Labs believe they have cracked, is a monitor that presents to each of your eyes what you would see at that exact point in space if the object was really there. A Light Field Lab giant cinema screen would allow the audience on the left to see one side of the car while the audience on the other side of the room would naturally see the other side of the car. Far from clunky, this technology would deliver to each eyeball the correct view on the object exactly. Your brain would be hard pushed to know it is not real since even a minor head movement would provide the correct stereo effect and you would actually be able to see around corners of the object, unlike today’s stereo cinema.
The end game for the company is a room built of such monitors which effectively is the Star Trek Holodeck. Such a room could appear as anything, from a forest to a shopping mall. It would be made up of a truck load of monitors and their backend display computers, but no matter where you looked, the world would optically appear to be real.
For many researchers this is a significant serious research target. For entertainment companies, this is a very viable location entertainment, concert or cinema stage for a huge variety of entertainment. For example, a cinema could appear to have a traditional screen on one wall, but the ‘action’ could seem to come into the room if the theatre ceiling and side walls were also built with Light Field Lab monitors.
How viable is the Holodeck concept ?
Light Field Labs already have a prototype monitor running and it is on the success of this prototype that the company got its’ latest seed capital. The initial modules are 6 inches by 4 inches, and they can seem to project images into a 3D space. This prototype will have a resolution of 16K by 10K, but that number is misleading as one person would never simultaneously see all that resolution at once. The production modules will be closer to 2 feet by 2 feet in size. The best way to image the holographic monitors as both ultra high res but each 2 x 2 monitor might appear, at any instant, as sharp as an HD monitor. Naturally, sharpness is a function of contrast, viewing distance and density (pixels per inch). If you stand well away from any 2×2 normal HD monitor it would be virtually identical to a 2×2 monitor showing 4K footage, assuming the contrast levels were the same. High resolution is most important if the monitor is large and you are close. So Light Field Labs are designing their monitors to connect seamlessly into a wall.
How it works
The ultimate window
Lightfields are complex but not new, there are great examples of Lightfields being created by companies such as Otoy in VR and the notion of a Lightfield has been around for decades. If one has a volume then a light field can be said to be what the world would look like from any place inside that volume looking out. A light field is recorded not from one point, which is the traditional camera approach (at the filmmback), but literally from any place in the volume. Virtual cameras allow us do that in 3D with VR, we can move anywhere and the VR head gear will render a view from that point. The Light Field Lab solution is significant as it does not require the computer to know where you are. The monitor is not ‘aware of you’ in any tracking sense, you don’t need to wear anything special or in any way be detected. The monitor just looks ‘right’ from exactly where ever you stand in front of it (notwithstanding viewing angles see below).
A simple way to think of a Light Field monitor is as a perfect digital window. Like any window, as you move around in front of it you can see different things, you can see past a tree limb or on the other side of a fence, if you move relative to the window. The Holographic display can do this if it can present you with the right data or rays of light that you would have seen if you had been actually looking at reality. But it gets better, there is no reason for the images to just be on the other side of the window, they can project out in front.
If the monitor can present you with the ‘information’ or light rays that match an object sitting in front of the monitor then an object will appear to float in the room. It will seem to be projected from the monitor. But it will only do so if the physical monitor is behind it. The illusion is broken by the edge of frame. While artists ‘imagine’ such displays as breaking out into the room, such drawing and renderings are inaccurate. It may seem it is out in the room with you but it can only exist within the viewing pyramid that exists from your eyeball to the screen, it can never break that space.
But how does it work??
The screen gets a vast amount of data, which contains more imagery than you can see in a ny one instant. Conceptually, the screen is a series of glass balls. If you imagine a line from your eye to anyone of these tiny balls, any such line will hit at 90 degrees – it is the very definition of the balls surface normal aimed at your eye. For that angle and that angle alone the screen provides you will a pixel that is correct for just that one angle. What might be seen on one side of the glass ball as red would not be red from any other angle. Again conceptually imagine the ball is a tiny dome render of the screen, except you only see one part of that projected dome, the part that is pointing right at you. With thousands of ‘balls’ each showing you only the right colour for that exact angle, each of your eyes would see different images and each person standing next to each other would see a different image. As it is producing all the relevant angle ‘solutions’ or pixel colours all the time, the system does not need to know where you are in the room. The system just floods different views of the content in every direction, all the time, and if you look at it, you will see the ‘same’ information or light pattern that you would have if the object had been really where the monitor is.
“True reconstruction of every single photon in space, at the density the eye needs for visual acuity, so when you control where the rays converge in space, you can make an object look like it is where ever you want”
It does this by having ‘Quantum components’, so called as they are nano sized materials (not to be confused with Quantum computing which is in no way being claimed). The screen has a light source which is built on tech that is already existing, (understood established technology), in front of this is a Nanoparticle Energy Surface, this can be thought of as the thing that generates the relevant colour, “this is what creates the 4D raster of Light” comments Karafin. On top of that layer is the Holographic Encoded Projection Waveguide (this is the ‘quantum’ component). This top layer controls what you see from what angle. It mediates the imagery into directional rays we see as the holographic pixels. This is what directly colliminate (accurately align) and control the rays of light. Together you can place a set of rays of light in a way that matches what would have been seen if there had actually been an object. It is giving the viewer a “true reconstruction of every single photon in space, at the density the eye needs for visual acuity, so … when you control where the rays converge in space you can make an object look like it is where ever you want, in front or behind the screen” explains Karafin.
Another key difference between this approach and stereo cinema displays with glasses centers around where your eye focuses. Clearly in a cinema with stereo glasses one still focuses on the screen plane, even if the stereo effect makes the object appear in front of the screen (or behind it). This is not the case with the Light Field Lab monitor, “With our tech, your eye focuses at the point of light / holographic object. It is a ‘real’ image, where convergence and accommodation of the eye are matched. The point of light is the location where you see the real-world object in space and when the ray bundle converges at a density that meets the visual acuity of the eye, there is virtually no difference between the real and the synthetic object” explains Karafin.
The first model has a screen size of 6″ by 4″ and a ‘resolution of 16K by 10K (not that one sees all of that at once). This model will be available to researchers and early partners hopefully by the middle of this year. This model has a sizable edge bevel as it is built to quickly disperse heat. The next model is 2 x 2 and has almost no bevel and is designed to be built into a huge video wall. The next model is called the S650MP ad has 650 megapxiels and will be 25.5″ by 22.5″ and a pixel density of 36K by 20K (!) This monitor is design to be coupled with a set of other monitors into giant walls or rooms. For example, a theatre might be made with a 24 feet by 7 ft screen, which would result in a combined ‘resolution’ of 385K by 75K. This target cinema version is codenamed the S28GP.
Limitations or current specs
If you like us you have a million questions, so here is a summary FAQ:
What is the density of the display? It appears like HD resolution but it is much more dense than that.
Can anyone make an image to display? Yes Light Field Labs will be content agnostic, they want all content makers to be able to make Lightfields that can be played back on their monitors.
Can you film a Lightfield? Yes that is what their old company Lytro make (see above).
It must burn through data and bandwidth? Yes at the moment each monitor needs a 300MBPS feed
What is the refresh rate? Light Field Labs wont say, but we understand rates such as around 60 fps is currently running fine and there is not immediate issue here (ie. it might be able to go higher at the time of release, but it is unknown).
What is the ColourSpace? Again we dont know but it seems like the company wants to service processional film makers so P3 or something close is expected. Again, there is no indication that there is an issue here.
How bright is it? Bright, but no specs published but the new clever tech builds on technology used in normal monitors to a point.
Viewing Angle? As you become more side on to the monitor, the experience must decrease. At some point the screen would be edge on and you would see nothing (90 degrees to the screen normal). For normal viewing you would be looking straight at the monitor (0 degrees). It is understood from some work done (not by Light Field Labs directly) that the monitors right now are great to around 60 degrees, which is not as much as a modern LCD, but more than enough to be able to have a set monitors surround you in an installation and the monitors work effectively.
The Biggest Limitations
The main issue with holographic monitors is maintaining data rates. Given that there is no sensible way to ‘broadcast’ holograms, each installation will require dedicated GPU computing and networking.
This technology works best with decent sized screens. No doubt someone will consider a full Light Field Lab holographic smartphone, but for the Karafin and the team this is not a priority, as the image can only exist inside the viewing pyramid (or precisely in front of the screen), so a larger screen will always be more impactful.
The Business Plan.
The company plans to first address the professional entertainment market, where large capital and infrastructure would be no barrier to adoption. But the company is very focused on a medium term strategy that would embrace consumer level monitors and see this technology widely available.
To sell into the location entertainment market the company needs to be able to manufacture the S650MP. It provides the size and the bevelless form factor needed.
The strategic guidance is expected from the new investors which will be critical to enable wide scale manufacturing and adoption. Around this time the company will also support OEM and other business models.
Wait one more thing, the Next Generation,
After these first major stages are complete, the company plans to tackle the other huge aspect of recreating the Star Trek Holodeck, which is being able to touch virtual objects. This is a concept known in academia as volumetric haptics. “We aren’t disclosing much information about this technology at this point in time, however, it’s the ability to project both the holographic image as well as the tactile sensation of the object, all from the same surface and without any external accessories. With that said, it’s not something that will be demonstrated until after the initial product optimization” comments Karafin.
Some may see the arrival of a truly holographic monitor as another in a line of ‘gimmicks’ such as 3D TVs which failed to ignite the public’s imagination. An opposing point of view is that glasses free monitor walls such as this will be the killer tech beyond Magic Leap style AR that will fully transform our notion of entertainment. My own research leaves me to personally believe that such immersive experiences will be unlike cinema and unlike Stereo projection in their ability to communicate emotion. It is possible that the human brain may process such fully immersive experiences differently than we do passive screenings. Such a full immersion would be a leap beyond VR into a collaborative, immersive and very much shared experiences. The ability to provide images that are optically almost identical in density, direction and focal resolution makes this approach the most realistic visual experience yet attempted.
”Light Field Lab will target its real-world holographic experiences at both professional and consumer markets. Eventually, it hopes to build holographic video walls with hundreds of gigapixels of resolution”. Karafin is open about being inspire to make a Holodeck for many years, and the direct inspiration from Star Trek.