Closer to reality than virtual
By Clara Chong, Director
For over a century, storytelling for drama in traditional films has existed within a frame, a visual boundary that has defined the language and grammar of traditional film.
In contrast, immersive drama has been evolving in traditional VR, including VR180 and VR360, with one of the first VR video systems patented in 1962.
Apple Immersive Drama filmed with the Blackmagic URSA Cine Immersive camera stands apart from both traditional film and VR, but there has been significant confusion about what Apple Immersive Video (AIV) actually is.
From the traditional film world, AIV is often dismissed as simply VR under another name. From experienced VR filmmakers, AIV is sometimes seen as another iteration of systems that have existed for decades. But AIV is a fundamentally new format. It sits beyond the limits of traditional VR or traditional film.
What distinguishes AIV from earlier VR systems is a combination of extreme technical factors that bring it closer to normal human vision than what we associate with VR systems. This includes the 180° horizontal and vertical views that exceed human peripheral vision, ultra-high resolution approaching what the human eye can discern, true stereoscopic 3D, very high frame rate at 90fps, and 1:1 world mapping inside the Vision Pro.
It’s AIV’s 1:1 relationship between every pixel and a vector into the real world that allows for immersion without distortion. Combined with the pixel count and frame rate, this gives AIV a new level of spatial and temporal resolution approaching that of the human eye, and makes AIV closer to reality than virtual.
It’s this degree of accuracy that differentiates it from any other traditional VR system or traditional film format.
These differences create a new language and grammar for Apple Immersive Drama – from writing and blocking, through editing and final review.
The first scripted film captured in AIV was Submerged, a World War II submarine film directed by Academy Award-winning filmmaker Edward Berger (All Quiet on the Western Front).
Main Course Film’s new Apple Immersive Drama, The Dobos Connection, produced in collaboration with fxguide, further tests the narrative possibilities with AIV.
The Dobos Connection was shot entirely on the Blackmagic Design URSA Cine Immersive camera and edited and finished in DaVinci Resolve Studio using the full Apple Immersive and Spatial Audio workflow.
“The URSA Cine Immersive camera and DaVinci Resolve Studio ecosystem makes capturing, editing and finishing Apple Immersive Video accessible and controllable at the level that professionals expect from high end camera and post production systems, despite the inherent complexities of the new format.”
Ben Allan ACS CSI, Producer/Cinematographer
Two approaches to immersive drama
There are currently two distinctly different approaches to immersive drama: “VR-Style Immersive” and “Cinematic Immersive”.
(1) VR Style Immersive
The VR-Style Immersive approach allows performance to unfold in extended takes with minimal edits.
Camera positions may change between scenes but there is no intercutting within a dramatic beat, performance unfolds in real time and stereoscopic continuity is preserved by avoiding complex cutting.
This approach treats immersive drama as a virtual stage where the viewer is placed within the sphere of performance, observing events unfold in stereoscopic depth but without editorial construction shaping the pacing or performance.
(2) Cinematic Immersive
The Cinematic Immersive approach, in contrast, constructs narrative through editing. Similar to traditional cinema, there is intercutting within scenes, multiple camera positions inside a single dramatic exchange, designed proximity shifts, narrative shaped through editing to expand or compress time for both pacing and performance.
AIV enables the emergence of a new cinematic form that is distinctly different from both traditional framed film and traditional VR.
AIV doesn’t replace traditional framed formats, it extends the way that filmmakers have always chosen between a range of storytelling formats. Like choosing between Scope or Academy or IMAX, AIV will suit some stories and filmmaking style more than others. But the cinematic reality of AIV is that has more in common with the scale of the shift in thinking from stage to cinema than with different screen formats.
It is this new Cinematic Immersive approach that we adopt with The Dobos Connection.
Simple is hard
Martin Scorsese is quoted to have once observed “There’s no such thing as simple. Simple is hard.”
AIV has already established the format as an unmatched powerful experience in sport, action, and music – Live NBA games, the money-can’t-buy rock star experience of Metallica, the exhilaration of F-18 fighter jets in action in Flight Ready, pushing the creative envelope with VFX in The Weeknd’s “Open Hearts” music video, the exhilaration of the outdoors and the camaraderie of extreme sport in World of Red Bull: Backcountry Skiing.
But while it’s easy to be distracted by spectacle in AIV, AIV also excels in conveying a sense of compelling authenticity at the same time as high-level professional storytelling. The quiet beauty of minimalism with Boundless: Arctic Surfers, the immediacy of being with the exotic animals in the Wild Life series, the warm glimpse behind the curtain in Alicia Keys: Rehearsal Room, and the focus of extreme athletes in the zone in Highlining & Ice Dive of the Adventure series. There have also been steady additions to new AIV narrative documentaries amongst the Apple Immersive community, such as Jeanne-Rose Tremski’s Mario Luraschi, which effectively uses proximity to convey the warmth and energy of her subjects and the spectacular extended scene across the skies in Open Air by Rogue Labs.
By placing the viewer inside the space, AIV invites the viewer to instinctively read body language, micro expressions and gestures in the same way they would in person. Without the boundary of a frame, the viewer shares the same environment as the characters, registering glances, hesitations and emotional shifts with the same immediacy as real life.
Submerged’s powerful opening shot, visceral experience of feeling the water level rise up, extraordinarily detailed production design and visual spectacle gave filmmakers a glimpse of what was possible for drama in AIV.
With The Dobos Connection, our goal was to explore how AIV impacts narrative storytelling in its simplest form.
The story is deceptively simple: two professors, one room, one quiet university weekend, a breakup, a Dobos torte, and the possibility of something unspoken.
Simple drama, however, is one of the most unforgiving forms of filmmaking, and a drama built around two characters in a room requires precision, nuance of performance, blocking, shot choice, rhythm, because you can’t hide behind spectacle.
Building the Grammar
Main Course Films undertook extensive tests before embarking on The Dobos Connection.
Beginning with simple static shots of scenery, then cutting together sequences of shots, formal studio tests to study the impact of proximity and camera height, before beginning a narrative documentary project Thé Fine Dining Bakery. We used the learnings from the documentary to inform the approach to the more complex drama project.
AIV is both a technical and perceptual shift. The focus moves from what the camera captures to what the viewer actually experiences, biologically and emotionally, and this changes the foundations of cinematic language.
Like aviation, where a jet may be capable of descending faster than passengers can tolerate, the challenge of AIV storytelling lies with the limitations of the human body and mind. The brain and body can only process so much motion, shift and proximity comfortably, so understanding these limits helps the storyteller create within viewer comfort thresholds.
The absence of a frame changes narrative control from choosing what the viewer sees to directing where the viewer looks. It’s about changing the psychology of what the viewer notices.
Attention needs to be directed through blocking, sound design, light and three-dimensional staging. This affects how the story in AIV as the script needs to think about where attention may fall, how it moves and how emotional emphasis is created without the boundary of a frame.
“Creatively, composing the score was pretty exciting because it needed to accommodate the wider viewing format while integrating traditional scoring methods. The scoring needed to account for the viewer moving their head to check the 180-degree view. But the fundamentals of traditional scoring helped to set a dramatic tone and draw attention to key moments.”
Carlo Giacco, Composer
Writing begins with the format
Writing for Apple Immersive Drama is directly shaped by the AIV format. The script format doesn’t change, but the grammar of storytelling does. AIV changes not only how drama is filmed, but how they are experienced.
Storytelling in AIV begins with understanding the viewer experience, where they can look, how long it takes to reorient with a cut, how proximity shapes emotion.
Writing the script for The Dobos Connection was about using AIV to amplify the quiet moments that change things between the two professors played by actors Lauren Clair and Thomas Ah Kuoi, their connections, eye contact, and the hesitation of unspoken words.
Music was also written into the structure of the script in 5/4 time to help keep the narrative slightly off balance, amplifying the asymmetry of love arriving late, unexpectedly, and imperfectly.
Every shot matters
AIV returns filmmaking to the precision of the celluloid era, where every shot is intentional due to it’s massive data footprint which impacts what to shoot, how much to shoot, how to preview, production and post-production workflows and delivery.
The extreme high resolution of the image is not just an incremental increase, but an exponential one.
This graph begins in the era of SDTV, delivering roughly 10 million pixels per second. HDTV felt transformative at around 50 million. A 2K film scan pushed that to 76 million, then 4K DCI digital cinema at 212 pixels per second became the professional industry benchmark.
Each step felt significant at the time, but to understand the enormity of the difference with AIV, the scale has to be recalibrated to show SDTV, HDTV, and even 2K as barely registering. 8K DCI reaches 849 million, IMAX 3D HFR climbs to approximately 1.1 billion pixels per second. Now, with AIV, the URSA Cine Immersive captures a mind-blowing 10.6 billion pixels per second. So, while each technological leap over the past three decades felt like a new frontier, AIV doesn’t sit on the same curve, it bends it.
The technical & creative complexities of working with both a new format with this amount of data means that a short drama in AIV is at least as complex as a feature film in traditional cinema.
The Dobos Connection, for example, is a 10-minute drama which generated well over 11 TB of data in a single camera, 2-day shoot.
While mindful this was a proof of concept and the production and post-production process will shift as more projects evolve and more is learned about creating Apple Immersive Drama using the Cinematic Immersive approach, there were 22 full edit versions and 74 WIP renders.
As a comparison, Thé Fine Dining Bakery had 6 edit versions, which highlights the complexity and additive challenges of drama over documentary. The comparison is even more significant when comparing this 10 minute short with the 110 minute feature film Dark Noise which also had just 6 full edit versions, but was a traditional – framed – film for cinema.
Designing the look
For the look of The Dobos Connection, I wanted a style that felt controlled, sculpted, and intimate, a kind of “immersive noir minimalism”.
Knowing that noir relies on side and backlight to create shadows, this was going to be a challenge with lighting positions to create this look usually inside the 180° AIV field of view.
The high frame rate of AIV also doesn’t easily lend itself to low light filmmaking, but on Thé Fine Dining Bakery, we discovered that low light night exteriors with a fixed lens at 90 fps was possible, so I was keen to push the boundaries further with creating a stylised look for drama that was unexpected in Cinematic Immersive.
Using the right location was key to successfully pulling off our noir look. Real locations need depth, scale and three-dimensional logic in mind, with depth having enough range to support blocking, viewer gaze exploration inside the Vision Pro, as well as emotional storytelling.
“We created a custom Show LUT using Resolve’s Film Look Creator to introduce a partial bleach bypass look. What we found was that even though this is a look associated with photochemical film and isn’t experienced in real life, it is easily accepted by the viewer and becomes part of their experience of the world of the film when watched within the Apple Vision Pro”
Ben Allan ACS CSI, Cinematographer/Colorist
Composition without framing
In traditional filmmaking, we rely on coverage – wide, mid, close, reverse and so on.
But the Cine Immersive has a single immersive focal length, which means no close-ups, no macro, no cutaways to force focus.
While the terms composition and framing are generally used interchangeably in the industry, without a frame, the concept of framing doesn’t exist, but the composition of the shot and the perspective of the camera are still essential.
Instead, a shot’s composition is defined by The Three P’s – Proximity, Positioning and Perceptual Salience.
The position of the viewer’s presence inside a scene is determined by where the camera is placed and then the composition of all the elements in the scene around that position. This means that instead of composing a shot for a frame in traditional storytelling, the composition of a shot for AIV means composing a space relative to the precise camera position.
“I believe it’s important as a performer to be aware that your audience will be in the room with you in order to better craft their experience…with The Dobos Connection, the audience is amidst some very vulnerable and intimate moments”
Lauren Clair (“Eva”), actor
The camera becomes the viewer’s eyes with the world all around them so storytelling is no longer about directing the viewer’s attention through the frame, but where the viewer might look, how they feel and what their body and brain can actually process.
AIV storytelling activates the same parts of the brain that help us relive, not just observe. This realisation impacts on everything – from performance, to pacing, to editing.
On The Dobos Connection, rehearsals were extensive – both via Zoom and in situ so that blocking could be tailored precisely. The script explored a subtle romance filled with nuance and emotion, so rehearsing in situ allows us to work out the movements of the actors to fit with the emotional beats of the script – and at the same time, create opportunities for effective camera positions and potential cut points.
“I felt I needed to ‘block’ with my whole body and remember what I was doing physically every take for continuity. I used my whole body because there was nowhere to hide…With traditional screen acting you might get away with mistakes by editing it out in cut aways or framing them out. Because the angle is so wide and fixed, there’s nowhere to hide.”
Thomas Ah Kuoi (“Beau”), actor
Planning was equally extensive as the technical precision of where each setup would be, how much to shoot when the data was big, working out at what points to transfer data, and how many variations in proximity within a scene I would need.
“I was aware that what was required was an alteration of my natural disposition and physicality in order to be faithful to the role. I feel I’m personally quite animated, both physically and vocally, and in playing Eva, especially in an environment where the audience sits in, I had to make relevant adjustments to my pace, the tone of my voice and, knowing any movement can re-direct the eye in this space, be aware of my actions being more deliberate. I approached this as a naturalism and relied heavily on listening to Tom and being sensitive to the physicality between us.”
Lauren Clair (“Eva”), actor
Our focus was on how the viewer perceives the sense of Presence and Emotion. We wanted to plan sequences of shots to create cut points and editorial options so that we could maximise the authenticity of performance and emotional intimacy through editing.
Where immersive becomes cinema
Apple Immersive Drama becomes cinematic in the edit, because it is in the edit that the narrative is shaped through intercutting, proximity shifts and pacing. Traditional cinematic editing only evolved beyond continuous staging after filmmakers discovered they could intercut within a scene to control emotional perspective and pacing.
Editing in traditional cinema is such a powerful tool because it allows the viewer to see each moment from the best angle, but it also allows the filmmakers to craft the experience with the best performance, moment by moment, often from different takes, and using expanding and contracting time for dramatic effect and for smooth narrative flow.
“The cheeky bugger component to scoring was the concept (or distortion) of time and how that affected my compositional style. The pace of the footage for the Dobos connection was slow, and any music composed that was rhythmically too fast jarred with the footage. As a result, the final score was slower than originally intended. Paradoxically, because of the AIV headsets’ innate immersive nature, external stimuli are not accessible while watching content. As a result, during playback of the final music with the AIV headsets, time seemed to pass more quickly. How’s that for a head spin!”
Carlo Giacco, Composer
Editing drama in AIV benefits from elements and techniques from traditional filmmaking, VR…and from real life, but the heightened awareness of the viewer’s physiological limits means that the challenge is not simply to adapt the old rules from either film or VR, but to rethink them from a new starting point.
In many respects, AIV sequences are governed by the same rules and expectations as traditional formats, with things like the 180° line rule and crossing the line functioning exactly as you would expect. However, every edit in AIV doesn’t just give the viewer new information; it moves them to a new place, and every time this happens, their brain needs a moment to catch up.
“I was taken back with how wide every shot was and getting used to this feeling of being a third wheel witnessing these encounters unfold.”
Thomas Ah Kuoi (“Beau”), actor
What might be a 2-second shot in traditional filmmaking may need 6-10 seconds in AIV – not for technical reasons, but to preserve comfort, comprehension and emotional continuity. The length of a shot is determined not only by story, but by how long it takes the viewer to find the action and settle into the space.
Previewing edits inside the Vision Pro is essential because familiarity with the sequence can mute your instinctive response to shifts in perspective. You can’t unknow what you know, so it becomes difficult to judge how surprising or disorienting a cut might feel to someone seeing if for the first time, which is very different to traditional editing. While you start to gauge what will and won’t work inside the Vision Pro, it’s really helpful to get feedback from someone who hasn’t watched the edit yet.
The Dobos Connection created significant editing challenges. The combination of a small number of characters in a real-world location with continuous dialogue and nuanced actions leaves hardly anything to cut away to – and nowhere to hide edit cheats.
What quickly became significant was the importance of the viewer’s gaze and how we used choreography – blocking – to create new camera positions.
The accurate depth perception and 1:1 world mapping makes it easy for the viewer to instinctively follow the characters’ eyelines, but it’s critical to use blocking to allow options for different camera angles to intercut comfortably. By choosing emotional beats to allow physical space for new camera positions on set, this allowed us to intercut close proximity coverage in a way that enhanced emotional impact while remaining seamless within the Vision Pro.
“Audiences are now so screen savvy that experiential is where they’ll have the appetite for next. We’re also pretty chronically online these days, so experiential entertainment may well be a kind of antidote in that we’ll feel as if we’ve shared something intimate again.”
Lauren Clair (“Eva”), actor
Another technique we discovered was to cut mid-action, rather than before or after it. In Immersive, cutting mid-movement helps the viewer intuitively register continuity, maintaining their mental model of the world and the story.
“Technically, due to the inherent circular vision of AIV, working with the footage required that a zoomed-in 16×9 be cut out of the centre and placed on a split screen next to the original footage. The 16×9 side enabled closer viewing of the dramatic elements to better understand what was going on, and the original footage side allowed checking for any dramatic spill from the 16×9 side. For this project, we decided that the music would be in stereo and all other fx/dialogue would be in Apple Special Audio Format (ASAF). It would be very cool to score future AIV music in ASAF to take advantage of the 360-degree realm, placing music elements to make the most of what the format offers.”
Carlo Giacco, Composer
An integrated ecosystem
The ecosystem Apple & Blackmagic Design have created from camera through to post, and delivery has made the AIV workflow easier than even traditional stereoscopic workflows, despite the additional challenges of the format.
The Blackmagic URSA Cine Immersive camera is the first camera on the market to meet the unique requirements of the AIV format, and does most of the complicated stereoscopic work automatically in the background.
The Cine Immersive’s in-factory calibration connects each pixel to a point in the real world. This metadata stays with the images all the way through to the Vision Pro, where all of the pixels are remapped out to their real-world position but adjusted for the individual user’s eyes in real time. What this means in practice is that it can capture and deliver with a degree of accuracy that we’ve previously only seen in the real world.
What this ecosystem means is that AIV is not simply another immersive format, but a fully integrated cinematic medium that allows Apple Immersive Drama to be crafted with the same precision, editing flexibility, creative scalability and narrative control, while redefining storytelling without a frame.
Technology context
It’s helpful to consider AIV in the context of technology.
Real technological innovations take time to evolve before they become widely adopted and part of everyday life. If you look at the developmental journey of Alexander Graham Bell’s invention of the telephone to the powerful iPhone you can now fit in your pocket, early versions rarely resemble what follows.
If we look at the Apple Vision Pro in a similar context, where it sits in the technology timeline, it has more in common with the Motorola Bag Phone of the 1980s. Like the way the Bag Phone made the telephone portable for the first time, the Vision Pro has introduced the concept of spatial computing.
Someone looking at the Bag Phone in the 1980s might have been able to imagine a phone that could fit in your pocket one day, but would have been hard-pressed to imagine that it would also become a computer, a music player, a Sat Nav, a camera and a video camera.
While the Vision Pro is already sleek and sophisticated, it plays a similar role in introducing an entirely new category. And we are still at a stage where we can only begin to imagine what the hardware will evolve into – and what that evolution will mean for Apple Immersive Drama.
Beyond the frame
Early cinema with a frame was simply recorded stage performances – one wide, static shot. But as filmmakers explored the medium, they developed a whole new narrative language for filmmaking as they began exploring shot size, camera movement, montage and editing to control time, space, and emotion. From Silent to Sound, Black & White to Colour, Film to Digital, Mono to Surround Sound, each transition expanded the grammar and language of cinema, such as continuity, the 180° rule and cutting on action, all of which never existed in live theatre.
Framed cinema taught filmmakers to control attention through composition and montage. Without a frame, a very different visual language applies.
Submerged marked the beginning of narrative storytelling in AIV, demonstrating scale, spectacle and the visceral potential of the format.
With The Dobos Connection, we have shown the emotional power of how AIV sustains simplicity – the subtlety of performance, the tension of proximity and the weight of unspoken emotion.
“Making The Dobos Connection proved to us that it is both possible and worthwhile to apply cinematic principles to the Apple Immersive Video format in drama production. The powerful capacity to shape the intensity of the emotional experience using cinematic composition, lighting, look design and editing is possible without breaking immersion when done correctly.”
Ben Allan ACS CSI, Producer
The extremes of these projects reveal the range of Apple Immersive Drama – from spectacle to intimacy – and the potential for dramatic storytelling across genre and scale.
Just as early filmmakers had to discover continuity, montage and camera movement through experimentation, we are now discovering the new language and grammar of Apple Immersive Drama.
If cinema with a frame defined the past century, Apple Immersive Drama marks the beginning of a new chapter – one where cinematic storytelling is shaped by The Three P’s of Proximity, Positioning and Perceptual salience, and we are only just beginning to understand the breadth of its creative, emotional and technical possibilities.
How to Watch The Dobos Connection on Apple Vision Pro.
The “Theater” app (by Sandwich Vision) for Apple Vision Pro, which supports indie immersive content and can be downloaded here.
When you have the App: the link for the film is : theater://view/the-dobos-connection