The production company Diamond Bros recently captured immersive concerts for artists Black Pink and Doja Cat in virtual reality (VR) for Meta. They used high-resolution extreme wide-angle cameras, such as the Red V-Raptor 8K, to capture the performances in 8K 60fps. The footage was processed using Assimilate Scratch Live for live effects, warping and monitoring. The team delivered the final content in a de-warped equal rectangular format, allowing for an immersive experience in the Meta VR space, we spoke to co-founder and co-director Jason Diamond about their process.
Doja Cat, who completed the North American leg of her eye-popping “Scarlet” arena tour, released her first VR concert last week. The experience was created by producer/directors: the Diamond Bros in partnership with Meta. The immersive stereoscopic experience is called Doja Cat: The Scarlet Tour in VR and was filmed in Detroit date. The show was hailed as “a thrilling spectacle that blends reality and imagination, complete with cutting-edge visuals and beats.”
Blackpink, the hit South Korean K-Pop band of Jisoo, Jennie, Rose, and Lisa, also released a VR concert experience in Meta Horizon Worlds’ new venue, Music Valley at the end of last month. This concert allows fans to experience the final show of the “Born Pink” world tour, filmed at the Gocheok Sky Dome, via the Meta Quest headset.
These are just two of the latest immersive experiences from the Diamond Bros, the directorial partnership of Josh and Jason Diamond in New York. Jason will be well known to fxguide readers from his co-hosting of the VFXShow. Along with their third partner, head of Production Scott Gracheff, The Diamond Bros has grown into a full-service, Emmy Award-winning creative production company with creative partnerships and strong connections with various key immersive and virtual production teams. The brothers have a deep resume and technical skill base. They were involved from the outset with Frame.io, and today, they have a special affinity for virtual production and VR immersive experiences.
Meta reached out to the Brothers in the middle of August 2023 for the BlackPink concert. “Obviously, we are excited, and we worked pretty fast as dates got moved up,” comments Jason Diamond, “It was supposed to be at the end of September/early October, and then all of a sudden, it was mid-September.” To make the project more complex, – it had to be shot in Korea.
For the actual BlackPink concert, the team shot with a mix of eight modified Red Raptor cameras with Canon 8-15mm zoom lenses and three Canon R5C cameras. Red Control Pro controlled the eight Raptors cameras via a Mac laptop. The Pro app offers advanced features, including native iPad support, multi-camera control, quick settings overview, FTP file access, advanced LUT, CDL, PRESET management, or independent image orientation. But importantly, the app is also fully designed to simultaneously monitor live streaming from several cameras.
The team had live fiber-cabled feeds for 4K/60 SDI monitoring to Video Village, using a Blackmagic ATEM Constellation 8K, with two 60-inch 4K monitors for program/preview and multi-view of all eight Raptors. The team then fed the SDI/HDMI onto a laptop running Assimilate Scratch Live, which allows real-time de-warp of the wide angle/fisheye imagery and live monitoring into their Meta Quest 2 headsets. This allowed the directors to see the footage in VR to check blocking and framing – and make any adjustments needed to the cameras. The stage was massive, 150 feet wide by 80 feet deep, with an additional big thrusting stage section down the middle as a catwalk linking to another 20 by 40 ft stage further down the space.
“We’re using BlackMagic Gear and Red Control Pro to both control all the cameras and put ourselves in a pseudo broadcast environment where we can use a switcher to bring up a program view of something we want to see and be able to see in a multi-view all of our sources,” Jason explains.
This is how one might set up a live event. The concert was not live, but this approach was the most logical way the team thought to approach the recording. “The magic comes in for us from a VR standpoint, when we use that switcher to feed, Assimilate Scratch Live effects” Jason adds. “Live effects is Assimilate’s newest suite that is designed for Multicam sources as opposed to traditional Scratch, – which is mostly a post-production tool working with just normal clips”.
Scratch Live Looks allows an easy setup for on-set video monitoring and live grading up to 8K. The powerful de-warping of imagery in real-time for director review plus artist and management preview, were key to this workflow. “We picked our camera positions, set them up, and pointed the cameras in the direction we thought would work, but then we put on the VR headsets and adjust everything,” Jason explains. “We do all the standard camera positioning you would do in a traditional production, except in this case, we can do it in the headset and know exactly what it will look like.” This replaced having to take a file, export it, and then manually sideload it to see how it looked, which would be much slower and more cumbersome. “We can do a live switch on the switcher and, in the headset, see the image change in real time. As directors – and for the rest of our crew, especially our editor, – we can preview moments from the show in the headset and then know what will most likely be the key edits and shots of the final show.”
For the Doja Cat concert, the team decided to shoot in stereo and used the same Red cameras but with the 5.2mm f/2.8L Dual Fisheye, 180˚ VR lens. Originally developed for EOS R5, R5C, and R6 Mark II. The locked dual fisheye wide-angle lenses are ideal for 180˚ 3D VR video capture. The Diamond Bros. had three special-purpose Canon R5Cs for the Korean concert but did not use them for Doja Cat concert.
The production delivers in 8K-60, (4320×4320 de-warped) so live monitoring in VR on set is invaluable. In the post, the team edited, color graded, de-noise, and did the final de-warp in DaVinci Resolve. When shooting with the fisheye, two images are recorded on one sensor so the stereo reads just like a standard high-resolution video signal to all the broadcast gear. “It’s just like a normal 8K, 16 by 9 video frame,” he comments.
For the editing of the projects, the team uses an Apple Mac Studio. “We use the M-Ultra 2 Mac Studio,- full version, all the ram, all storage, – M2 Ultra,” Jason explains, “We use the biggest M2 studio, which arguably is not too expensive compared to what it does, and that you can fit it in your backpack. It’s crazy good – it’s a workhorse.”
The team shot around December 4th, then delivered Doja Cat on January 18th. “So it was a really fast turnaround over the holiday”. The team was approximately just seven people during Post.
Looking ahead, The Diamond Bros see great potential in the VR, AR, and immersive environments space, especially with advancements in technology and the growing interest from companies like Apple who are joining established VR/AR companies such as Meta. They believe the future holds exciting opportunities for creative expression and technological innovation.
