VIRTUAL PRODUCTION: A NEW ERA IN FILMMAKING
By Wyatt Cagle
It’s been just five years since we got a glimpse behind the scenes of ‘The Mandalorian’ and its groundbreaking use of a new filming technique known as "The Volume." But what exactly is The Volume and virtual production? Simply put, it’s a cutting-edge video production technique that combines traditional filmmaking with digital technology to create dynamic backdrops in real-time. This method is being adopted across various industries, from film and television to video games and advertising.
Virtual production is both magical and challenging. We recently completed a large-scale virtual production for an Intel product launch, and I can confidently say it was both liberating and restrictive. The virtual production workflow offers countless creative possibilities—but it also comes with its share of potential pitfalls. Currently, the technique excels with specific types of scenes and settings. In-camera visual effects (ICVFX) are particularly effective for epic or fantastical productions that have sufficient resources and preparation time.
The Growing Appeal of Virtual Production
In our recent project, we managed to travel from the streets of New York City to the jungles of Brazil, and even up to the International Space Station—all within three production days. This wouldn’t have been possible in such a short time frame without the use of a virtual set. By combining real locations, elaborate art department builds, and virtual sets, we achieved a level of production quality that also saved our client’s budget.
Key Types of Virtual Production
1. LED Volume: This technique involves projecting CGI (and sometimes filmed elements) onto large LED walls, rendered in real-time during filming. This setup allows actors to interact with virtual environments they can see on set through monitors or headsets. However, this process requires extensive asset creation to generate detailed digital environments.
2. Virtual Cameras: These are used to capture scenes within fully digital environments, offering filmmakers the freedom to move and explore virtual spaces while mimicking traditional camera movements.
3. Motion Capture: Motion capture technology uses sensors to track actors’ movements and translate them into digital characters or effects.
4. Projection-Based XR: Similar to LED volume production, this method uses projectors instead of LED screens. While it provides immersive visuals, it also comes with unique limitations, such as the need for precise alignment and calibration.
Virtual Production: An Overview
*The Mandalorian* revolutionized filmmaking by combining techniques from moviemaking, live broadcasting, VR, and gaming. Volume studios generally consist of large LED walls that create an immersive “wrap-around” environment, blending live action with virtual worlds in real time. The low latency of this setup means that digital projections can adapt instantly to camera movements, providing a seamless visual experience.
A significant advantage of virtual production is the control it offers over lighting, reflections, and interactive elements. It also provides a sense of immediacy for both actors and filmmakers, allowing them to see the combined final product while shooting. It's amazing to have the option to film in sunset lighting all day long.
Hybrid Approaches: Virtual Production and Green Screen
Virtual production also allows for hybrid approaches that combine green screen techniques with XR (Extended Reality) environments. By shooting at high frame rates, along with high refresh rates on screens, filmmakers can capture both green screen and XR content simultaneously. This approach allows for multiple layers of footage to be captured, including green screen elements, tracking markers, and XR—all on the same sensor with minimal time offset. An example of this technology is Ghostframe, which enables shooting in high speed and capturing valuable data from “in-between” frames that would typically be discarded. Leading to even more possibilities in post-production.
Real-Time Rendering: A Game ChangerA breakthrough in virtual production is real-time rendering technology currently with the UNREAL engine, which digitally composites multiple elements directly in-camera during filming. This real-time feedback loop allows for faster decision-making and enhances collaboration between the director, cinematographer, and visual effects team.
Virtual Production: Pros and ConsLike everything in the video production business, you need to know the pros and cons of every tool and technique.
Advantages:
- Immersive environments with precise control over lighting and reflections
- Cost efficiency and time savings at scale
- Extensive assets and libraries available within the Unreal Engine workspace
- Creative flexibility and real-time collaboration
- Ability to sync lights with the content on the volume
- Sustainable practices with less travel and physical set requirements
- Versatile applications across film, television, advertising, and gaming
- Seamless integration with CGI and motion capture workflows
- Potential to capture a hybrid of chroma key, tracking data, and XR footage
Disadvantages:
- Not suitable for all workflows and storytelling styles
- High cost of setup, particularly when combined with traditional location or stage-based shooting
- Requires extensive pre-production planning and digital asset creation
- Limitations in achieving complete realism, scale, and physical interaction
- Certain lighting conditions can be challenging to replicate
- Calibration and setup can be time-consuming
- Physical space limitations and the delicate nature of LED screens can restrict certain types of shots (e.g., rain, pyro, snow, or stunts)
Virtual Production in Advertising
Virtual production is becoming increasingly popular in advertising, especially within the automotive sector, which has always embraced process studio setups for shooting driving scenes. Recognizing its potential, many agencies are now seeking partnerships with XR studios and post-production houses (such as South Coast) that offer hybrid solutions for quick turnarounds.
Examples of Virtual Production in Advertising
One common giveaway of XR footage is the hyper-shallow depth of field look, but as real-time rendering systems advance, they will better mimic the depth and focus shifts on LED walls. We’ve addressed this challenge by using strategic art department partial sets that build up real foreground elements, striking a balance between digital and physical to sell the illusion.
New Roles in XR Content Creation
The rise of virtual production has led to the creation of new roles and responsibilities in XR content development. For instance, the Virtual Production Supervisor, such as our own Derek Dziedzic, acts as a liaison between directors, cinematographers, and technical teams, ensuring smooth integration of real-world elements with digital environments. These roles require expertise in both traditional filmmaking techniques and cutting-edge digital technologies.
AI and Virtual Production: Emerging Possibilities
Generative AI is opening up exciting possibilities for XR content creation. By using algorithms that can autonomously generate diverse content, AI is poised to automate aspects of the creative process, while still maintaining a unique and imaginative touch. The integration of AI will likely enhance virtual production workflows, revolutionizing how films, TV shows, and other media content are created.
Conclusion: The Future of Virtual Production
As technology continues to advance, the potential of virtual production is expanding rapidly. Bespoke and DIY solutions are emerging, making it easier for creators to experiment with XR and AI in their workflows. With growing adoption across industries, we're likely to see a wide range of innovative ICVFX content on screens in the near future.
Virtual production has already transformed the way stories are told, and it offers endless possibilities for those willing to embrace it. If you're interested in exploring how virtual production can elevate your next project, South Coast Media Services can help you navigate this cutting-edge approach. With our expertise and tailored solutions, we can bring your creative vision to life while maximizing efficiency and impact. Contact us to learn more about how we can partner with you on your next virtual production adventure.