How "virtual production" is making next-gen movie magic

The green screen of the future is essentially a massive computer monitor, powered by video game software that can generate almost any environment a filmmaker might want in a TV show or movie.

• The tech, pioneered by visual effects (VFX) firms like Pixomondo, ARwall and Industrial Light & Magic, is called "virtual production."

Why it matters: Virtual production is helping filmmakers, actors and VFX pros create visually stunning shows and movies for the rest of us to enjoy.

Details: Pixomondo, a leader in the space, has three virtual production studios — soundstages surrounded by massive LED panels, which use the Unreal video game engine to depict everything from mysterious alien worlds to cavernous spaceship interiors.

• Actors and props are shot in front of the screens, as digitally generated content is rolling behind them.
• That gives actors a chance to see and react to what's happening on those screens — a huge leap forward from green screens, which are, well, just green, with content added in post-production.

Artists and designers build each virtual environment with input from a given show or movie's creative team — a back-and-forth process that "becomes kind of like jazz," says Pixomondo virtual production and VFX supervisor Nathaniel Larouche.

• The tech is a natural fit for sci-fi and fantasy storytelling in particular — it's being used to shoot two "Star Trek" shows and Netflix's live-action "Avatar: The Last Airbender" series, for instance.

One of virtual production's biggest benefits, says Larouche, is that it moves a good chunk of visual effects work from the post-production process into the actual shoot.

• That can make shows faster and easier to produce — and also enhance collaboration between directors, cinematographers, VFX pros and so on.
• And building a digital library of virtual environments can reduce long-term production costs. "As you go from season to season, you build up this giant library of things, and you can just pull in and kitbash this brand new environment from all of the pieces that you built in the years prior," Larouche says.

Flashback: Larouche says that, in some ways, virtual production is the spiritual successor to rear projection — a technique in which actors are recorded in front of previously shot movie footage rolling on a screen behind them.

• "It's bringing this brand new real-time technology back to the original roots, when visual effects and production were working together and things were actually being shot in camera," he says.

For Glen Keenan, director of photography on "Star Trek: Strange New Worlds," the magic of virtual production lies in the lights.

• With a green screen, he says, lighting — key to achieving a convincing visual look — is "complete guesswork." But Pixomondo's walls can themselves function as a highly adaptable and malleable lighting source, giving him and other filmmakers a powerful new creative technique.
• Still, Keenan sees virtual production as a supplement, not a substitute, for green screens. "I don't know if it's ever going to replace anything," he says. "I think it's a new tool."

Yes, but: Though building digital environments can be easier and cheaper than making big practical sets or filming on location, it still requires time and effort. Keenan's team, for their part, is trying to be thoughtful about when to go big with virtual production, versus when to keep things simpler.

• "As with any success story, there comes ambition," he says. "Everyone gets a little bit, 'What if we did this? What if we did that?' There's a constant play between taking a risk and doing what we know works."

What's next: Improvements in the underlying game engine and the components used to make these giant screens will open even more creative possibilities for virtual production tech.