How The Weather Channel Is Pioneering Mixed Reality for Live Television

By Janko Roettgers

LOS ANGELES (Variety.com) – “Turn around, don’t drown!” Standing in a street with rapidly rising waters, The Weather Channel meteorologist Chris Warren is imploring drivers to steer clear of flash floods. Moments later, as if to prove his point, a car floats by. Only, Warren isn’t risking his life to record the network’s latest extreme weather segment. Instead, he’s being taped in front of a green screen in the network’s studios in Atlanta, and the dangerous flooding is being added via visual effects in real time.

The flash flood PSA is just the latest segment coming out of ’s Immersive Mixed Reality team, which has been producing similar videos about tornadoes, storm surges, wildfires and climate change. “It’s all about getting away from the flat map,” says weather presentation director Michael Chesterfield. Adds Warren, “It illustrates what we can’t always show in person.”

The Weather Channel began investing in immersive mixed reality in late 2017 to respond to a growing demand for video across all platforms, explains chief content officer Nora Zimmett. “The problem with forecasting weather is there is no video of the future,” Zimmett says. “So we decided to create what the future would look like based on data.”

The network has been recording about one mixed-reality segment per month since June 2018 and is getting ready to massively expand these efforts. By next year, The Weather Channel wants to feature immersive mixed reality in 80% of its on-air programming. Immersive video will also play a big role in the network’s partnership with Jeffrey Katzenberg’s Quibi, announced earlier this month.

The control room of The Weather Channel’s Immersive Mixed Reality studio.

To gear up for this enormous undertaking, construction crews are set to arrive at the network’s headquarters in Atlanta this week, where they will demolish part of its regular studio to make room for a massive green screen stage. Not only will the new set allow for more expansive segments, but meteorologists will also be able to walk directly from the anchor desk into mixed reality environments.

The Weather Channel is recording these pieces with the help of Epic’s Unreal game engine, which has long powered video games like “Fortnite,” and is increasingly being adopted by Hollywood studios for visual effects . While rendering similar effects used to take days, Unreal operates in real time. Warren is acting in front of a real camera, which is also used to track the virtual environment. When an SUV floats past him, the camera operator zooms in for a close-up, thanks to a preview monitor mounted to his camera rig that shows the composite image.

The camera operator sees both a live view of Warren in front of the green screen, and a composite image of him superimposed onto a flooded street.

Warren, who dons a Weather Channel outdoor jacket for this shoot, has similar monitors near his teleprompter. In addition, a projector beams a special color-filtered version the composite video right onto the green screen itself that isn’t picked up by the studio camera. This allows him to react in real time to cars and other virtual objects going by. “There is a sense of acting and drama that seals it,” explains Weather Channel vice president of design Michael Potts, who has been leading the network’s mixed reality efforts.

Some of the videos have indeed been quite dramatic. In one clip, which recently got mocked by John Oliver, a tornado seemingly rips apart the network’s studio. However, members of the immersive mixed reality team insist that they’re not taking creative liberties, instead using actual weather data to calculate the velocity and trajectory of every object hauled around by tornadoes or storm surges — a process that takes weeks of preparation.

The Weather Channel meteorologist Chris Warren revises the script for the network’s latest immersive mixed reality segment.

There’s another reason The Weather Channel is moving from static advisories to immersive-video presentations: People simply don’t listen to facts and figures alone. Some ignore warnings and evacuation orders, thinking that they’ll be just fine. Showing them how high the water can surge in an average living room, or how quickly even a heavy SUV can be carried away, can save lives, argues Zimmett. “To know that people are not only watching us, but changing habits or changing their family safety plan based on these visualizations, is the greatest return on investment that we can hope for.”

Speaking of safety: Mixed reality can obviously help keep on-camera talent safe as well. “Nobody wants to be in harm’s way,” says Warren. “We’d never put somebody into a wildfire,” adds Chesterfield — something the network was safely able to do in another mixed reality segment.

In most of its mixed reality videos, the network also goes out of its way to show the audience that the on-screen talent is safe from those believable-looking perils. Meteorologists like Warren often stand within a gray circle that marks a kind of virtual safety cone, shielding them from the destructive forces of nature. “It’s a mechanism that allows us to go right up to the edge,” explains Potts.

It’s a position the network also finds itself in with its ambitious plans for mixed reality, which are unrivaled in the industry. “The Weather Channel is a pioneer,” says Potts. “No one else is doing it. There is no leader to follow but us.”

The Weather Channel is about to tear down this section of its studio in Atlanta to make room for a bigger green screen stage.

Granted, news networks have for some time experimented with different ways to visualize data, including CNN’s infamous holograms, but Zimmett doesn’t think the competition has caught onto the potential for this technology yet. “They use it as very attractive production window dressing,” she argues. “It doesn’t serve a purpose.”

Going forward, the network is exploring bringing its mixed reality segments to VR headsets. And one day, it may even use real-time immersive media for local forecasting, according to Zimmett. One of the projects currently under exploration would allow viewers to point their phones at the street outside their window, and get an accurate augmented reality simulation of tomorrow’s weather. Says Zimmett: “I’m tired of seeing just a number and an icon that comes from 1975.”