The Future of Virtual Production: Hopes and Challenges, with Asa Bailey – Part II

Part 2 of our interview of Asa Bailey, Director of Virtual Production and Founder of On-set Facilities, a company dedicated to virtual production and computing solutions. We talk about the future of virtual production, the hopes and challenges that have arisen in the past few months and what we can expect from the spread of LED walls and virtual production in the film industry.

Read Part 1 here: The Virtual Production boom, with Asa Bailey from On-Set Facilities – Part I

If you’re interested in the future of virtual production for filmmakers, be sure to read our two-part interview about the first steps of two DPs into virtual production territory.

The challenges of virtual production

Alvaro Lamarche Toloza, Virtuals Co-founder:

Now that LED stages are sprouting up all over the world, do you see any new challenges coming up in the near future of virtual production? 

Asa Bailey, On-Set Facilities Founder:

I think it’s important to mention the reason why virtual production is reduced to “in camera LED virtual production”. The important part is that for us humans it’s really visible, it’s massive, big, it’s shiny, it’s exciting, although they do no different than a headset: they look into the virtual world and bring it up. The thing is, a headset is a very individualistic experience, whereas an LED stage is a party, a group experience and a huge window into the virtual world. And this is why it has grasped the imagination of filmmakers and producers and studios. They’re like, wow! And they can see it. And I think it’s the first time that the virtual world’s ever been properly looked into through a huge window. And so that’s why it’s blowing up.

An LED stage is a party, a group experience and a huge window into the virtual world. And this is why it has grasped the imagination of filmmakers and producers and studios.

Asa Bailey

This is why I talk a lot about the holistic view of real-time virtual production. Because a lot of the challenges come from not understanding what it is. Once you get over that then okay, how do we use it? And one of the biggest challenges is that there’s a lot of people that do not understand what virtual production is.

Alvaro:

Aside from a general lack of understanding of what virtual production is, have you encountered any specific issues while working with OSF?

Asa:

Those that have grasped the fundamentals and who get into the practical elements of how you do it get into a heap of problems. The first one we encountered was latency. We needed systems that were as low latency as possible to create the illusion of real-time interaction between the real and the virtual world.

A lot of systems that have come from other industries that are trying to seize the opportunity and move into this space bring with them an unusable architecture. When we used other systems, we’ve found a lot of latency, which then breaks the illusion for the user experience on set. Because then, it’s not moving properly or it’s slow, and you can’t have slow. So moving systems to a low latency architecture from their beginning is critical.

That’s the foundation, that’s the computing. You then get into the peripherals. What’s a peripheral? A Mocap suit, whether it’s an Xsens or a Rokoko or whatever, still plugs into a computer; the displays, the LEDs from where we see it are just huge monitors. They’re huge windows into the virtual world. And a virtual world is different to a video file, and a lot of systems that are being utilized in this area come from a playback or media server heritage. Well, that’s fine if you want to put very demanding video files down the tube to the walls, but we don’t wanna do that. We want to create a nonlinear stream to that wall, which could be going this way or that way, which could be going up or down. 

So essentially the video screens to us are huge monitors, and they come with their problems, color calibration, quality of light. Obviously you’ve got to get the basics right, with your pixel pitches and your refresh rates and all of these sorts of things. The thing is, you don’t usually shoot a monitor, but you’re gonna shoot this LED. So there’s a lot of visual exploration to be done in how the light affects it. You know, everything reflects light, but not everything generates light. And that’s the slight difference that we’ve got when we are looking at an LED. If you look at the wall behind me, it’s reflecting the light from the window, but it’s not generating that light and throwing it at you. It’s reflective from the sun. So we’ve got all of these kinds of explorations going on.

LED walls: lighting, color and the role of the DP

Alvaro:

That’s a pretty big topic, color and light in virtual production. And it has come up every time you try to dive a little deeper into what it actually takes to have a successful virtual production set with an LED stage. Shooting these huge monitors, as you put it, you lack the control that DPs will usually have, and the understanding that they have and the foundations that they have of solid color management.

Asa:

This is why we still need the DP. We need the DP, we need the cinematographer. I’ve moved over slightly to the dark side of technology. I’ve gone back to my roots in user experience design. I don’t have the experience of a three decade long career in light. So when I come to set, as the director of virtual production, I’m bringing the technical, I’m bringing the creative as well, and that’s my jurisdiction, my boundary. Then I turn to the director of photography and I say to him, does that look good to you? It’s not my eye that says no, those skin tones are just not quite there, cause I’m not the judge. I’m the arbiter of the user experience on set, of the virtual world. So the DVP does not go against, he fits in there with the director of photography, the VFX supervisor, the director as well. We all come together and I’m just the creative technical lead.

I try to shorten that learning curve as much as possible to get them up to their standard of shooting, where they’re happy as quickly as possible. You see, when it comes to the color management, I look through the camera and if the camera says it looks good, it looks good. In my color chain, as long as I’ve set everything up to be 10 bit HDRI, the screens are calibrated, the systems running, the correct signals are put out… Well, go to the camera.

We then have a number of spaces that I can manipulate to help the DP achieve what he wants. I can adjust the camera settings, I can adjust the processor settings and I can adjust the engine settings. There’s three. Now the DOP would normally ask for another light, some form of filtration on his camera, to change the exposure. He only has a certain number of options to change in this new environment. I have all of them, and that’s where the relationship works.

Bringing filmmakers into virtual production

Alvaro:

One of the issues I believe in virtual production is that it’s a lot of tech and a lot of different elements that come from different areas that people may not be familiar with. How do you handle this when working with the film industry, for example?

 Asa:

Yeah, it’s foreign to a lot of people in film. Not many people understand that if you’re to transmit data, you usually have to encode it and unencode it at the other end.  It’s not that difficult for me, but the truth is, you shouldn’t have to know, and you shouldn’t have to focus on it, that’s what we’re here for. You should learn by osmosis, you’ll learn by standing next to us. That doesn’t mean they have to then do it, but at least they can understand what we’re doing.

Here’s an ironic situation. All we do is have empathy for the rest of the people that are around us. That’s our job: to have empathy and to understand and to give the solutions that are required. And ironically, nobody understands us.

It’s a timing thing. We can’t all know everything straight away. There is a learning curve and an adoption curve. I’m involved in virtual production so it’s difficult for me to gauge where we are on the adoption curve sometimes, but I do get the feeling that we’re still right at the beginning, and only at the moment are the early adopters coming into the space. Even with the explosion of stages, you’re only seeing an explosion of stages because every time somebody sells a stage, it’s a big deal. When somebody buys a new camera, nobody goes and writes an article about it.

I do get the feeling that we’re still right at the beginning, and only at the moment are the early adopters coming into the space.

Asa Bailey

And this is one of the demons that sits on my shoulder. What happens if this just molds into production? It becomes normal. What happens to the VP department then? Well, then I’ll become a director of technology. And my crew, instead of VP engineers, they become on set engineers. It doesn’t really change anything. It just means that we’ve achieved what we wanted to. It’s a weird thing to ultimately want to not exist, to ultimately embed ourselves into the business and into the industry so that we’re invisible. But it does pose the question: when does that happen? That could take 25 years. So let’s not worry about that now.

The future of virtual production and its impact on the film industry

Alvaro:

What impact do you think virtual production will have on the future of filmmaking? Do you see any positive changes from the adoption of VP on a larger scale? What is your opinion on the future of virtual production?

Asa:

About the positive effects and the future of virtual production… I think I’m of the generation that has watched the world get pillaged by our fathers and our grandfathers. Even if I look at how much I’ve traveled, I don’t dare to think how many times I’ve been around this planet. And I’m not the only one. It’s so easy to jump on a plane to send a package anywhere and it’s gotten out of hand. Virtual production is virtualization and the ability for us to travel without traveling. The ability for us to work together without having to come together. All of these are parts of virtual production. 

Virtual production is virtualization and the ability for us to travel without traveling. The ability for us to work together without having to come together.

Asa Bailey

Remote works very well with the ecological and the environmental side of things. I firmly believe that if we get our way – and I’m talking about the other big tech companies – you won’t need to travel. You’ll want to travel. That’s two different things. You want to go and see your grandma? You get on a plane, mate. But you have to attend a trade for something that’s just not really that important? That can be done virtually.

Alvaro:

In Patagonia, for example, you can shoot very interesting landscapes, but it’s a very fragile ecosystem. And you know that if a shoot happens, if a crew of dozens of people hang around there, they’re going to destroy the place. There’s a lot of value as well ethically in not not going there and not being there, just sending three people. Maybe just to scan a couple of things.

Asa:

Exactly. I mean, it’s a valid reason: sending three people to Iceland to scan. What’s the other option? Twenty four people and loads of kit. When you can take three people and a case.

I’ve been directing in Spain. I directed a global campaign for one of the biggest brands in the world. I shot seven commercials in one week. I wasn’t even there. I have my crew, I’ve worked with them before. I have crews all over the planet that I’ve worked with that know me, know how I am, which is a very important part of making it work smoothly. I directed those commercials through a combination of NDI streams coming at me into my control center, me being on camera and directing the crew, some 20 people, including the client and everybody that wants to be there. And then we were also beaming to Russia, Italy, Germany, and they were watching as well. This was back in the middle of COVID. I think that it has shown to all who were involved in that project that you don’t need to do it another way. Why do we need to fly our head of brand from Moscow to Madrid? We don’t.

It’s a nice thing to do if you want to, but again, it’s gotten out of hand, hasn’t it? I think there’s an opportunity for virtual production to really help solve some of these problems that we’ve caused.

Asa Bailey
Director of Virtual Production

Asa Bailey is the founder of On-Set Facilities and has consistently been recognized for his innovative use of on-set technology as well as his unique approach to storytelling using virtual production systems. His work ranges from conducting virtual scouting sessions to building LED stages and developing camera tracking solutions for demanding shots.

Through On-Set Facilities, Bailey is routinely hired by producers worldwide to design, install, and support robust virtual production environments both on-set and cloud based. In this capacity Bailey has worked with DreamWorks, UNBC Universal, Universal Pictures, Unity, and the ARRI Systems Groups among many others. 

Today, Asa Bailey continues to direct his own projects and participates in various industry events, having delivered workshops for NVIDIA, the Directors Guild of America and MIPS. He sits on the board of Advisers of both the DAVE VFX School at Universal Studios and the Studio UK Government Working Group.