A quiet revolution is taking place among cinematographers and VFX professionals. Real-time filmmaking is quickly taking center stage, as demonstrated by the numerous demos and write-ups about the use of Unreal and Unity in production; last spring, the first (virtual) edition of the Real-Time Conference gathered professionals from all over the world to discuss the future of the tech. People are enthusiastic, to say the least, some calling it a game changer for a number of areas, from film production to live events.
Laymen may have a hard time realizing the potential of real-time rendering and more importantly of the game engines that are behind this push. After all it’s all still CGI content, so what is so different about it?
To answer this question I decided to get my hands dirty, although I have no technical background whatsoever in 3D software. Here’s an interesting test: what could a complete beginner like me achieve with the Unreal Engine? The answer is a short film I created called Le Dormeur du val, but we’ll get back to it later.
Table of Contents
- My not-so real-time journey: how it all starts
- Adapting Le Dormeur du val by Arthur Rimbaud
- Creating the Unreal Engine level
- Is real-time filmmaking that big of a deal?
My not-so real-time journey: how it all starts
You know the song: spring comes, and with it a global pandemic that shuts down Paris, where I live. Like many others, I find myself stuck at home, all projects on hold, suddenly unsure about my close future. I did the only sensible thing left: setting up Blender, Unreal, and starting to follow tutorials and online classes to see what I could learn.
Excel can be boring, sometimes
Up until recently, I had occupied business-related roles in games and VFX, closely working with artists and developers all the while sticking to my own guns – spreadsheets, press releases and the like. I learned the theory of 3D and digital humans in particular as it became my field of expertise business-wise. However, I couldn’t sculpt a simple donut to save my life.
Le Dormeur du val was my first creative work in years, despite holding a bachelor in film production. Years ago, I gave up on cinematography and started a non-artistic, business career after entering a prestigious school, figuring it would be a waste to go for a less profitable option after making it thus far. Nevertheless, the desire to tell stories and create new worlds remained in a corner of my mind, which is why I became interested in virtual production and real-time filmmaking in the first place.
Enter the Real-Time Shorts Challenge
The Real-Time Shorts Challenge was organized by MacInness Studios, led by industry veteran John MacInness (who would’ve thought!). A great portion of the world population was then quarantined, a great deal of movie productions had come to a halt. Virtual production, an opportune solution to keep on working, became a hot topic. This is where the RTSC comes in: filmmakers would have 30 days to create a short film using the Unreal Engine, working remotely from all four corners of the world. A perfect opportunity.
MacInness Studios provided two starting scenes to participants, of which I chose to use a single digital human, a rigged soldier model. I wanted to try and do something different with it, and that’s when I thought about adapting the poem Le Dormeur du val by Arthur Rimbaud. With very little experience, alone and from my small Parisian apartment, I set to dive into real-time filmmaking by creating a short film in less than 30 days in what you could call an artistic sprint.
Adapting Le Dormeur du val by Arthur Rimbaud
Le Dormeur du val is a short piece about a young boy peacefully dozing in the sun, lying on a generous bed of thick grass and vibrant flowers; as the poem progresses, we realize that this idyllic scene represents a soldier lying dead on the ground. It was written by Rimbaud in 1870 when he was just 16 years old and is most likely influenced by the Franco-Prussian war of the same year, a stinging defeat for France and a time of turmoil for its citizens.
Impressionism and iconography: references, inspiration
The visual style of the short is inspired by the art of the time: the early years of impressionism and works from artists like Sisley or Monet in the 1870s; both painted landscapes from the surroundings of Paris, places that Rimbaud himself could have visited during one of his many wanderings. I also took inspiration from earlier paintings such as the “Dead Man” by Manet, which is the reference for the soldier’s pose. Such mentions were an attempt at creating an echo between the main character, a 21st century Navy Seal, and the time of the poem.
The verses depict a luxurious clearing and use a certain number of important figures, among which colorful flowers that surround the main character or a warm sky from which a generous light pours onto the scene. I added a couple of new elements: the poppy, which is a common symbol to commemorate fallen soldiers and is very present in the Afghan war zone; I also used a tree trunk from Megascans and a rifle as ways to structure the space around the main character and to suggest the presence of death and war in an otherwise peaceful and lively scene.
Real-time filmmaking: less is more
Completing a short film in less than 30 days was a challenge, especially as my goal was to create a finished, standalone piece. I often see 3D artists work on demos that either have no apparent story or appear to exist as a fraction of a bigger storyline, and while I do understand why it is so… I find it frustrating nonetheless. To think about all the marvelous spaceships, amazing animations, crazy characters that have been rendered only to end up indefinitely shelved on a hard drive!
The overall project is consequently fairly simple as I set early on on a couple of limitations, based on my own understanding of what were the most time-consuming aspects of the tech. I also had no hardware to work with other than my computer, so handheld virtual cameras were out of the question – though I do think that virtual shaky cams are a little overrated and overused, but that’s a different topic. My own rules were:
- Very few animations and simple camera movements;
- No facial animation and consequently no in-frame dialogue;
- Everything would take place in one small setting.
Having a clear framework of dos and don’ts reminded me of the classical unities of French theater. 17th century playwrights, heavily inspired by antique dramaturgy, followed a set of rules which limited the scope of their plays to a single day, a single place, and a single plot: these were the unities of time, place and action. Some of the most renowned French plays were created while respecting these constraints, proof that such limitations shouldn’t be a hindrance to creativity.
Creating the Unreal Engine level
I had followed a couple of tutorials and online classes prior to the challenge in order to achieve a basic understanding of the different tools I would use: Foliage mode, Sequencer, lights and materials. Getting started with Unreal and real-time filmmaking is quite simple, and I think that any beginner can get into it and start creating in a matter of days.
Foliage, Megascans, lighting, materials: creating the scene
I started by looking for shortcuts that would help me save time. Starting things from scratch is a great way to learn but unexpected roadblocks can be frustratingly time-consuming and could prove problematic given the 30-days deadline. I took some time to look at free marketplace assets and found a couple of packs that would provide essential elements: landscapes, foliage, clouds. After opting for a simple landscape, I found an interesting spot and started blocking the scene itself. I then created a small setting, using Quixel Megascans and Bridge to import rocks, ground meshes and surfaces to create a realistic environment that could represent the clearing described in the poem. Using the foliage tool, I added a generous amount of grass, clovers and flowers with little regard for optimization apart from a couple of lightmap adjustments. The foliage tool in particular is very powerful and was very fun to use, whether it was to create a tree line or to add a patch of flowers in the grass.
Another crucial element was the lighting, where the Unreal Engine also shines in simplicity as well as performance. An outdoors, daytime setting is fairly easy to set up, using for the most part a directional light (the sun), a skylight and a sky atmosphere to quickly create different lighting conditions. Spotlights, rect lights and reflection capture volumes were of great help when highlighting or fine-tuning certain areas. This was particularly important as I tried to build four different lightings, from noon to dusk, using baked lightmaps as my GTX 1070 couldn’t really handle raytracing – yes, real-time filmmaking without raytracing isn’t the same, but it’s doable nonetheless.
Despite my complete lack of knowledge regarding materials, they proved very useful to quickly produce a couple of key effects, like a flowing river with silver highlights, an important part of the poem. It was also interesting to see that I could change material values over time with Sequencer, which I tried out with the main character’s face material, increasing his paleness as the day went by. More importantly, this tutorial allowed me to quickly create a “brush stroke” effect to imitate an impressionist aesthetic with a single post-processing material. I also explored other options such as this Kuwahara shading effect, though the closing deadline made me drop that option.
Unreal Engine Sequencer: a powerful real-time tool
Unreal packs a lot of very useful tools for cinematics, most of which will look familiar to cinematographers and professionals of the movie industry. Camera work is quick, as simple as dragging and dropping a new camera into the scene before moving it around in first person. From a film production perspective, a very appealing aspect is the ability to multiply cameras at will without worrying about lenses, positioning, cranes or any kind of equipment. You can drop a dozen cameras, from fisheye to telephoto, place them high up in the sky or partially underground, in physically inaccessible places, and compare them to find the most interesting shots. I still remember the struggle to find good equipment when I was a film student so the ability to achieve the exact shots that you have in mind is great.
Cinematographers will also feel at home with Sequencer, which roughly works like your average editing software. If you’re familiar with Final Cut or Premiere, the transition will be pretty quick although there’s more to it than stitching camera cuts together. In fact, Sequencer is a very powerful tool that can handle animations, blueprint triggers, material values and more. I used it to create a rough cut of the short, which allowed me to see if everything worked together and to make adjustments on the fly, editing the scene to find a better composition. In the shot above, I was able to move things around to create an echo between the lines of the background mountains and the silhouette of the soldier; the sunset of the valley is also the sunset of his life.
Is real-time filmmaking that big of a deal?
I can’t speak for the whole industry and neither can the industry be considered as a whole. After all, traditional VFX pipelines work well and its pain points might not be solved by real-time tech alone. LED walls as showcased in the Mandalorian production are still very expensive and require a fair bit of post to work out. At the same time, though, LED stages have sprouted all over the world and industry veterans showed genuine excitement at the Real-Time Conference.
What I did find evidently mind-blowing is how accessible and easy to learn the tools are, and how quickly you can reach the stage of final pixel. 3D animation has always been considered a tedious and expensive task, a highly demanding craft that leaves little room for improvisation. But we are seeing the beginning of a new ecosystem in which any rookie can pick up a project and create something that’s, at worst, still very pretty.
The lower barriers to entry is great news for individuals or small teams that want to create something entirely within a real-time engine, either as an indie project, as previz or as a pilot for something bigger. If you add the demand for content from streaming platforms, we might very well see the rise of new studios, new creators, new ideas; at the same time, we can expect projects that are riskier, more original, and more niche than before. Time will tell, but if you want to see change… the good news is: with real-time filmmaking, you can do it yourself.