“Utilizing well thought out DevOps practices can take a team of any size from chaos and uncertainty to a smooth running machine that preserves one of your most valuable assets - time. Less problem solving and more time spent realizing a creative vision.”
Marcel Samek, Wevr | WVS CTO
Introduction
Talent, passion, drive, and excitement. Most creative teams are fueled by a desire to make something amazing. The early days of a project are when the flames of artistry burn the brightest. But eventually deadlines, mounting anxiety, and last-minute code scrambles can create a chaotic environment all too common in the game building process, whether you are using Unity, Unreal or other game engine. This kind of relentless churning can exact a serious toll on teams and individual members. Tempers flare, once supportive attitudes devolve into blame and resentment. And then there is the very real possibility that you won’t make the deadline and it was all for nothing. What began with excitement and camaraderie is now in danger of merely limping across the finish line, the end result underwhelming if not profoundly flawed. Worse yet, the greatest threat is collapse - the project isn't completed and never released!
Or maybe a team seems to have it all figured out: they have a version control system that they like, their asset management is on-point, and they’re testing and debugging their builds efficiently. These projects are some of the best to work on, and for good reason. But every project thrives on iterations, and being able to iterate even a little more consistently can lead to a noticeably better end-result, even for a well-organized team.
Running a small indie team is not too dissimilar from a project at a major studio. In both scenarios a team needs to focus their talents on game mechanics and perfecting the product and not expend precious time and energy worrying about mishaps in the production pipeline. Making games and interactive projects is a complex convergence of design, art, story and engineering. And code revisions have the potential to create a “domino effect'' that can corrupt implemented systems or scenarios, which then have to also be revised and fixed. Necessary art updates can also slow performance and create unforeseen problems. It is extremely difficult to predict where these problems will arise, but, once they do, they need to be located and fixed which burns time and resources.
This is where the concept of Devops can provide a framework that allows a team to create and test their assets with minimal effort and time. As Marcel Samek, Wevr CTO states: Utilizing well thought out DevOps practices can take a team of any size from chaos and uncertainty to a smooth running machine that preserves one of your most valuable assets - time. Less problem solving and more time spent realizing a creative vision”. Whether you are using Unity, Unreal, Godot or another game engine, every interactive game thrives on iterations - full turns of the crank where changes can be integrated and evaluated in fully playable builds. Even if you already have a smoothly running process, you will benefit from being able to do those turns of the crank a bit more often. That’s where quality comes from.... So even if you already do everything in this document, you probably already know that doing those things a little bit more efficiently is a pathway to noticeably improved results.
Setting up your team
Teams are more than a developer with an idea. They are designers, producers, artists, animators, programmers, engineers and sound designers all working together to create something unique and bring it across the finish line. Each member of a team brings their own talents, opinions, and methods to the project. Organization is the name of the game when it comes to working as a team.
Whether you’re a 3 person team or 300, setting up your team right from the get-go is vital. More often than not, teams today work remotely distributed across time zones. Making sure all the developers on the team are set up with the right version of the game engine, and content creation tools, as well as getting the right level of access to the repository or shared storage is critical, but also time consuming and often requiring an IT person or department, or you’re taking your engineer’s time. Even a basic step such as setting up a secure connection from a local computer to the team’s repository to sync files securely, such as setting up SSH keys requires technical steps that may be challenging for non technical artist team members. Moreover, teams often hire contractors that join a team for a short period of time, so it’s important that they are productive immediately from day 1. A well managed DevOps solution can provide a cloud based service with a web interface to make team setup much easier and less error prone.
Establishing the right production procedures at the start of a project is vital, otherwise error-prone practices spread across the team and result in compounding problems as the team grows and the project becomes more complex over time. Basic things such as naming conventions for files and directory structure can go a long way in minimizing frustration for the team. It’s all too often that artists are searching their local disk for files for older versions with intractable names like: Level_Geo_final_1_v2. etc… Even if files are kept on shared storage like Dropbox, it is time consuming to track source assets and the dependencies between files created, exported, and imported in engine. With a DevOps solution, all team members can be set up to work in the same well-managed repository with the optimal balance of files stored in the cloud and files synced locally.
Teams, whether a small or a large studio, often work on multiple projects at the same time, sometimes for different platforms (ex. one project is a PC game, while another is a prototype for a mobile game), and sometimes using different engines with one project on Unity while another on Unreal. This is particularly true of teams that do work-for-hire projects for multiple clients that have different project requirements. Setting up your team with a well organized and easy to use project management structure from the get-go increases productivity for all team members and makes it a lot easier to work with external stakeholders. A unified DevOps platform that enables a team to set up multiple projects, with different target platforms, and also supporting different engines enables a unified development and production methodology across different projects, providing huge efficiencies for a team.
“A well-organized team can avoid using disparate storage for code and data and leverage a unified repository across the project. A shared repository enables consistent version control and efficient iteration, and unlocks the ability for automated workflows. Moreover, working on multiple projects at the same time, for different platforms, and sometimes using different engines, calls for a unified development methodology and production pipeline. Without solid DevOps teams lose valuable time and resources that could be focused on making a better, tighter product.“
Neville Spiteri, WEVR co-founder and CEO
Easier iteration with version control
Developing a game from concept to ship is all about iteration and managing changes as the project develops. Making changes is the daily practice of developing games, for all team members involved. Whether you’re a designer making design script changes, or an engineer changing the feel of a player control mechanic, or an artist changing the level of detail of an environment, the art of game making is the art of managing change. And your best friend when managing change is a unified version control system and DevOps solution.
Here is an example of when version control can be important: a new mechanic is being added to a game, and an engineer is coding it in their favorite coding environment using GitHub. Meanwhile, a 3D artist is creating different ways to model the mechanic, and they’re iterating on those assets using Blender, Maya or other content creation tools, and storing files in Dropbox. But maybe they add too many polygons and the frame rate suffers, so they have to change their model. Or the engineer rewrites their code and breaks something linked directly to the rig. Different team members might iterate separately, and that often means new versions of files are floating around the team’s mix of storage systems. The collaboration and the sharing of these files in real time becomes extremely chaotic, often just accepted as part of the development process, but very often negatively impacts the build, and the whole team. (i.e. code and 3D data are mismatched and gameplay bugs are introduced, worse still, the build breaks).
Artists often don’t use version control because it’s too hard. The benefits of version control don’t seem to outweigh the costs. When a team loses significant time hunting down misplaced assets, it likely has to do with version control issues. An artist is working on an asset one day, revises it the next, and after an art review, the team decides to go with the previous version the artist did the day before. This simple change, reverting to a previous version, is much more error prone without version control. Unity and Unreal builds may compile but parts of the project can break. Seemingly out of nowhere a frame rate drops or there’s an unintended lighting change or an environment element goes black. Everything worked perfectly the previous day but now it’s a mess.
Artists iterate on source assets in programs like Blender or Maya, and the assets (geometry, animation, etc.) are exported in a game-engine acceptable format such as FBX or USD. Meanwhile, engineers are iterating on game mechanics and game systems code. And designers are also making changes in the editor. Team members are working on different parts of the game and working through iterations and versions as the project evolves. Managing iterations from all team members quickly becomes unwieldy. Ben Vance, Game Designer at Wevr shares, “There's a lot of moving parts and lots of unforeseen things that happen. You can write code that you think is standalone and works by itself. That's great. And then when you merge with the rest of the code base and assets, something starts happening that you didn’t anticipate. And that's just the nature of game development. You're not gonna avoid that completely. So you want to be able to diagnose the problem as quickly as possible. You want to have a system in place that can minimize the damage.”
This type of workflow can be handled perfectly by version control software, but most artists end up creating multiple versions of files and handle it manually instead. This confuses the whole team and makes it harder to tell what asset is being used in the newest build. And the further back it happened, the harder it is to find and fix. Which is why a strong version control strategy can help a team work together without breaking things. This is especially critical when teams work remotely.
Understandably the biggest roadblock for implementing many version control solutions is getting team members to adapt to a new and unfamiliar system. This can be more difficult when a team is scaling to a larger group. It’s best to set this up early in the project when there's just a couple people involved, even though it doesn't seem to be worth while, because if you wait until the project is larger and more complex, it will take too much time and significantly interrupt the workflow.
"Those are the sorts of pitfalls you see when things aren't straightened out at the beginning or thought through or planned. It can create a lot of friction and slow the project down, and the cost of reorganization is usually so significant that you avoid it completely. And then, what you have to do is sort of make a cheat sheet to figure this stuff out and the mess will slow you down and hinder production and testing for the rest of the project."
Marcel Samek, Wevr
An adequate version control system should provide options for developers but also be accessible for less-technical team members. While programmers tend to be adept at version control branching techniques, many artists simply aren’t, even though their work is just as crucial to the end result as the code. An intuitive and well mapped out, graphic UI can make complicated merging easier for artists as well as designers and less technical team members. It also allows you to see when an asset was modified and on which branches it is used in case problems arise.
“There's a lot of moving parts and lots of unforeseen things that happen. You can write code that you think is standalone and works by itself. That's great. And then when you merge with the rest of the code base and assets, something starts happening that you didn’t anticipate. And that's just the nature of game development.”
Ben Vance, Independent Game Designer, Wevr
Code and assets originate from multiple sources, from in-house developers, artists, external contractors, and open source repositories. This is where version control comes into play, enabling all assets and code to be managed in one unified repository. Changes and history are automatically tracked transparently across the team. Team members move on to the next task while what came before is assumed to still work. As deadlines approach, you might not have time to actually play through the game. The game is longer and longer, there's more minutes or hours of play. Nobody has the time to play through that on a daily or even weekly basis. And so you reach the point where when people do play through, they find that things that were working are now broken. This is why it’s so beneficial to implement a version control system at the start of the project. Compared to simply storing all files in a cloud folder like Dropbox, there are immense benefits to employing version control across the whole team, for both engineers and non-technical artists. This true no matter what game engine you’re using, Unity, Unreal or other.
Besides creating a common library for project resources, it can also enhance the collaborative relationship between artists and engineers, freeing up the team so they can code, animate, design, or do what they do best.
Utilizing Automation
Automated Builds
Automated builds are a way of validating everyone's work without relying on one person or machine, as well as beginning to streamline the feedback process (and more feedback makes a better game). It may seem counterintuitive to set up automated builds early on with even the smallest of teams. It’s also really important to establish the practice of doing full builds, versus just running in the editor given the all too often heard statement “What, the build is broken? But it runs fine in Editor!”. A good build automation strategy allows a team to organize different kinds of builds for targeted reviews and feedback. This helps avoid missing deadlines throughout production, and avoids devastating hard-to-debug crashes at the end of production. More frequent iterations, builds and build reviews is one of the most important factors to improve the quality of the game.
But game-engine builds (Unit, Unreal, etc.) are time consuming and sometimes a pain to configure, often requiring asking the lead engineer for help. Moreover, builds can take several hours and then break at the very end. This can be not only frustrating, but costly. Builds are typically managed by a lead engineer who ensures that all the code and assets with all the dependencies and necessary components are set up correctly. Once set up right, DevOps practices make it really easy to automatically generate different flavors of builds - for example, a test “gym” build that’s just for animation reviews, or a level build to test gameplay in one specific part of the game, or a specific demo build based on a special demo branch, etc.
“Test gym builds are a great way to easily try out gameplay ideas. Teams can benefit greatly from a strategy of using test gyms where an individual developer or sub-team can implement an idea and easily build and test it without stepping on the main development process. A DevOps solution with test gym build automation is a game changer for dev teams and their stakeholders”.
Eyal Erez, Independent Game Engineer, Wevr
Being able to easily generate different flavors of builds makes it easier to work, playtest and get timely feedback. It also makes it easier if your team decides to change course. To do this means constantly integrating changes and creating new builds. Manually doing so can be error prone and complicated, and it interrupts more important tasks and eats up precious time. The solution is a DevOps platform supporting branch management and build trigger controls. As a result, your team will be able to identify and fix problems right as they happen and before they create larger and more time consuming problems. If there is a need to roll back changes to the “last known good build”, it is readily identifiable and available. A DevOps approach allows your team to ensure they are venturing ahead without risking serious derailment.
Other automations - Renders and Testing
In addition to automating builds, there are other workflow automations that can be enabled with DevOps which can be a major accelerator for teams. Though less commonly known in game development, automating renders and asset validation and testing can speed up the iteration cycle and reduce resource costs.
A picture is worth a thousand words. There are many situations in which a team can benefit from quickly sharing and reviewing images or video clips of parts of the game, rather than playing through and reviewing a build. For example, environment artists may want to show 360 renders of interior environments for feedback from their art director, or a lighting artist may want to kick off a series of renders with different shadow settings and do side by side comparisons. The right DevOps approach can enable these workflows. DevOps provides the ability to automate renders directly out of the game engine.
Testing code and assets and making sure you’re not “breaking the build” can be a time consuming and laborious process. DevOps can be used to automatically test code and assets before full builds are generated. One best practice to avoid issues down the road, is to commit shorter code fragments more frequently. With a DevOps solution you could automate unit tests as well as test builds that verify that your incremental code changes work as expected and don’t negatively impact the rest of the team. Another big opportunity to increase iterations on builds and reviewing progress more frequently is to validate art assets to ensure they don’t break the build before you check them into the mainline. For example, you customize an import step before a build is kicked off, automate exports from source content creation tools and imports into the engine project file, validate assets in formats like FBX and USD, etc. Dev teams that haven’t been exposed to DevOps solutions may not even think about implementing such measures because they are costly to develop from scratch. Leveraging a robust DevOps solution provides a major leg up.
“Velocity is key. With cloud automation you can focus on your game project and not have to develop and maintain the pipeline infrastructure required to deliver tests, builds and renders. Working asynchronously, not tying up your local workstation, saves the team time and increases iterations. Devs and artists can review and get feedback based on team-wide builds, and feel like the whole team is working in the same studio even though everyone is remote across time zones.”
Anthony Batt, Wevr CoFounder/EVP, Director Harry Potter VR
And one last note on utilizing automation in DevOps… the cloud is your friend. Cloud platforms provide turnkey compute and storage services in the cloud. DevOps really shines when leveraging cloud computing and running all your build, render and testing automations in the cloud, keeping your local development workstations free for your work. In a world where dev teams are increasingly working remotely, distributed around the globe on-prem servers and infrastructure are less viable. Remote is the new normal. With powerful Devops automations in the cloud you can get the whole team working, seeing, playing and reviewing the same tests, renders and builds daily across time zones.
Summary
One of the key goals in improving game development is to get your team setup and iterating from idea to playable builds as quickly and efficiently as possible. Play testing build iterations frequently using Unit, Unreal, Godot (or other game engines) is the best way for a team to optimize their time and focus on the improvements that matter the most to make the product better. This in turns improves team collaboration in a compounding loop of increasingly better development. Leveraging devops can help you achieve just that.
In this paper we covered some of the more immediate benefits of leveraging DevOps to improve game development, starting with 1) Setting up your team, to 2) Easier iteration with version control and 3) Utilizing automation.
- Wevr Virtual studio is a cloud platform purpose-designed for creators and teams developing games and real-time 3D applications using game engine.
- We look forward to hearing from the community and sharing more learnings as we go. Please feel free to reach out to us at hello@wvs.io for any inquiries.
Glossary of terms
- Build Automation - The process of automating the creation of software builds.
- Cloud computing - Using 3rd party server infrastructure services for storage and compute, such as Amazon Web Services.
- DevOps - A set of practices meant to increase a team’s efficiency in developing and delivering a project.
- Merge conflicts - When two different, conflicting changes are made to the same code.
- Repository - A central location where files can be stored and retrieved.
- Project artifact - Any software development or production asset used in a game development project, usually code, or binary files and their metadata.
- Version Control - The process of tracking and managing changes to software, documents, art assets, etc.
- Branching - The duplicating of something that is being version controlled, performed by a developer who wants to work on some asset or code fragment, etc. without modifying the original copy.
