After working on multitude of projects, some that shipped and some that didn’t, I realized that: “Starting a game is easy, Making a game is hard, Shipping a game is even harder”. Often, the issue of finishing a project is to really take the features to the finish line, the content, and more important, the product as a whole. What makes that part hard is that you can always make things better, you can always tweak and improve, and the hard thing is to say “Stop! this is enough, lets ship this thing!”. Often, you have people who are there do say this “Stop!”. But also, when you get to the end, things are just not good enough and the team has to crunch hard to get over the hump of good enough, and ship the product.

In 2008, when we were finishing Mercenaries 2, we were crunching like crazy to get things finished and ship the game. The big problem we had is that we had to crunch to get the basics finished. Those things were: getting our streaming system working properly off disk, get our levels loading working properly, get our game in memory, in framerate, getting our Ps3 up to par etc… Pretty much the things you need to have to ship a game.

Starting the new project, we were talking about doing things better. Some of those things were team wide processes: having a more solid pre-production, a design that has been prototyped and proved out to work before going into full production. But some of us were thinking of the technical issues, and how we could resolve them to make the development smoother. Unfortunately, we quickly went back into doing similar mistakes and I decided to leave. But this made me think of the situation, and trying to materialize ideas on how to make things better. I looked back at previous projects and realized that its very hard to not crunch at the end, because you can’t predict the issues you’re gonna have. But I believe there are a few things you can do to your project, specially when starting something brand new, with tech from scratch (or almost scratch) to get your project in a better shape at the end and not suffer through crunch, or do crunch to deliver something of higher quality instead of just something shippable. But what are those things ?

I will come to that list, but thinking of those things, you will notice that not only they will help you with the last phase of your project, but they usually help throughout the development. When I tried to make that list, I tried to think of the things that you need to have in your game, but doesn’t make it fun, or pretty, and doesn’t affect the quality of your game – actually some of things do, we’ll come to it – but are things you just can’t avoid to do. Most of the things I will list apply to both consoles and PC. PC development can be a little different due to its variety of configuration, and its also a platform I don’t know very well, so you’ll have to see if this applies to you.

1. Loading and Media.

It is obvious that when you make a game, you need to pretty much load files, extract the data and “run” this data. You usually have something that loads, and runs. But something I’ve seen a few times is the lack of a good mechanism to load files, unload files and go from levels to levels. At the start of the project, you just get one level that loads. You can probably have multiple levels, but people just load the level they are working on and just work like that. You get to the end of the project, and boom, now you have to put your game together, and you can’t go from levels to levels without crashing.

On top of this, you need to load your files from Media. Either Disk (DVD or Blu-Ray) or Download-able package. This is not something to take lightly. Getting your game run off media means multiple things. Getting the game to boot, getting your project properly setup to burn the files you need : Not forget system files, getting the directory structure setup which can mean rearrange your data pipeline, automate the system so that you can burn disks on demand, and more important, and often the part that is a surprise, and takes the longest to fix, make sure the load times are decent.

2. Memory and performance

Something that all of us game developers always fight is Memory and Performance:

About memory, to ease the end of your project, deliver every milestone always in consumer memory. I’ve seen teams that want to show their best by using all the devkit’s memory, and show a world that they wouldn’t be able to build and ship in consumer memory. It important though to not show “smoke and mirrors”.

You could say something similar about Performance. In fact, the philosophy between Performance and Memory is rather the same. You need to make sure your performance is always within a threshold of your target performance. You shouldn’t add lots of features without optimizing what you already have to keep your game in a controllable state. You can quickly get yourself deep into a hole that is very hard to get out of.

So for both performance and memory, its important to be reminded where you are. So always have printed on screen the framerate, and the current memory usage. And get your tracking tools early in development. I would say this is one important thing to have in your technology. Lots of tools to track down memory usage and performance.

For performance, good on-screen profiling tools (with nice colored bars) are essential. Try to get this to show for all your processors (GPU, CPU and of you’re on Ps3, SPUs), and threads. Show the current timing for your different systems early, have the tool setup so tracking performance is easy. Get your profiling configuration early in development.

For memory, get your tools to output the memory usage for each of your assets in your levels through a good report file (a CSV file can be easy to read in Excel). An on-target memory tracker is also essential to track down dynamic memory allocation. I saw different way to approach this, but my prefered approach is to record a small callstack with each allocation and print it out when requested. Also, when it comes to memory, make sure you never have memory leaks. A good way to test this is to go from one level to another, and make sure all the memory for the previous level has been properly released.

3. Build Machines

For any kind of medium to large software project, it is quite important to separate the everyday workspace to the actual build of your software. I have seen multiple times in my career projects where the lead, producer just sync code and data on their local machine, build, do some changes, and generate builds from their local machine. Those builds become THE build that they deploy manually to the different users on the team, and to the publisher when there is one. This can cause a desync between the members of the team which causes errors, weird issues going on, and worse, an official build of the game that have some weird debug code in there that someone forgot to remove before making their build. Also, having a human making builds is pretty tedious and error prone specially on milestone weeks as it is not rare that the team and individual will work late, and can get pretty tired.

So you really want to have a machine, completely virgin to any external software, local code or data. Its only role is to sync to the latest from the source control database, and build code, then data and make THE build. Doing so can verify that the database is consistent, for example, that someone didn’t forget to check something in, and it also helps with deploying a nice clean build from the current state of whats checked in. This process can be automated to make things a lot easier, and if your courageous you can push the process to have scripts that will collect all the built files and make isos that can be directly burnt onto a disk, or packaged for online delivery. And to go further, with this, you can have your build machine generate all the different skus (for example, europe and us skus). Having all this process very separate from the day to day team work. Depending on your budget, you have from one to multiple build machines. You can have one per platform, one for code, one for data etc… This allows to build your game faster and separate code and data teams. To finish on this part, if you can make the usage of your build machine really easy, you can make your producer, project lead or whatever become the person to trigger the build machine to make the builds that needs to be deployed.

4. The shell

A few projects that I worked on didn’t have a shell until very late. Mercenaries 2 was one of them and it really bit us in the butt. A shell, even minimalistic is essential during the course of your project. Often the shell drives the flow of your game, but also of your data. On top of that, you will be forced to test early the loading/unloading of levels since once you have a shell, you should be able to go from the shell to your game, and back. And finally, once you have a shell, you can expose all the levels available to the user that they can load/run. Then while your project gets bigger and bigger, you can add more options to your shell. On Destroy All Humans!, we had a shell pretty much from the start. We were able to ship every milestone with it, and use it as a Hub for all our current levels, whether they were production levels, or prototype levels. And we were able to develop the flow of our game using the shell. Adding the options to load/save a game, graphics/audio options etc… became very easy. Also adding and removing a level was dead simple, and because this happened often – deciding to cut levels, or add new ones. It was important for us to keep the project as a whole updated, and everyone knew what was going on. Load the shell, and you now know the levels of the game.

5. Platforms

If you’re doing a cross-platform game, try to sort out your architecture and pipeline as soon as possible. Do not do that as an afterthought. I would even say that you should have all platforms progress at the same time. It is better to do a little less features every milestone, but have them work on all platforms. You will quickly notice that if you have structure your code properly, you will get to make it work on all platforms, with proper performance and memory constraints rather easily. But if you try to focus on one platform during your development, and “port” to the other platforms at the end of your project. You will end up sacrificing quality and worse, the programmers will spend precious time porting the code to the platforms instead of polishing/optimizing/debugging the current state of the game. This is a very easy trap to fall into. The reason is that usually focusing on only one platform, specially at the start of your project makes your development much easier. The producer will usually pick the easiest platform to develop for, and will push for more and more features. But this is very counter productive from a technical point of view. It often, to not say always, forces you to make technical decisions that will hurt you later in the project. The platform situation also goes hand-in-hand with memory and performance. Keeping those in control at all time on all platforms will be very beneficial throughout the project.

6. Crashes and Bugs

This will sound obvious, but you’d be surprised the number of teams that say “I know” but don’t really do this. To avoid the big crunch in the end, you need to have an eye on your bugs and crashes throughout your project. One good way to do that is make sure you always prioritize bugs and crashes over features, and keep your bug count under a threshold. If the number of bugs and/or crashes gets too high, its time to stop everyone from working and have a bug hunt/bug fix day or even week. You can apply this rule to memory and performance as well. It is important to keep those under control.

Conclusion

Most of those things should also help your project throughout development. For example, if you’re working with a publisher, you will most likely have milestones and deliverable. Wouldn’t it just be easier if you could just send a disk of the version of game that the publisher could just boot on any TestKit in their office and just experience the current deliverable like a normal user from the outside world. Believe me, this kind of situation will always win you points in front of a publisher. It makes everyone’s life much easier, saves you lots of hassle, which translates into lowering the cost.

The funny thing about all this is that anyone with a bit of experience will read this and think: “This is so obvious, we should all do this !”, but how many of you, even with experience have been on a project where you get to the end of the project, and never booted the game running of disk, or starts making the game fit in consumer memory when getting to Alpha…

The truth is that often those problems emerge from bad prioritization at the production level. But while it sounds harmless at the beginning of the project, it affects your technical team badly at the end. And your precious programmers who could spend more time polishing the game, spend their time just getting the game to ship.