TDD, BDD, or any other form or development driven by tests first are very popular approaches in web development or even non-web developments.<p>I'm curious to know if these approaches are also considered a MUST for AAA game development.<p>Is there anyone that has worked with these approaches while developing games or know any big AAA game company that uses test first approaches as something required during the development?
In my experience, no, they don't write tests first, and in fact AAA game companies write almost no automated tests at all.<p>Instead, AAA teams have large staffs of cheap manual testers--high school and college kids who think they're getting paid to play video games. The managers in charge of these QA teams work their manual testers to the bone, encouraging them to work late and do unpaid overtime. These testers inevitably burn out, and but there's always a new batch of kids coming in to replace them.<p>When manual QA is that cheap, it turns the economics of automated testing on its head. Why maintain an automated regression suite when you have humans who will do the testing for you?<p>There's a similar effect on the dev side. A lot of inexperienced developers are willing to take a significant pay cut to work at a AAA game company, just for the dream of working on a AAA game. The devs, too, are notoriously managed as an expendable human resource. "You'd better keep working overtime, or I'll just replace you with another kid who'll work harder than you for less money."<p>As a result, AAA game developers burn out pretty quickly, too, leaving the game industry for other software companies that both pay better <i>and</i> offer a better work/life balance.<p>As a result, you'll find that a lot of AAA teams are relatively inexperienced. Sure, there's a core of long-time devs on every team, technical leads who can point the way, but it's not an environment where you can get solid mentoring on the best practices of software development.<p>Even if one team does pick up TDD, who knows if the same team will even be there in another year or two? Good software practices are regularly "rediscovered" on AAA dev teams, and then forgotten, as the team turns over and their tribal knowledge fades away.
If you want to experience that in the AAA game industry, I work at Rare Ltd (UK, part of Microsoft Studios). We've been doing TDD and Continuous Integration for about 5-6 years now starting with Kinect Sports Rivals. For Sea of Thieves[0] (our current project), we've extended that to Continuous Delivery and it's been just a complete game changer in regards to overtime and crunch. For example, the onboarding exercise is basically a TDD kata that has to use our coding guidelines and build farm so you get to know how we work without having to dive right away into Unreal Engine 4. We use TeamCity for our CI solution.<p>And yes, we are recruiting : <a href="https://www.rare.co.uk/careers" rel="nofollow">https://www.rare.co.uk/careers</a><p>[0]: <a href="https://www.seaofthieves.com" rel="nofollow">https://www.seaofthieves.com</a>
I work on a game called Path of Exile.<p>At our studio we don't do TDD, but we do continuous integration with a lot of asset testing.<p>In the early days, we tried doing unit tests, but honestly, it's very hard in a game. We had little success testing gameplay code.<p>What we have a huge amount of is asset tests.<p>We have 4 times more non-programmers than programmers working on the game, so most of what is created isn't code anyway.<p>Every time anyone commits anything, it goes through our continuous integration pipeline, and any asset that changed, or is dependent on an asset that changed, is loaded and tested. This includes test loading entire levels.<p>Test loading every asset catches most types of failures.<p>We try, where possible, to add specific tests for common mistakes in order to catch things as early as possible.<p>For example, if an animation has "walk" or "run" in the name, then we test it also contains footstep events in the animation somewhere. If a model is used as a monster we test that it has an attachment point called "frame_upper" onto which various effects can be attached. We look for things like bounding box of an object being far away from the co-ordinate origin (a common mistake when making a big maya scene with lots of exported objects in it).<p>We also do a test start of the servers, which includes loading all of their data, and a test start of the client.<p>All of this means that a build that goes through green is unlikely to cause crashes for other people trying to work on the game.<p>We do, of course, also have an excellent QA team.
As a one-time indie game developer who went to work at a 250-person games studio, my answer based on limited experience is "probably not".<p>BDD in particular is tricky, because beyond the level of abstraction of the engine, you don't really know what you're building. How do you express that a thing should be fun, or weapons reloads feel punchy, in a BDD spec? How do you automatically test that?<p>Games code exists purely to elicit reactions in human brains. We don't have the technology yet to examine the desired state change in the target system.<p>Regular apps have the same problem, but with less expensive assets and less iteration. "As a user, I should feel that the login transition is slick" is a spec I can imagine that is UX-related. However, we can probably all agree on what slick is, and failing that you could user-test it to prove that the implementation is acceptable. A game is made of many complex adaptive interacting systems, so every change needs subjective validation.<p>On the subject of complex adaptive systems: many games feature them and so exhibit fundamentally unpredictable emergent behaviour. This can be hard to test for.<p>The most testing I've done in a games project is to figure what it is you're trying to build, then unit test the implementation when you've worked out if it's fun.
Short answer: No. Gamedevs have to keep showing testing zealots the door.<p>Long answer: The problems with the approach are manyfold. There are a few points where testing is automatable, but they don't describe entire game projects well.<p>* What are you testing when you add gameplay? You are testing for a whole set of design concerns across the project, not just a technical specification or quality of service metric. You cannot do it in isolation and get the results you want at the speed you need, because you need tons of feedback to discover what a complex game is currently strong or weak at. The only reasonable way to gain the necessary feedback is to allow prototyping code to drive near-term changes, manually playtest it, and then factor out the most stable parts of the resulting soup where you can. This works against test-first because you don't really know what you're testing. You throw in a feature on a hunch - "maybe this will give the experience we're looking for" - and see how it behaves hands-on. If it works, there are still usually ramifications and elaborations that didn't surface up-front. And once it does work it's hard to break out into a sandbox because the interesting part of the behavior is in the coupling to other systems and features.<p>* Games deal with vast quantities of mutable state strapped onto relatively straightforward data pipelines: the parts that are most testable are engine and toolchain core elements like asset builds and rendering systems. These parts do see unit tests and integration tests, although there isn't total consensus on how much testing is needed or how it should be done. They are the most like business and application software, though, since their development can be more structured around technical goals and quality metrics.<p>* Many bugs are ultimately data bugs. Some designer or artist set up the wrong binding of assets or made a valid yet broken combination of parameters. Games tend to fly apart very suddenly when they operate on bad data. But the code is doing exactly what it should do for that set of inputs - and if it's a failure it's at the level of process and tooling, not the runtime code.
I worked for a AAA game company in the game server department and I would say that with the schedule and how the shop was directed it was nearly impossible to do TDD. I can't speak for the engine side, but my guess is that it was not possible over there either.<p>I bet like anything through, if you go to shops that have strong direction, can work with a small team and over deliver, those standards paradigms are used.<p>Check out the tests in the Quake3 Source Code. It is a shining example for sure.<p><a href="https://github.com/id-Software/Quake-III-Arena" rel="nofollow">https://github.com/id-Software/Quake-III-Arena</a>
My biggest observation with AAA games is that the coding is actually a fairly small part of the work. Most of it is one huge asset pipeline. And I don't think we have particularly good tech for testing what "looks right". Maybe computer vision will get that good eventually, but it's not even a thing right now. The same applies to the gross properties of renderers.<p>Even in big-spec-up-front triple-A games, there's a lot of exploratory work where you honestly don't know what you want. BDD and TDD don't help there either, rapid prototyping does. (TDD presupposes you know what you want, BDD presupposes someone does and you need to find and talk to them.)<p>None of which is to say that there aren't many places where test friendly design and regression tests wouldn't help. But the surface isn't the same as the average business app.
Hi, this article [1] isn't quite what you're after but you might be interested in it anyway.<p>[1] <a href="https://engineering.riotgames.com/news/automated-testing-league-legends" rel="nofollow">https://engineering.riotgames.com/news/automated-testing-lea...</a>
Croteam applied lots of automated testing by bots in The Talos Principle to ensure that changes in level design cannot lead to a situation where the game cannot be completed anymore:<p>> <a href="https://venturebeat.com/2014/11/04/the-talos-principle-underwent-15000-hours-of-playtesting-but-not-by-humans-interview/" rel="nofollow">https://venturebeat.com/2014/11/04/the-talos-principle-under...</a><p>> <a href="http://www.gdcvault.com/play/1022784/Fast-Iteration-Tools-in-the" rel="nofollow">http://www.gdcvault.com/play/1022784/Fast-Iteration-Tools-in...</a>
Having never worked at a gaming company but having a friend who was the head of testing at one very recognizable shop, the answer was definitely no, at least not there. Its definitely a big enough industry that I'm sure some does though.<p>Also, last time I wrote any 3D code, it at least appeared to be incredibly hard to test that some visual thing was just so. Of course there are still things like business logic that surely could be tested, but it's just not going to be as straight forward as having some code generate some HTML string and then asserting things against it. Those last-mile things you might want to test are always a challenge IMHO.<p>I learned a lot of my coding chops at a Rails shop that was very in to testing, so that's definitely my perspective, YMMV.
In my experience no. Testing was not even considered. I worked on mobile games 5 years ago so things may have changed.<p>Most of the time we had strict timelines with feature milestones. We were building mobile apps of licenced titles.<p>The lack of TDD was replaced with full time QA, sometimes multiple on the one game.
Factorio is a good example of a test driven developed game:<p><a href="https://www.factorio.com/blog/post/fff-186" rel="nofollow">https://www.factorio.com/blog/post/fff-186</a>
I'm not in the game industry, but from what little I know, this is probably going to be a tricky question to answer.<p>Reason being, a high proportion of AAA game development is done using off-the-shelf game engines. Of course it's still necessary to write code for the game logic and features, but because of the stable base this code is written against, it might not be seen as necessary to follow a strict TDD/BDD approach.
A colleauge working at a AAA Soccer game gave some insight about that a few years back.<p>Their automated testing was mostly bruteforce stuff like,<p>* Placing a character at penalty spot and making it shoot with every possible angle and strength, making sure the ball didn't launch to space.<p>* They would leave a computer vs. computer game running for hours / days to make sure no kind of random crash would happen.
I worked on a AAA game. I controlled the networking code (web services really) down and the entire server infra for our project. All of that was full TDD but it was also all greenfield by me who had done heavy TDD before and working with an external team who was all TDD. I even wrote integration tests from a stub client that ran on the playstation as part of the build to hit my server and a stub server to do wire testing for the client as part of my builds. Also had a full load regression test as part of my build that ran every checkin on a smaller slower cluster.<p>The main game had no tests... except that you need to realize that we had 480GB of asset compiler integration tests and 32GB of game engine integration tests.<p>As a heavy TDD person it took me a long time to realize that this can be very sufficient though it's a longer turn around loop. The test coverage just from being able to boot the game up is quite large. This is something that many non-game type programs have real issues with, they can't easily just "run all the user data". I know we had one project where we did a lot of TDD but it wasn't actually as good as just running the entire input set every build which I eventually switched us to. This was a non-game data processing system. Doing this + unit testing gave us great coverage and few bugs.<p>Generally major bugs were found within 15 minutes by another engineer, eg the next person to sync, build or run the engine or art compiler.<p>I do think we could have unit tested the core collections libraries in the engine to great effect and faster turn around but the build was not really segregated in a way that we would take advantage of this. It was just a monolithic build; the collections objects were heavily header included so there really would have been a build step to built them into their own test exe, run them then build the game. So I'm not sure how much benefit we would have gotten. I think in 4 years I saw one collections issue that lasted past just launching the engine and not having it crash.
I am not working directly in gamedev but I have some insights.<p>I've seen growing interest in automated testing from mobile apps.<p>Basically guys were writing regression testing with framework like Selenium (UI testing) that goes throughout most common scenarios.
(I think that was described by huuugegames.com on some presentation)<p>The main idea was to let QA focus on "exploratory" tests. (Testing rules of the game, design decisions or performance)<p>TDD & BDD can be used gamedev but not in places you will think of first.<p>I will simplify it vastly: programmers make tools for designers to use.<p>Tools are made by programmers - tools can be done with TDD and/or BDD.
Games are made by designer - they use tools to make rules, systems etc.<p>Therefore gameplay is make mostly out of data (designer's parameters set in engine/tools) - you can't test your data with TDD/BDD.
Game development projects are usually focused on tech/data/network/engine, asset creation/pipeline, behaviors and gameplay.<p>In the tech/data/network/engine areas there may be TDD/BDD if the engine is to run multiple titles beyond the current project, if not the budget may not even allow for time. Possibly other areas may have tests in the behavior/gameplay areas: components, AI etc, this depends on the current dev culture usually.<p>Really the main thing in game development is <i>shipping</i> and building <i>fun</i>, if tests help you do that faster/better then you do them. I think most devs developing anything that is not game specific but tech/data/network/engine related likes to have tests to rely on, time permitting.<p>Most studios I have worked with that do some TDD/BDD/automated testing it usually happens on the "tech" or core/engine teams rather than the "production" or "live" teams. The tech teams are working on engine/network/data while the live/production teams are using the engine/tools/scripts to make assets and gameplay. The tech team isn't as locked to game launch dates/crunches and has more time to be thorough because most of their code will last across multiple titles.<p>For the most part, in the asset creation/pipeline, behaviors and gameplay area, more of that is visual and QA based. There are testing companies and departments setup just for the gameplay/feature testing. So much of game development is going for fun and a good mechanic that actually playing it and testing it live is more effective than automated tests in some areas. Although there are some automated test tools in Unity[1], Unreal[2] and custom engines maybe for very rote areas of gameplay (physics engine, collectables, asset loading, very common unchanging actions etc).<p>Many times on the production/live side of game development, gameplay and behaviors change so much that any automated testing becomes a lag on dev/iteration time, or aren't updated and stagnate as the changes happen so quickly day to day. This is especially true in iterative development during prototyping or pre/post-production, since it is mostly visual the benefits are outweighed by the speed required to ship.<p>[1] <a href="https://unity3d.com/unity/qa/test-tools" rel="nofollow">https://unity3d.com/unity/qa/test-tools</a><p>[2] <a href="https://docs.unrealengine.com/latest/INT/Programming/Automation/" rel="nofollow">https://docs.unrealengine.com/latest/INT/Programming/Automat...</a>