Get the latest Education e-news
 
  • Postmortem: The Nameless Mod

    [06.11.09]
    - Jonas Wver

  • 3. We had to compromise on quality

    By the time TNM was released, our standards were pretty high. We screened our actors thoroughly for audio quality, we rebuilt entire levels dangerously late in development if there were too many problems with them, and we'd long since stopped implementing any feature if we didn't feel that we could take it all the way. But of course, beggars can't be choosers, and we sometimes had to compromise because we couldn't pay people.

    The two areas where I think this is most obvious in the final product are level design and voice-over. For the voice-over, it was a hard and bitter struggle to secure solid actors for the major characters, but I think we largely made it. In terms of minor characters, the picture looks a little different. Many of them were recorded by ourselves or our girlfriends or by random fans. Even our good actors occasionally had sub-standard recording quality, which becomes obvious far more often than it should.

    In terms of level design, the problems were technical as well as aesthetic. Some of our level designers were quite good at producing pretty architecture, but didn't seem to grasp the design principles that made Deus Ex's missions so exciting. Others had brilliant ideas and a great understanding of design, but their levels looked boxy and uninteresting. Some levels that were otherwise nice and well designed had been constructed entirely off-grid, causing substantial technical issues later on, and it's painfully apparent that we never had an art director or a concept artist, making our levels vary erratically in style and tone. Far too late did we understand the concept of dividing level design tasks amongst a designer and an environmental artist, and even then, we never had the resources to do it in the first place.

    Though this problem was emphasized because we were forced to make do with the generosity of volunteers, I imagine that too short a supply of talented team members who meet your standards of quality is a problem even commercial projects face, and I suspect that if I had a fool-proof solution for it, I could use it as the basis for a whole new article. In the future, we aim to conquer the problem by counting on a small team of only the most skilled and dedicated rather than designing a project that demands the participation of 50 contributors of varying talent and enthusiasm. In short, we'll embrace the indie ethos and stop pretending we're an AAA studio.

    4. We didn't test enough

    We tried though. We gradually brought over twenty testers in to play the game starting several years before we were finished, and set up an online bug tracking system for them to report back to us with. We set up a forum specifically for the testers, and we sent out guide documents to everybody detailing how to install the test build, what sort of bugs they could expect, how to report them, etc. We evaluated our bug database and set up priority fix lists towards the end of the project to make sure all the really problematic bugs were fixed before release. When TNM was released, it had no major bugs that we knew about.

    And yet the first journalist we sent the release candidate out to, before the download link was publicized, found a bug in the kill counting code which would close off an entire map prematurely due to a recent fix between the two last release candidates. We then spent an entire week fixing bugs in some sort of infernal post-crunch, followed by another month of slower, steady fixes. By the second patch, we've fixed around 240 problems, and we've more or less brought TNM into the state we thought it was in when we released.

    There are three things we could've done to improve our quality assurance. First and most importantly, we should have been more careful to test our own work during development. Far too often, features and assets went into the game without any testing. In a game as heavily dependent on emergent gameplay as Deus Ex, it's critically important to apply very thorough testing to everything you implement; if we had tested each of our new additions in combination with as many different factors as we could think of, that would've significantly reduced the amount of combinations that slipped through the cracks.

    Secondly, we should have set up some auto-testing. One of our programmers coded up auto-testing scripts a couple of months before we were done, which was far too late -- we only managed to apply it twice before release. Since our plot had so many branches and variations along the way, being able to fast-forward through certain common branches would've been a great help earlier in the project.

    Finally, we should have managed our testers more strictly. Unfortunately I'm not sure most of our testers would've responded very well to tighter management, but I suspect five dedicated and well managed testers would've been far more useful than a score of testers left mostly to their own devices. We tried to get our testers to commit to particular kinds of playthroughs, hoping we'd get all the branches covered, but it never worked because most people who volunteered to test weren't interested in committing to anything beyond playing the game and writing down the major bugs they found. In summary, our primary mistake was that we skipped the organized alpha testing and went straight to beta testing.

Comments

comments powered by Disqus