Playtesting is Sovereign, Part 1

By Lewis Pulsipher [08.10.10]

 Whether you're working on video games or you're designing tabletop games as a way of learning game design, constant playtesting to improve the game is the major path to success.

The biggest factor in the playability, the successful gameplay, of a game is not the quality of the ideas, nor the strength of conception, nor the marketing skill, nor the skill of artists or programmers. It's the quality and quantity of playtesting and the resulting improvements in the game. In the end, if enough people in your target market play the game and enjoy it, then it's "good"; if they don't, then it isn't! While a poor game may sell well thanks to marketing and other factors, ideally you want to create a good game to give you the best overall chance of success.

The number of inexperienced people who think they've successfully designed a game, yet haven't playtested it at all, is remarkable. Playtesting is playing the game to find out how it can be improved, and then improving it to try again. The process is both incremental-a little at a time-and iterative, again and again. The playtesting stage is the start of successful design, not the end.

Let's clarify something. I am talking about playtesting to improve gameplay, not testing to squash programming bugs. Some people call the former "fun testing" and the latter "bug testing". Bug testing is often what video game makers mean when they talk about "testing", and this testing takes place late in the development cycle, when the gameplay and appearance are set in stone (because it's too late to make big changes). This testing (misleadingly called "Quality Assurance") is aimed at making sure the game works the way it is supposed to, not at whether the way it's supposed to work is good or not.

"Bug testing" essentially does not exist in tabletop games, although it is important (and often forgotten) to test the production version of a game, as converting the prototype into the published version can introduce its own set of problems. (A small example: the boxes on the Population Track on the 2006 Britannia board are inconveniently small; this new version of the board evidently was not actually tested. They are larger on the 2008 printing.)

The "natural" way to design a game, as used in tabletop games, was long unused in the video game industry, but is coming back into custom. A playable prototype is produced as soon as possible. It is played, revised, played, revised, played, revised, seemingly forever, until a stable "good game" has been produced.

The lead designer/programmer of Unreal Tournament III, Steve Polge, described how important it was for Epic to be able to playtest within a month of starting, and constantly thereafter. (See the video with the UT3 Ultimate Edition.) Epic's Gears of War 2 was playtested something like forty thousand hours (that's the equivalent of 20 people working a normal schedule for an entire year). For Civilization IV Firaxis used a team of only 7 or 8 people to make a working prototype as early as possible, then played it constantly throughout the production process. They then added many more to the team for production artwork and polishing. (See their 2006 GDC presentation, available in video form with the Civilization IV Game of the Year edition).

The "wannabe" designer's assumption that the first prototype will be just fine as it is, before it's even played, is a product of both ignorance and the tendency to oversimplify the role of game design. If you think that a good idea makes a game, you might be excused for thinking the prototype will be "just fine". In fact, the idea is just a starting point.

People in other fields of art and entertainment revise their work often, even if they don't test it on others. Beethoven had notebooks filled with musical ideas and revisions of his work. He actually completed four versions of the overture for his sole opera -- Leonora 1, Leonora 2, Leonora 3, and Fidelio, the one finally used. You can hear the improvements when you listen. I prefer Leonora 3, but this mini-symphony was too monumental to be used as an overture to an opera, so the composer tried another tack for Fidelio. He matched the work to his goals and requirements, something every game designer needs to do, especially when employed to make a particular game.

Ideally, a game designer has the time to do this with every game, but this example is extreme even for Beethoven, and would be extreme for a game designer, to finish three versions of the same general work before settling on a fourth. The difference from games is that Beethoven was producing a passive kind of art, something to be presented to the audience when done rather than to be tested with an audience. Because games are interactive, the only way to modify them sensibly is to ask the "users" what they think and feel.

When you design a game, you try to see in your "mind's eye" how the game is going to work, but until you play it, you simply cannot know what is going to work and what is not. The first few times you play, many things will change (provided, of course, that you're willing to make changes, which is a major characteristic of a game designer).

Granted more experienced designers can foresee more weaknesses and eliminate them before reaching the prototype stage. But every designer, regardless of experience, is likely to change the game significantly when it begins to be played.

What you absolutely cannot do is convince yourself that whatever you like is what other people will like, that the way you play is the way other people will play. You are not your audience, you are not typical (or you wouldn't be designing games), you are too close to the game: you cannot rely on yourself.

There is no substitute for extensive playtesting. Your initial prototype is almost certainly going to stink. Get used to it.


Playtesting stages and the 80/20 rule

"It's not so important who starts the game but who finishes it." - John Wooden, 10-time NCAA Basketball Champion

"Nobody is as smart as everybody." - Kevin Kelly

The playtesting and modification process may make as little as a 20 percent difference to the quality of the final game, but that 20 percent is the difference between an indifferent or downright awful game and a good (or better) game. This final 20 percent improvement takes 80 percent of the time, but it's time well spent, necessarily spent.

Playtesting is time-consuming, tweaking rules or programming is time-consuming. Think of all the mediocre or really bad video games you've played: probably the majority of those games failed because of inadequate playtesting, not because there was something wrong with the idea or the initial execution. Companies such as Blizzard, Epic, and Valve, which can playtest as long as necessary, and which aren't afraid to say "this game is a bust" and cancel it, are the ones that consistently produce outstanding video games.

There are three stages to playtesting: solo playtesting (also called "alpha"), local playtesting ("beta"), and "blind" or "external" playtesting (often spoken of as part of the "beta" stage). While there are various ways to name these stages, the stages themselves exist, although sometimes video game companies leave out the "external" testing stage.

Solo Testing

It's hardly surprising that video games start with playtesting by the individual(s) making the game. But few tabletop games are meant to be played alone. Yet in solo playtesting of tabletop games, the designer plays the game solitaire, playing all the sides independently as best he can.

At this stage the designer is trying to get the game to a state where other playtesters have a good likelihood of enjoying it, and ultimately of playing it through to the end. At solo stage the designer might try a portion of the game and then stop because something isn't working, or because he has a better idea. When asking other people to play a game I almost never stop a game in the middle, or try something that might be so bad I'd want to stop, though I know of designers who think nothing of doing this. I have been known to change an obviously screwy rule in mid-game, but that's usually when I'm playing alone, not when others are playing.

Most video games are designed to be played alone, and if there's a more-than-one-player component, it's usually impossible for the designer to play several sides by himself.

As I gain more experience with tabletop versions of games, I find myself often using a small computer to write extensive notes as I play a game solo the second or third time. These notes help me later remember how the game works if I haven't written a full set of rules.

Local Testing

At the local playtesting stage, people are asked to play the game through. For a video game these are usually employees of the development studio. For tabletop games, the designer usually teaches local gamers how to play.

At this point a video game must be more or less fully realized, fully playable, so it can take much longer to reach this stage than for a tabletop game. At the beginning of this local testing for a tabletop game I may not have a full set of rules if the game is fairly complex, I just have notes about how to play, and some of the details are in my head. As local playtesting goes on, I make a rough set of rules, then finally write a full set of rules.

However, if the game is simple, or like others I've designed, I sometimes write a full set of rules before anyone other than myself plays the game. I have to judge how much writing time I'll waste because of major changes in the rules; at some point I'll think it's worth the time to write the full rules because really major changes are unlikely.

The ability to play from notes rather than full rules is a major reason why it is much quicker to design a tabletop game. With an electronic game all the details of the "rules" (the game mechanics) must be settled precisely before the programming of the prototype can be completed. The programming (which enforces the core mechanics of the game) is roughly the equivalent of the rules of the tabletop game.

As the local playtests occur, I write down notes about what I see and hear, and especially about answers to questions that need to be incorporated into the full rules. A video game designer will do the same thing, observing how the game is working and listening to player comments, especially comments about what doesn't work or what is frustrating. By the time I have a full set of rules, I usually refer to the rules for detailed questions, to see if the rules cover that question and whether it is easy to find that information. You might be surprised how often the designer of a tabletop game not only doesn't remember the rules, but can't quickly find the relevant rule in the written rules. The former happens because real designers have LOTS of games in mind, so they don't try to remember all the details-they write them down. The latter is a defect of the rules, and must be fixed.


"Blind"/External Testing

The third stage is "blind" testing, where someone who is not involved in developing the game is given the game and must play it without any intervention from the designer. For video games, there may be people from, or hired by, the studio to observe this testing, but they should absolutely not try to influence the players in any way. For Civilization IV Firaxis recruited people from the Civilization fan community to test the game. In some cases it's better to have people who like the type of game you're making, but have no connection with the particular franchise or series. For example, if you're making "Halo IV," you want external testers who are Halo fans, but you also want people who like shooters but for some reason haven't played recent versions of Halo much.

At least one company specializes in setting up external testing for studios that cannot or will not do it themselves. The slides for a presentation at the Triangle Game Conference about this kind of service can be seen at this link.

For tabletop games this is a big test of the rules, somewhat akin to video game "bug testing". Are the rules clear enough that people can play the game from the rules? What questions do the blind testers come up with, and how can the rules be improved as a result? Unfortunately, nowadays people are often poor rules-readers, so I advocate use of video tutorials to help people learn how to play a game, yet those tutorials are rarely done until the game is "done".

I know from experience with published games that there will ALWAYS be people who misread rules, sometimes willfully. 99 percent clarity of detail is about the best you can get using the English language. Similarly, most video games have many glitches until the first patch is issued several months after the game is released.

Even when you don't intend to change the rules, rewriting them introduces unintended consequences . When you rewrite to change a rule, the repercussions are often larger. So a remarkable amount of testing is needed. This is something like the experience of programming, where you change the program to fix a "bug", but sometimes another bug results.

In a sense, video games can jump to "blind" testing quickly, because by their very nature these games hide the rules from the players, enforcing them through the programming. This is a major advantage of video games over tabletop, that no one needs to read and understand a set of rules.

The most interesting testing occurs when I have let a game "lie fallow" for a year or more. By that time I don't remember much of the rules, so reading them is almost like someone reading them for the first time. I confess I sometimes wonder what the heck I was thinking when I wrote this or that!

In the video game world it is difficult to quickly and cheaply make big changes in a prototype. This is one of the problems that all makers of electronic games face, and a major reason why some video games are not very good. By the time the development studio has a prototype ready for testing outside of the company, it's too late in the schedule to make many of the changes that playtesting reveals are necessary or perhaps only "desirable".

Rules testing and bug testing

Software of any kind requires "bug testing", looking for failures in programming that cause the software to not work as desired. The equivalent for a tabletop game is rules testing. There are two parts. First, do the rules cover all possibilities? Second, is that coverage sufficiently specific and concise that almost everyone is going to understand what it means?

Change, change, change-love it or fail

"The most important thing to remember is this: To be ready at any moment to give up what you are for what you might become." W.E.B. Du Bois

You must be willing to change your game again and again and again in a search for, not perfection, but something close to it.

Sooner or later the Law of Diminishing Marginal Returns comes into play: the amount of time it will take to identify and successfully implement an improvement will not be worth the value of the improvement.

Possibly the most important skillset of a game designer is the ability to analyze play of prototypes, identify problems, and provide practical solutions to anything that undermines the quality of the game.

Analyze

This involves self-criticism: you must learn to recognize faults even when you are the one responsible for them. The goal is not to "do nothing wrong", it's to "do a lot of things right". If you're only interested in doing nothing wrong, you'll probably end up with bland pablum, an unexciting and uninteresting game. You're better off working to make things go right, and if a few go wrong, that's worth all the right that occurs.

Identify

A game can appear to work pretty well, and still have flaws. The designer must identify those flaws even if no one else has recognized them! Because once the game gets out to the public, the collective minds of many players WILL identify those flaws. Again, "Nobody is as smart as everybody."

I have had games that had not changed significantly in months of play, which I felt were "done", then I recognized I had to make a significant change.

Solve problems

Everyone can help here, and playtesters who can identify and solve problems are worth treasuring. In video game terms, increasingly everyone on the production team is expected to try to identify problems and propose solutions. This is yet another example of the maxim, "two heads are better than one". It's also a reflection of the "1-10-100 rule" fundamental to participative management schemes such as TQM (Total Quality Management). In any case, the game designer(s) must ultimately be responsible for deciding what solution to pursue. The game designer must be "in charge" yet still cooperate with the rest of the team and keep them feeling that they're contributing to the game.

Return to the web version of this article
Copyright © UBM TechWeb