Playtesting is Sovereign, Part 2

By Lewis Pulsipher [09.02.10]

 [GameCareerGuide here presents a continuation of Lewis Pulsipher's playtesting article, which continues the discussion by explaining what to do with playtesting feedback you get and how to evaluate it. You can Part 1 by clicking here.]

What to Do With The Feedback

"One of the early lessons to learn in writing is that feedback is good, but must be held at arm's length." - Brandon Sanderson (fantasy novelist who is completing Robert Jordan's "Wheel of Time" series)

"Most of the letters we'd get were almost a standard form. They were like, 'Dear Sid. I liked your game Civilization. Here are the five things I would change to make it a much better game.'" Sid Meier about responses to the original Civilization video game.

Game design, when taken to completion, is highly interactive. Playtesting sets good games apart from bad, and playtesting is (or should be) interactive.

I include Brandon Sanderson's admonition here because it applies to some extent to game playtesting. Novel authorship is very personal, game design is much more a group effort, even if there is just one designer and many playtesters, certainly when there is a large team creating a video game. You must listen to playtesters, but you have to keep in mind that playtesters each have their own likes and dislikes, and the game designer is the person who must keep in mind the original "vision" of the game and follow it to a conclusion.

On the other hand, game designers must practice not being ego-involved. Playtesting is an invitation to say what is wrong with a game as well as what is right. Comments are about the game, not about you. Playtesting is an invitation to say your game sucks, not that you suck.

I am very low-key in beta playtesting, preferring to watch reactions of people rather than try to solicit opinions, in part because people often won't say negative things even when asked. I also try not to play with/against others, as 1) the designer playing in a game tends to skew results and 2) when I play, I do a worse job of playing, and a worse job of evaluating the playtesting, than if I did either alone. As I'm that strange sort of person who enjoys watching games as much as playing (a combination of people-watching and "what happens next?"), why play?

Playtesters tend to be polite. It's hard to find out what they really think. I am skeptical that a feedback questionnaire will make a difference, though many designers use them. Rather, I sometimes try the "Six Hats" method (devised by Edward de Bono) when playtesting; specifically I'll ask players successively to put on their black hat (the judge), then the red hat (intuition and emotion) to see how they assess a game, and then the yellow hat (the positive side of assessing an idea) to see what they like about a game. With local playtesters I sometimes ask them to think of ways to make the game better (the green hat). Use Wikipedia or google "de Bono" or "Six Hats" for more information.

Keep in mind also that people tend to like games they win. When the losers like the game you can be more certain of the value of the feedback!

You don't need to "defend" your game to playtesters. It's your game, not theirs. They may not like what you want, and they can explain why, but in the end you have to decide what's best.

When it comes down to it, should a designer in playtest stages do what he wants with the mechanics of play, or what the playtesters recommend?

I believe I'm very receptive to what players suggest (or what I see that they would prefer, as they play). If people take the time to play my game, I ought to be receptive, else why bother? I think playtesters may be more likely to offer suggestions if they know the designer is receptive to them. They're certainly pleased when they play again and see that I've changed the game because of their suggestion.

A playtester's comment may cause a change in a game, but not quite (or not nearly) the change they thought of. Playtesters point out potential problems, designers are responsible for solutions (though solutions, too, may originate with a playtester).

This is where the "scientific" part of game design comes to the fore. The scientific method involves controlled experimentation toward answering a question, observation of results, new hypotheses, and further experimentation. Wikipedia's description of the scientific method (accessed 14 April 09) can be taken as a guide to what you're doing:

To be termed scientific, a method of inquiry must be based on gathering observable, empirical and measurable evidence subject to specific principles of reasoning. A scientific method consists of the collection of data through observation and experimentation, and the formulation and testing of hypotheses.

Your data is collected by observing playtesters, or by getting their accounts in writing or orally. Your hypotheses involve ways to make the game work better. You test your changes through more playtesting and see what happens.

Yet scientific method will take you only so far. Experience helps, knowledge of games helps, background knowledge (e.g., of history and physics) helps, analysis and creativity help.

What is obvious to experienced designers is not necessarily obvious to the inexperienced. Unfortunately, it is easy to find "wannabe" designers who have an idea and think they have a "golden egg" that will be a blockbuster game. One reason why this attitude exists is that they don't listen to playtesters, or that their only playtesters are family members who are too nice to say that the game needs changes.

On the other hand, especially in the video game world, many players think they know "the secret," and are willing to expound at length (as shown by Sid Meier's quote above). There are all kinds of "fanboys" (and girls) who will accost the designer of a well-known, successful video game and tell the designer how it "ought" to have been designed. This is ridiculous, but doesn't stop it from happening. At some point, no matter how good your game is, you'll encounter such folks. If I had a dime for every person who said Britannia is "very unbalanced", even though playing data shows that it is very well balanced, I'd have a new tractor. The trick is to find out what the really good game players, and the ones who are willing to study and think about a game, have to say.

There are elements in a game where there are two choices, and one will work as well as the other. Which is used is pretty much arbitrary, or so it seems to the designer. In those cases, it really is wise to choose as the playtesters choose, if only because they'll see, when they next play, that you're listening.


Specific Items to Watch For

"You tell people what you do for a living, and they're like, 'Oh, you play video games for a living.' No, I play a game that's not as fun as it should be, that's broken, until it's no longer broken. Then I give it to other people to have fun with." - Cliff Bleszinski (Designer of Gears of War etc.)

"However beautiful the strategy, you should occasionally look at the results." Sir Winston Churchill

When at a playtesting session you should write down anything you want to keep track of. I keep printed copies of the rules at hand so that I can change them, or I write all notes on scrap paper and then, after the game, transfer to computer.

Here are some specific factors to monitor when playtesting. I've divided them into categories, but you're going to be looking at all of them, to a greater or lesser extent, whenever the game is played. I'll list them first, then briefly discuss each one.

What you're doing

What the players are doing

How the game plays

How the game works

What You're Doing

"In all affairs it's a healthy thing now and then to hang a question mark on the things you have long taken for granted." - Bertrand Russell

Break Old Habits. Sometimes we do things out of habit. We should be thinking all the time, doing things because it makes sense.

Testing my prototype boardgameThe Rise and Fall of Assyria, I realized I was letting the "parent game" (Eurasia) govern what I did. In Eurasia there are two sets of pieces for different empires, and when players get a third empire their oldest is replaced by neutrals. This is a matter of piece limitations. But in Assyria each nation has its own set of pieces, so the limitation doesn't exist. Consequently, I could change the rules to allow a player to score for all of his old nations, not just for the two most recent. But I had to recognize that I was going by an old habit, and that wasn't until the third play of Assyria.

You can't be satisfied with "it works". "It works" is good, and if you've not designed games before you can congratulate yourself. But "it works" won't do for real game design. It's got to work in lots of different circumstances, with lots of different players. One of my prototypes resulted in an interesting abstract game that people seemed to like playing. But I gradually noticed that whoever was in the lead halfway through the game almost always won. That's much less than ideal, so I had to extensively revise the game, even though no player had yet complained about the problem.

Outliers. Playtesting follows the usual "bell curve" or "normal curve". Most playtest games will be near the middle, the fairly typical play, and some will be out on the "unlikely" ends of the curve. Consequently, one screwy result doesn't mean you have to overhaul the whole game-you may have just played one of those outliers-but on the other hand you must have rules that take into account those wildly varying occurrences. A game that can cope with both the normal and the extraordinary occurrences is what you want.

"Feature Creep". In any design, whether of a building or of software or of a class or of a car, there's a tendency to gradually add features, or to "enhance" (read, complicate) existing features. The general term for this is "feature creep". It is particularly bad in video games, which are usually released on a strict schedule. Generally these schedules don't leave enough time for the original conception; every feature that's added makes the game more likely to be unfinished.

This is a poison that must be vigorously opposed. Playtesters will suggest additions to the game; don't use them unless you're convinced that the addition is worth more than the complication. (When you're playtesting you can try something to see how it works out, but remember that one try may not give you a good handle on it-the sample space is too small.)

"Chrome". This is a term for special rules or entities that give the game a special flavor, but also complicate it. For example, having leader pieces in a game about history is often "chrome" (though if the main point of the game is the person, rather than the person's nation, this would not be chrome). Each bit of chrome should improve the game more than the added complexity "disimproves" the game.

"Feature creep" and "Chrome" are related to "Keep it simple". New features, or more complex features, tend to make the game less simple. Yet once the game is in a fairly healthy state, the guide should be to simplify. As I have quoted before: "A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away."

(What about those who don't believe in "Keep It Simple? Remember that even complex games, video or otherwise, can be reduced to a few simple things that are happening, the "essence". More people play the game because of the essence rather than because of the "chrome" or "features".)

"That's not the way to play". Don't get the idea that people won't do something because it's "unsporting" (like "camping" or "turtling"). Some players will do whatever the rules allow. If you don't want people to camp or turtle, you'll need to design the game so that such strategies are unsuccessful. (Even if you call the camper names, if he wants to win, he'll camp and the heck with you, you loser noob!)

During playtesting, look for ways to play that many people might not think of, but that will dominate when used. This is where it's really good to have some excellent players among your playtesters.

Why Use Dice/Chance? Dice are perfectly reasonable parts of a game, but only if there's a reason to use dice. Don't use dice (or, in a computer game, randomization) "just because dice are in games"; in particular, DON'T use dice for movement of pieces unless it really makes sense.

In the real world, movement is generally pretty measurable and predictable. Combat is not nearly as predictable. So use of dice in combat makes much more sense than use of dice in movement. And there are lots of ways to do it, varying greatly in how much chance is involved.

Combat "one on one". Avoid the "one on one syndrome". Let mass play its part in battle. Don't require fights to be one against one even if there are ten attackers and two defenders. The ten should be able to overwhelm the two, unless there's some terrain or other reason why the ten cannot attack the two at the same time. Having units fight one against one, serially, destroys the advantage of superior numbers, which removes a major reason to maneuver.

The purpose of maneuver in wargames, of having maps and boars to move on, is usually to concentrate superior force on a target. If it doesn't matter how many enemy are at the fight, why bother to maneuver? And if superior maneuver doesn't play a part in battle, will it be reduced to mere dice-rolling randomness?

This one-on-one stuff reminds me of the Iliad, where the Greek and Trojan heroes would sometimes fight each other one against one while the rest watched. You hear of it also in China's Romance of the Three Kingdoms. It's quite unlikely this happened much in the real world.

What the Players Are Doing

User Interface Convenience. This applies to all games, but especially to video games: can the players use the interface to control their game conveniently. How many video games are ruined because it's too fiddly to make the moves the player wants to make? Keep your eyes open for this kind of thing.

Rules difficult to grasp. What do the players find hard to grasp. In my prototype Age of Colonization (AoC), players had trouble grasping the difference between movement of units and placement of units. I used the same distinction in an abstract stones-and-hexes prototype, and no one has a problem, showing that context makes a big difference. Even though players "get it" after playing, it might be necessary to change something that's hard to grasp. In AoC I changed the rules extensively to recast/eliminate the distinction, and the problem disappeared.

When people are playtesting a game, after 20-30 minutes, if I walk away do they still know how to play the game? For all but the most complex games this should be true even at first play.

What do players tend to forget? This isn't quite the same thing as what's difficult to grasp. Some options just don't stick in people's minds. Is there anything you can do about it? Is there some play aid to help people remember? In a video game, can you show something on the screen to remind people of the option?

What do players not bother to use? Some rules/options exist but no one uses them. If the threat of using them is not making a difference in the game, then perhaps you should eliminate the option. For example, in my hex-and-stones game "Law and Chaos" I originally allowed people to move a piece rather than place one. This happened rarely, as it was usually better to place another piece and increase the number on the board. So I eliminated the possibility, except as an "optional rule".

Horns of a Dilemma. Are there enough plausible decisions in the play to make the players think, but not so many that "analysis paralysis" sets in. Even in a simple game, if a player can do only two of five possible actions in a turn, is there tension here or are the plays obvious? As one commenter put it, do the players sometimes feel "so much to do, so few actions allowed?" That would be a good thing, for a competitive game.

Player interaction. Do the players have to take the actions of other players into account? Yes, some games are virtually multi-player solitaire, and some players are happy with this. But most players want to be able to affect other players with their moves.

How the Game Plays

Length. A game is always longer to new players, of course. But if it takes too long for new players, will they play again? Length is quite dependent on how much players enjoy what is happening in the game. The original boardgame Civilization can take 8 to 12 hours, but those who love the game don't find that time weighs upon them.

Downtime. Downtime is the time people must wait while someone else is taking a turn. This can be a problem even in a video game that's turn-based. A great advantage of turns is that they give people time to think. If the game isn't encouraging thought, and they must wait too long, there's a problem.

Game balance. Even if the game is symmetric (all players start with identical situations), there may be an advantage to playing first (or last). Chess is symmetric except for who moves first, but move-first is a huge advantage. For a single-player "interactive puzzle" video game, are the challenges matched by the rewards? Is the game fundamentally too hard or too easy?

Dominant Strategy. Look for any dominant strategy ("saddle point"). This is a strategy that is so good that a player who wants to win must pursue it; or a strategy so good that some will pursue it, yet that strategy renders the game less than entertaining. For example, in a Euro-style 4X space game I've designed, one player found that by getting together a sufficiently large force, along with certain technology research, he could completely dominate other players who weren't pursuing the identical strategy. I want the game to offer a variety of ways to success, so I had to change the rules fairly extensively. This is why it is important to have testers who are dynamite game players, so that they'll find these strategies during testing, rather than have someone find it after the game is published. I'm lucky that I can be such a player myself when I put my mind to it.

Taking it to the Max. Can extreme behavior within the rules break the game? Sure, if someone pursues a bad strategy, they'll lose. The question is, is there some extreme strategy that results in an unfair or unenjoyable game?

Adequate control. Do the players feel that they can exert a measure of control over what happens in the game? Remember, any (strategic) game is a series of challenges and actions in response to those challenges.

Analysis paralysis. Are there too many things to watch for or keep track of, or too many choices, so players either freeze up or give up on figuring out what is the best thing to do? There are always "deliberate" (slow) players, the question is, is everyone slow or frustrated?

"Derivativeness". Just because something is done in one game does not mean you can't do it in yours. In fact, there are few original ideas in games. But try not to make your game TOO derivative of another. E.g. there are many deduction games, but if yours has items and people and rooms and questions and is otherwise a whole lot like Clue (Cluedo in the UK), maybe that's too derivative to be commercially viable.

How the Game Works

Stages of play and pacing. You probably learn this in solo testing, if you do solo testing (which I strongly recommend). Are there identifiable stages in the game, especially ones where the typical run of play changes? E.g., in chess there is the early, middle, and end games. Pieces are deployed in the opening, mix it up in the midgame, and so forth. An exploration game has the expansion period followed by consolidation and then (usually) conflict. As another example, Britannia has historical stages, the Roman conquest, then the Anglo-Saxon conquest, then the Viking raids and settlements, and finally the struggle to be king in 1066.

Replayability. This is less important in our "throw-away" age, but almost every excellent game is one you can enjoyably play again and again. Video games tend to become obsolete, but centuries-old board and card games (chess, bridge, go, etc.) can be played repeatedly. Some games have limited replayability because knowledge of the story makes a big difference, but these are exceptions, not the norm.

Player interest/enjoyment. What part(s) of the game seem to be most interesting to the players? I'm not in favor of trying to figure out "fun", because fun comes from the people who are playing more than from the game design itself. And there are many games that I wouldn't call "fun" (including my own Britannia) that are nonetheless interesting and even fascinating.

Scale it down. Can you change the scale of some aspects of the game? This especially involves numbers. If a player earns 50 resource points for occupying a particular location, and the least expensive item you can purchase with those points costs 10, why not divide everything by 10 and have the numbers be 5 and 1? (On the other hand, some people just LOVE big numbers, provided those numbers are tracked by a computer rather than by the players. So you can make it 50,000 and 10,000, in a video game.)

Is there a way to combine two functions into one? Sometimes you can improve play by simplifying, without significantly reducing the choices the player(s) make. Can one thing take care of two questions or combine two decisions?

For example, two of my students designed a simple, not-historically-based wargame. They collected resources (represented by plastic coins) from "mines" they occupied, then spent that money for new units and to replenish existing units. Each unit consisted of several stackable pieces. This all happened over the course of multiple turns, e.g. collecting from the mines every third turn.

The setup was also lengthened by handling dozens of coins. I'd already suggested reducing the numbers by simplifying costs. For example, it was gain 50 per mine, spend 200 per infantry or 300 per tank unit, 25 per replacement piece within a unit. Why not divide everything by 25 for 2, 4, 6, 1?

There is some pleasure in handling lots of coins, even if they're plastic. Yet in a published game this function would be fulfilled by paper money, most likely, to reduce costs. And even "play money" is expensive these days. So after several playtests I finally put my foot down and said, let's do away with physical manifestations of money altogether, and let the individual stackable pieces that made up the units help keep track of money. That is, each turn a player collects from his mines (so no one has to remember it's the "third turn"), all the money is spent immediately for individual pieces, and when enough pieces are accumulated at the base, a new unit can venture forth. This also left pieces to be used to replenish damaged units in the field. The result was that no money was needed.

I asked why there was a turn delay between building a new unit, and moving it. "It represents training and so forth". OK, but in playtests new players often forgot or didn't immediately understand this. And this was a rather abstract wargame, not a realistic one. So in keeping with the idea of simplifying the entire economic cycle, I suggested letting the new unit move immediately.

Who's keeping track? This especially applies to tabletop games, but can come up even in video games. Are the players forced to remember or keep track of things that aren't part of the enjoyment of the game? Some mechanism should make this easy, if it must be done at all. For example, in a tabletop game if something happens only every third turn, there had better be a really simple, more or less foolproof way to keep track of "every third turn". In a video game, the computer had better keep track of it and make it easy for the players to check the current state. The better solution is to find a way to change the rules so that "every third turn" isn't used. For example, can it be something where a third of it happens every turn?

Is there a way to eliminate it, or make it easier to keep track of? This is more common in tabletop games, but even in video games, if the player(s) tend to forget to pay attention to something that is important, then it's "hard to keep track of" even though the computer IS keeping track of it.

In general, anything that happens only periodically (every fourth turn, say, in a turn-based game) can be a problem.

Components and Play Aids. Do the physical parts of the game help play flow smoothly, or does something need to be changed? Is there too much record-keeping? How can it all be simplified?

Playtesting, for a game designer, is about being aware of what happens, and receptive to changes to make better things happen. It doesn't matter who thinks of the change, what matters is that a solution is found and implemented. Keep your mind in gear!

Return to the web version of this article
Copyright © UBM TechWeb