Student Postmortem: Carnegie Mellon's Northrop Grumman Recruitment Game

By James Portnow [06.21.07]

 With game schools cropping up all over the world and more traditional industries looking towards video games as a new and powerful tool for training, marketing or even alternate of revenue sources I thought it might be an appropriate time to discuss a project we did for Northrop Grumman.

Northrop Grumman approached Carnegie Mellon with a dilemma: they couldn't hire enough qualified engineers because most college grads wanted to work for Microsoft, Google, or Activision. They needed something to make them ‘cooler'. Clearly, our answer was a game.

After talking with them for some time we found out that they did most of their recruiting at college recruiting fairs, so we made a five minute experience to draw people to their booth. Early on we decided that our job was just to attract people, the recruiter would sell the job so long as there were people to sell it to.


This presented a group of interesting challenges, from how to design an almost instructionless experience to making the game portable enough for the recruiters to take with them to recruiting fairs but far and away the most interesting challenge was working with a big corporate client that new it wanted a game but didn't really know what a game was.



Without further ado: the trials, tribulations and triumphs of working with Northrop Grumman.

 


 What Went Right

The Team

I'd argue that the team matters more than the project. I'd also argue that we had (almost) the best possible team for this project.

The team was Trey Davenport, taskmaster extraordinaire (producer), Giray Ozil, shader wizard and graphics god (lead programmer), the ever lovely Maria Barot, code mistress and sweet priestess of color (programmer), Ashutosh Mimani, my accidental brother in adversity and a man who will work himself into the ground to get it done (programmer), Samik Bhowal, wellspring of energy and control systems crafter (programmer), and myself, James Portnow, lead designer and ne're-do-well.

Now for the part that's relevant to you the reader... As you can see the team was programming heavy. This allowed for rapid prototyping and quick iteration. The ability to make large changes quickly became key to crafting the experience as we were never stuck with a decision and we could let playtest be the final arbiter when a problem arose that we did not know how to solve.


Moreover, as a designer, having an amazingly strong programming team pretty much let me blue sky. I remember asking if something was possible given the engine that we were using and getting the response "No. We'll have done by six." True to their word, six pm rolls around and the programming team has built major new components into the engine in order to do what I asked. You couldn't ask for more as a designer and, honestly, the product turned out ten times better because we were able to continuously improve the design.

This programming strength also allowed us to have a dedicated designer and producer. Given the size of the team this is incredible but was paramount to the success of the project. Our team was fairly democratic, with the person who was most passionate about a subject usually getting the final say on the issue, but just having someone whose job it was to have the last word on design or scheduling streamlined the process greatly. Without having someone whose job it to make sure we were following the most efficient path towards our goal and someone else to define what the goal was I doubt we could have delivered a finished product in the limited time available.

You probably also noticed one thing that this team is lacking... an artist. We had to contract out for our art and this hurt us greatly, but we'll get back to that later.

The Client

Our contact within Northrop Grumman was a man named Mark Conger. There are three factors that I believe set Mark apart from your standard corporate client:

Previous experience in the games industry: Mark's industry experience gave him a better idea of the limitations of the project. It also allowed him to give valuable feedback regarding the design of the game. He was a great intersection point between the corporate world and world of video games. He often helped us meet the needs of the corporation without sacrificing too much of the gameplay.

Boundless Enthusiasm: Our project was Mark's pet project. He honestly wanted it to succeed. It wasn't just "part of the job" for him. He flew out several times to meet us and even went so far as to storyboard out his thoughts on how to design one of the major sections of the game.

Trust: Mark trusted us enough to give us a pretty free hand in the creation of the game. He understood that this was our field of expertise and that we wanted to make the best possible product for him.

This was an unbelievable boon for us. I've heard tales of game makers having to drag their clients kicking and screaming to make a good game (or worse yet, not being able to). Not having to waste time or energy fighting to be allowed to make a good product was instrumental in, well... delivering a good product.


5-Minute Experience

Early in the project we decided that we were going to make an experience that lasted no more than five minutes. This allowed for high turnover at the recruiting booth and didn't take too much time away from the participants at the job fair. This also greatly limited what we could do with the project...

But these limitations became some of our greatest strengths. With a team of six and three months to work we had no chance at rivaling any of the current gen flight sims, luckily we didn't have to, we only had to make our five minutes as good as any five minutes they had to offer.

The five-minute experience also forced us to be realistic about what we could include in the game. Much of this project was about cutting features rather than adding them. Initially we had a detailed set of countermeasures that could be deployed from the plane but we but the entire countermeasures feature when we realized that it would take to much time to teach the player how to use them.

Playtest

Every week we'd find a nearby university, haul the entire experience out there and play test rigorously. We'd have the users fill out surveys after they had completed the experience then Trey, our producer, would compile the results into a series of charts and graphs for that we'd discuss at the next team meeting. The experience taught us a lot about test methodology.

At one test we were testing out two different arena maps. The survey asked the users "which terrain did you like better, the first terrain or the second terrain you played". Every one of the users answered that they thought the second terrain was better. This baffled me as I thought the first one was significantly more fun, so I switched the order in which the users played the maps. Low and behold everyone still answered that they liked the second map better.

By the time the players got to play the second map they had a better handle on the game in general and so enjoyed the play experience more. When we asked them about maps they naturally assumed that the change in map was responsible for their improved play experience. To avoid this in the future we showed any given group of play testers only one map to play on and asked them to rate its "entertainment value" on a 1-10 scale.

The shear amount of play testing we did coupled with close examination of our own methodology was a key element in improving our game. Most of all it helped us determine what was intuitive to a player (as our game was to be played largely sans-instructions). Whatever we found ourselves telling players again and again during the play test were the things that we worked on making more intuitive.

Lastly play testing was very important for dealing with our client. It was an easy way to resolve disagreements. Anywhere where the team's opinion and the client's opinion differed we'd simply conduct play testing and do whatever appealed most to the end user.

Team Building

Early in the year our producer jokingly suggested that we all actually (gasp) play video games together. We called it team building and that's just what it was. Surprisingly playing together had an amazing effect on the team. First it gave us a shared experience other than work, it gave us a way to hang out and something to talk about that wasn't work related, this brought us much closer as a group. Second, if you are cooped up in a room with the same six people for fourteen hours a day, spending a half an hour killing each other can be quite healthy. Third it gave us a way to recharge. No matter how dedicated you are, if you work continuously for twelve hours your work will start to flag. Lastly it gave us an incentive, I remember more than once saying phrases like "After we get the missile system let's play a game of UT".


 What Went Wrong

The Client

While working with Mark was great, working for a big corporate client was not. Things move very slowly at Northrop Grumman, they think in much longer time spans than we do, which made interacting with them difficult.

To answer many of our questions we had to go through multiple divisions and it would often take weeks to get a reply. By the time we ended up receiving the machine they would actually be running the game on it was very late in the project so our ability to develop specifically for it was hamstrung by a lack of time.

Also, game decisions had to take a back seat to corporate decisions. Showing off the products was primary. We ended up going with a three quarters perspective rather than a cockpit view to better display the planes Northrop Grumman had worked on, even though a cockpit view might have been more intuitive for the players.

The Venue

We weren't creating a video game, we weren't creating location-based entertainment, we were tasked with creating a mobile location based video game experience.

It's certainly hard enough to say, but what were the challenges in creating such an experience? Let's break it down word by word.

Mobile... this experience had to be able to be set up by one person and carted around the country with ease. Moreover it had to be designed to be setup by someone who was, as they put it, "technical". It also had to be a "hands off" experience that would just run by itself so the recruiter could talk to the people coming to the booth.

We solved some of the mobility problems by using a single shuttle PC, faking networking by outing the game to two monitors (making it essentially a split screen game with the split coming in between the two monitors) and using 5.1 sound to emulate two separate stereo outputs which we then routed to two pairs of headphones. Even after all this effort the setup was cumbersome and more unwieldy then we'd like.

Location Based... The object of the game was to attract people in a crowed job fair. This meant we had to make it flashy without taking up too much space; that it had to call to students without any audible sound. Additionally it had to do all this with expensive technology that couldn't get stolen or broken. We tried to pick out the hardiest equipment we could find, we also recommended to Northrop Grumman that they paint all the hardware Northrop Grumman colors to make sure it is easily identifiable. Unfortunately, without making sacrifices in other areas, this was about the limit of what could be done.

Video game... This was a video game specifically aimed at college grads in a technical field. We decided early in the process that this meant it had to look as good as the average PS2 game. Here I have to give the programming team credit. I feel as though they knocked this one out of the park...without a dedicated artist.



Usability (Lack of 5-Minute Experience)

Even though we had all spent the previous three months working on short experiences, making an experience that was five minutes long and almost instruction free was an enormous challenge.

We cut feature after feature and programmed or designed a number of extraneous elements that didn't end up making the final product. The real difficulty here lies in the fact that, as a team, you end up getting to know your game intimately and thus are tempted to make it more interesting rather than easier.

We learned over the course of this project to bombard the player with feedback. Audio, text, visual, even color cues were used to convey what was happening to player in every way possible.

One of the problems we faced early on was that players couldn't tell when they were hitting their targets. We added a large amount of feedback to convey when the player was hitting their target and then play tested again. When we surveyed they players we found that they all knew when they hit their targets but, to our surprise, none of them could tell us how they knew. None of them had consciously assimilated any of the feedback we had given and yet they ‘just knew'. At this point I'm convinced, it's hard to give too much feedback.

Art

Having no artist on the team was a terrible blow. While we planned for this by choosing a game concept that was light on art assets there really was no workaround.

We contracted out for our art. While this worked alright for many of the major art assets, anything that might need to be tweaked or iterated on was became impossibly arduous process, which completely crippled our UI design.

Since usability was our focus and we were going with a test and iterate style of design we needed to be able to make minor changes to a lot of the 2d art on an almost weekly basis. This was impossible outsourcing the art and, as the studio we were working with wouldn't give us the .psd or .tiff files they were working with we couldn't even do it ourselves so there were many times when work on the project bottlenecked around the art.

In the end we ended up ripping apart the files they gave us and making our own editable files for much of the 2d art. This not only wasted a lot of man hours (as it was redundant and having a group of non-artists work on graphic design is always going to be slow) but it also slightly lowered the quality of the visual presentation of those sections of the game.

Having a single artist embedded on the team would have helped us greatly. If I could change one thing about this project I would have added an artist to the team.



Split Focus

One of the dangers of doing a project with students is that the project can never be the sole focus of people working on it. We all had classes to attend and homework to do, we all had to look for internships or jobs, and, while I'm confident in saying that everyone on the project averaged at least sixty hours a week working on the project itself, we couldn't devote all of our energy to the project.

Times Ahead

The relationship between industry and education will only grow in the games field. Educational facilities provide a remarkably cheap way for corporations to develop interactive media. Corporations provide an excellent source of funding for games schools. Symbiotic interaction is unavoidable.

This interaction though will require understanding and compromise from both sides. Game students won't have the same freedom to experiment and play with the media working for a corporation that they would working on their own projects. Corporations won't have the ridged control they would working internally or with another corporation.

Regardless, one thing is certain, the marriage of education, industry and games is here to stay.

Website and Trailer: http://www.youtube.com/watch?v=FLRQZYcnbz4
Developer: Round Table Games (a CMU group)
Publisher: Northrop Grumman
Number of Designers: 6
Length of Production: 3 months
Release Date:
May 17th, 2006
Platform: Computer (Dual Monitor Recruiting Fair Experience)
Development software used: Panda3d, Maya, Photoshop, Adobe Audition, Reason
Development hardware used: 6 Dell Precision 380 Workstations

Machine
Project Size: 70MB

 


Return to the web version of this article
Copyright © UBM TechWeb