Student Postmortem: Northeastern University's Shortfall Digital

By Mark Sivak and Seth Sivak [08.09.07]

What do engineering, automobile production, and innovation trees have to do with games? In the case of Shortfall Digital, they're all pieces of an educational game that started its life as a professor-made board game.

Shortfall Digital has been under development for several years at the department of Mechanical and Industrial Engineering (MIE) at Northeastern University. Conceptually, the game started its life as a board game about environmentally benign manufacturing and the greening of the supply chain in automobile production.

The original Shortfall board game was created, tested and assessed through an industrial engineering master's thesis funded by the National Science Foundation (NSF). The objective was to explore different ways of teaching engineering concepts and raising environmental awareness through collaborative and experiential learning.

With additional funding from the NSF, a team of students and faculty revised the board game and created a second edition, drawing on their expertise from the departments of engineering, multimedia studies, and education departments, and with contracting work provided by Metaversal Studios.


This version was tested with a group of undergraduate engineering students, who were subject to a pre-test and a post-test for knowledge retention, and a focus group discussion based on the gameplay. Their insights helped enhance the game and player roles and helped clarify how team play would work.

During the spring of 2007, we joined the team as senior undergraduate mechanical engineering students and worked on the game as an independent study project in our final semester before graduation. Our objective in this independent study was to implement the adaptations suggested by previous play testing, either by revising the board game followed by more focus testing, or by developing a prototype computerized version. After some initial research on educational games and effective teaching and learning, we decided to create an electronic version. This prototype is the browser-based game, developed in Adobe Flash, and is the subject of this postmortem.


What Went Right

1. The team. One of the nice things about being on this project was that we had a diverse group to help us. We worked with several excellent professors who had worked on the second edition of the board game: Jackie Isaacs in the MIE department, Donna Qualters from the Center for Effective University Teaching, and Ann McDonald and Jay Laird from multimedia studies. The interdisciplinary nature of the original team is an indication of the complexity of this project. Additionally, without the help of the other members of the team and graphics support from some fellow students, the project would not have been possible.

The original game resulted from a well-researched master's thesis, and the second edition had taken a massive overhaul of everything from gameplay to actual content. Without the solid base developed by the previous team as well as the group we worked with, it would not have been possible to complete anything in a semester-long timeframe.

We were granted a considerable amount of freedom for where we wanted to take the project. Our backgrounds were in both mechanical engineering and game design. The added bonuses of working on this particular project were that 1) we had previously created educational games and 2) we essentially fit the target market: undergraduate engineering students.

2. Streamlining the supply chain. Both versions of the board game are very complex and simulate a supply chain for automobiles. Each competing supply chain has three distinct tiers -- materials, parts, and cars -- with four players per tier. That amounts to 12 players per supply chain.
The different tiers compete as teams against each other, but also require each other for success, and the final win condition is determined by the total assets available after five rounds.

Each player on a team is given a direct role in the decisions made during his or her turn. The roles are CEO, production manager, R&D manager, and environmental manager. The board game also required an in depth tutorial and an actual human game master to facilitate the current events and changes from round to round.

We learned from student game tester comments that this complexity, while interesting, was taking away from the game. We decided that the entire structure of the supply chain and the gameplay needed to be reworked.

The first thing we decided to do was change the way the teams were formed. We put six people on a supply chain, two per tier. Then we reformulated the game to be a competition between two competing supply chains so the overall number of players was still 12. Restructuring the players allowed us to solve a lot of internal balance problems that were in the board game as well as simplify the player roles. And having competing supply chains strengthened the feeling of collaboration among the teams of tiers -- materials, parts, and cars.

Finally after all these changes occurred, it became possible to add more rounds to the game. Initially the board game only had five rounds because the first round involved a scripted tutorial that was guided by the game master. The digital version removed the need for the game master but was replaced with a narrative instruction packet handed out prior to the game, followed by an interactive tutorial. These changes allowed us to expand the game to 10 rounds, which meant we could create different aspects of the gameplay for shortsighted players and for those who had longer-term strategies.

3. Reorganizing player roles. Putting two players per tier completely restructured the roles of each team. The two new roles were innovation manager and economics manager, corresponding to the two user interfaces involved in each team's turn. This change helped eliminate a lot of confusion that the players had and helped consolidate the roles, which some players thought were too boring or simple. It also opened up a new range of player choices and tied together how each team needed to strategize and plan together for each turn.

We saw several other potential advantages to reorganizing the player roles. For one, the two new roles both had essentially equal power, whereas in the previous versions, the CEO had the ultimate decision-making power.

4. Improving the interaction by beating the networking problem. Switching from a tabletop game to a digital one posed a number of unique challenges for us. These challenges, coupled with the fact that we were under a strict time constraint and did not have much experience with the platform (Adobe Flash), we had to change the design a bit to facilitate functionality.

We decided early that a fully networked version of the game was simply too much for us to undertake in this independent study. However, we still wanted the supply chains to be able to compete against each other with the possibility of declaring a winner when the game was played in a classroom setting. We envisioned each team of six passing around a single laptop while taking turns playing.

One of our goals was to not just make a single-use game, but a replayable one so students could experiment with different strategies. Our fix for this was to use an innovative feature that revolved around setting pseudo-random numbers for the market changes and the order of the current events. Each game is numbered, and the game number links to a specific set of data. Therefore, as long as each team plays the same game number, all teams will be essentially playing the same game and can compare scores at the end, enabling competition in the same classroom or across the country without requiring an internet connection.

This dynamic also made it possible for people to play the game at any time. A group could play game number 6 and then a month later a different group could play the same game and they could compare scores. The groups do not have to play concurrently.


5. Restructuring "innovation trees" to maximize learning. Most of the facts and real world material in Shortfall are stored in what we call innovation trees. It was important to make the trees a highlight feature in the digital version of Shortfall instead of an unstructured, non-strategic decision card as in the original board game. This part of the game was where a lot of the fun and strategy would come into play.

However, we needed to balance the choices available to the players so that there was no one single path to victory. To accomplish this, we had to create a new way of handling the scoring. Rather than just scoring the profits gleaned, we added in a green score (after all, it is a game about raising environmental awareness). This provided an opportunity to create choices among options in the innovation tree. On each level of the tree, there is a technology choice that allows opportunities for either higher profits or increasing the team's green score.

To add to the strategy development, we added an innovation mastery. These are special Innovations that take a large number of prerequisites before you can invest in them, but that reap excellent rewards. Much effort was put into making the Innovations Tree and the Mastery options logical and as real as possible. Careful consideration was taken when placing the Innovation in the specific tree and level. We feel that the redesign and development of the innovation tree was a major intellectual contribution to the strategy of the game and that it is a truly polished feature.


What Went Wrong

1. Scope and time management of Shortfall online. While we were brainstorming, we tossed around some simply amazing ideas that would have made for an incredible game. We originally slated animation and networking as two of our goals for the project, but in the end, many features had to go to the chopping block due to over ambitiousness and poor time management.

Inexperience can breed overconfidence, which put our sense of time in a blitz, especially because we were two students with other classes and a rapidly approaching graduation date.

Despite several days of crunch time, we missed our first play test date. The game was simply not ready. Luckily, our advisers were kind enough to allow us an opportunity simply to discuss our game with the class, a mix of graduate and undergraduate mechanical and industrial engineers. Their discussion proved to be incredibly helpful, and we heard some great feedback about the game, despite it not being a finished product.

Once we had a final set of features in place, we still underestimated the time it would take to actually create and test the game. As much as we can attribute this problem to our inexperience and limited knowledge of designing games in Flash, it's also an extremely common painpoint even for professional video game developers.

2. Using Adobe Flash as a platform. We had very limited experience with Adobe Flash. This became obvious during our first iteration of the game, which we attempted to finish for our first play test. We simply did not finish on time.

We found it very challenging to have more than one developer working on the game at a time. What made it even more of a struggle was that we were pressed for time and had to work on everything in series rather than a few things in parallel. It got worse when we realized how badly we underestimated the complexity of the game itself.

After missing the initial play test deadline, we decided to totally rework entire portions of the game, a valuable move in the long run because we had learned so much about programming in Flash and building browser-based games that we could go back and change all the errors we had made earlier.
Reworking portions of the game also gave us a chance to optimize the code and clean up the work we had already done.

The released product, while not as polished as we had hoped, exceeds what we thought could be done given the timeframe.


3. Balancing green score, money, current events, storage, and random markets. Shortfall was built as a simulation to reflect the real life decisions and factors that go into making engineering choices. By nature, it's a very complex system with dozens of variables changing the dynamics behind the production and sale of automobiles.

The sophistication of the real life system made balancing the game the most challenging aspect of the entire project. We spent a great deal of time adjusting numbers and trying various strategies, but it seemed like we could never do enough. No matter how hard we tried, we ended up with a certain strategy that was working much better than the rest, and adjustment of those variables would switch the results the other way.

Feedback from the board game testing informed us that many players felt the game was not fair and that too much was left to chance. Our goal was to eliminate any major balance issues while reinforcing that player choices, not random factors in the game mechanics, actually facilitated the outcome.


Knowing that the game was going to support multiple players, with groups competing against each other, we felt that an obvious unfairness would draw attention away from the possibility of learning, as student would focus on what was unfair and not the actual topic of the game.

4. Instructions for gameplay. With the date of our first play test behind us, we made the decision to try the game out two months later on a much bigger audience: 80 mechanical and industrial engineering students.

The day before the students play tested the game, they heard a lecture on the same subject matter as the game's content: environmental issues involved in engineering decisions.

The day of the play test, we had 17 teams, each with its own laptop. The students took two knowledge tests, one before the game and one after, to assess what they knew coming in versus what they learned from playing, as well as their confidence level about the topics.

We offered only a very brief introduction and tutorial, avoiding a lengthy explanation of how the game worked because we wanted to observe how students might learn through trial-and-error. The groups also received information packets that discussed the game basics of how the market worked.

Unfortunately, we somehow had a miscommunication prior to the printing of these packets that left out a full innovation tree, which we felt would expedite player choices and give students plenty of information to help them form strategies. If we could do it again, we would say we must include a full innovation tree in the handouts. The instructions were presented as narrative memo handouts and were given to the students the day before the play test and then again the day of the play test.


Educational games are always a tough sell because they are typically built on a very limited budget (in our case, $0) and are not created by professional game developers -- although we had a lot of input from professional game designers.

By approaching the game design from the point of view of students who might play Shortfall, we really tried to make the game interesting and fun first, and just slip in the educational information where it would in fact add to the game, rather than sap it. Judging from the initial responses to the test play, it seems like this version of the game was at the very least an upgrade from the original board game.

Personally, this project was a great experience for many reasons. It was a fun excursion from our typical engineering studies and showed that engineers can in fact design a respectable and creative game.

The finished title is another step in the right direction for educational games. Hopefully, the Northeastern University and Metaversal Studios teams can continue this partnership and press on with the development of Shortfall Online - a fully networked computer-based game.


Web site and Trailer:
Developer: Northeastern University and Metaversal Studios
Publisher: Northeastern University
Number of Designers: 2
Length in Production: 3 months
Release Date: June 14, 2007
Platform: Web Browser
Development Software Used: Adobe Flash, Photoshop, and Illustrator
Development Hardware Used: Dell Precision 670 Workstation
Machine Project Size: 200KB

Co-authors Seth Sivak and Mark Sivak graduated with a bachelor's degree in mechanical engineering from Northeastern University. Seth will be attending graduate school at Carnegie Mellon University, where he will be working toward a master's degree in Entertainment Technology from the Entertainment Technology Center. Mark is currently a research assistant at Northeastern in the Mechatronics and Robotics Laboratory, researching how virtual reality games can help patients use robotic rehabilitation devices.

Return to the web version of this article
Copyright © UBM TechWeb