By Katrin BeckerBefore researchers can perform studies using commercial games, they must choose which game or games to study. The manner in which that choice is made and justified is the focus of this paper. Ideally, research informs pedagogy and when looking at game education it is important to be able to justify and defend conclusions drawn from game studies so they can inform best practices in design and development. As the number and sophistication of titles released in a given year continues to rise, it becomes even more important to look more seriously at how we are choosing the games we study, the criteria we use for those studies, how we support our claims about the suitability of the game for our purposes, and how generalizations to other games should be limited or qualified. This paper is a report on a qualitative meta-analysis of the methods used in choosing games for study and the implications that holds for both researchers studying games and educators teaching about games and game development.
KEYWORDS: Research Methodology; Game Studies
Digital games have been around for about 45 years now (Williams, 2006) and game studies as a recognizable discipline has been around for a little over ten years (Wolf, 2001). Digital games continue to grow both in popularity and variety and the discipline of game studies as well as programs in game design and development are growing right alongside them. Methodology in game research is largely still an ‘undiscovered country’ and although many useful approaches from other disciplines are used, the nature of the artifacts themselves (i.e. digital games) have sufficient unique elements to warrant the development of some game-study-specific approaches. A first step in the process of developing such approaches is to begin to classify the kinds of research done using games. The kinds of research conducted with and on games can be subdivided into several broad categories that necessarily affect the both what kinds of approaches are appropriate as well as what kinds of conclusions can be drawn. One category involves examinations of one or a small number of specific games, which is the focus of this article.
Examinations of specific games can produce various insights in the same way that examinations of specific literary or other artistic works can, and while there remains an interest in examinations of certain specific games for specific detailed purposes, such as Kurt Squire’s doctoral study using Civ III (2003), as the number and sophistication of titles released in a given year continues to rise, it becomes necessary to look more closely at how we are choosing the games we study, the criteria we use for those studies, and how we support our claims about the suitability of the chosen game(s) for our purposes. Often, studies of individual games are conducted with the intent to generalize at least some of the conclusions to other games and/or other players. Sometimes studies seek to address questions about entire genres. Given the number and variety of games with no cleanly defined delineations of genre, how can we increase the confidence that it is even possible to examine one game in order to make generalizations to other games?
Many games are no longer trivial or frivolous so the question of generalizability is not a straight forward question. In an established discipline, claims that a particular artifact or work meets certain criteria critical to the analysis are usually supported by something beyond the author’s say-so. As studies on, with, and of games become more accepted and common in mainstream academic research, it will also become more important to justify the choices of subjects. It did not appear as though such justifications were the norm, so a meta-analysis was conducted to determine the reporting frequency of explanations of subject choices in the existing literature. The analysis made no assumptions about the validity of the results reported in those papers, the purpose was specifically to find out whether authors explained or justified their choice of game(s). Several sources of game study literature were consulted (such as the DiGRA & FuturePlay conferences and the author’s own reference library collected during five years of doctoral research), along with a more general Google Scholar search in order to gather literature on games from various disciplines. Only studies that specifically mentioned commercial games were considered, and of those, only those studies and reports were selected where the question could legitimately be asked, “Why was THIS game chosen?”
THE ACADEMIC STUDY OF GAMES
Why is it important to justify the choice of game being used as an example in a scholarly article or for the purposes of study? In the early days of games studies there seemed little call for careful scrutiny of one’s game choices. We studied what we had handy and wrote about the games we happened to be playing. However, if we want to make the case that the game in question can lead to some broader insight or that it is in some way representative then we really should have some evidence to back this up. When a single game or a small number of games are chosen as the subject(s) of study they form part of the bounded system that is the case being examined, which in turn implies that there are identifiable aspects of the game(s) that makes the case of special interest (Stake, 1995). If we are proposing the use of a game for some serious purpose such as education or social change, or if the study of some specific game is intended to uncover some insight applicable to our agenda, then whether that agenda is the examination of its educational potential or the discovery of something that can inform design or development, then as academics we have a responsibility to explain why that game is suitable for our purpose.
One compelling reason for putting thoughtful effort into justifying the choice of a game used in a study is that it helps to make the study itself more credible. This has implications for the increased acceptance of game studies academically as well as for helping to improve relations between academia and the games industry. In a recent article offering suggestions for how the Academy could build stronger ties with the Games Industry, John Hopson argues that we should “(u)se examples from bestsellers. A good example from a popular game is more effective than a great example from something they’ve never heard of. Industry people often suffer from an ‘if-they’re so-smart-, why-ain’t-they-rich’ attitude towards smaller titles. Even if the small title is a perfect example of how the theory works, they’re going to be less likely to listen if they haven’t heard of the game ahead of time. Commercial success is one way of making sure that the audience will respect your examples, but you can also use titles that are well known or critically acclaimed but which weren’t necessarily huge blockbusters. It’s also important to keep your examples as current as possible, because many industry folks will see a three-year-old example as ancient history” (Hopson, 2006).
The field of game studies and game education includes a unique combination of academic and industry literature rarely seen in other disciplines. This synergy has many advantages but also causes some difficulties, especially in academic circles. In games, much useful information and insight comes from non-academic sources that are not normally subject to the kinds of peer review to which we in the Academy are accustomed and thus are not typically recognized as carrying the same weight as those that come from more traditional peer-reviewed sources. One way to help address this discrepancy is to ensure that the defensibility of the academic studies can stand up to close scrutiny, and one way to ensure that is to make sure that the specific games chosen for study can be objectively justified as meeting the goals of the study.
While the field of game studies is rapidly gaining momentum with more journals and conferences being offered every year, it is still in the process of building academic credibility and rigor, and defensibility of research as well as methodology remains critical.
Based on a cursory examination of over 1000 recent scholarly publications that mention ‘digital games’ it was determined that the games described there can be broadly categorized into three groups:
- Non-commercial games that have been designed or developed by the authors,
- Non- commercial games (including but not limited to serious games) that have not been designed or developed by the authors, and
- Commercial titles.
The classification scheme proposed in the following sections can be applied to studies involving any of these three broad groups, but it is the last group, namely studies involving commercial titles that are of interest here. For the purposes of this article, the last group includes commercial games designed or developed by the authors of the publication. A case in point is that of studies of violence in games, and highlights the importance of careful justification of the choices of games to verify their “fitness for purpose”. One of the criticisms of many ‘media-effects’ studies is that these studies commonly include a broad variety of games and treat them as though they are essentially interchangeable (Freedman, 2001). In a longitudinal study of violence in an online videogame, Williams and Skorik raised questions about the generalizability of games which have implications far beyond their own study. “The online database www.allgame.com lists descriptions of more than 38,000 different games across 100 platforms. To collapse this wide variety of content into a variable labeled ‘game play’ is the equivalent of assuming that all television, radio, or motion picture use is the same” (Williams & Skoric, 2005). As Dill and Dill (1998) have noted, “This is akin to lumping flms like The Little Mermaid with Pulp Fiction, and expecting this combined ‘movie viewing’ variable to predict increases in aggressive behavior” (1998, p. 423).
By not providing careful rationales for our game choices, we are not paying suffcient attention to the great variety of games available, and in doing so we risk nullifying any results that come from such studies. The existence of such a large number of games means that we cannot assume that one game is as suitable as any other for the purposes of study. In other words we cannot collapse all ‘adventure games’ into one category and assume that what we discover about one adventure game will apply to some other adventure game. Both Grim Fandango and God of War are listed as adventure games by Mobygames.com and both rank among the top 10 in that genre, yet it would be inappropriate to study the design of one for insights into why the other is successful. Similarly, studying ONE game does not necessarily allow us to generalize our findings to any other games. This not only has implications for results involving the players, but also for any conclusions we attempt to draw about the game’s design. Without careful qualification of the choice of game for a given study there can be no generalization to other games.
While a suggestion to force all games researchers to apply some sort of ‘scientific’ approach to their choice of games is clearly unreasonable, paying closer attention to how we choose games and making a point of explaining those choices can certainly help address legitimate questions about a game’s fitness for purpose in the context of a given study.
We may not feel the need to justify choosing Shakespeare’s Hamlet if our purpose is to make some generalization about tragedies, but video games are not in the same category as classic literature. Games have not yet attained the level of acceptance, nor of unarguable classification that classic literature has and until they do we should still be explaining our decisions – especially if we hope to realize some findings that could be applied to some other game or some other population. On the other hand, providing defensible rationales for game choices does not preclude the possibility of choosing a game because it is one we personally like, but we still need to address how that makes that game a worthy candidate for study. If we choose a game because it is popular, then we should be able to support that with facts or citations that can stand up to scrutiny. Further, we should be able to explain why a game’s popularity is germane to our study. Similarly, if we claim that a specific game is representative of an entire genre, then it is reasonable to back that claim with further references.
META-ANALYSIS OF GAME STUDIES
Since the question of how games are selected by researchers has not previously been examined the author conducted a qualitative meta-analysis (Delgado-Rodríguez, 2001) of what methods researchers reported using in choosing games for study. Papers and reports published over a five year period between 2003 and 2008 were examined with the goal of determining the reporting frequency of explanations of game subject choices.
Since there is no way to verify that a lack of information about the selection criteria applied to the choices of games indicated that none existed, it should be noted that a lack of explanation in the publication does not prove a lack of consideration for the study. It is certainly possible that carefully considered reasons motivated the game choices in many of the studies presented here, but that these were simply not included in the publication. The worthiness of the choice that was made was also not being examined here, and indeed many well-known game scholars were included in the list of papers examined. In many cases there would be little controversy over the claim that the chosen game has the characteristics described in the paper. In some cases there would also be no dispute that the particular type of game is a suitable choice (and perhaps even the most suitable choice) for the study as reported. Many of the reports have contributed to the body of knowledge in games studies in important and significant ways. The concern has to do with verification.
In order to report as widely as possible and since this study sought to discover why researchers chose the games they did, multiple publications by the same authors using the same game were avoided, unless that game was being used for a different purpose in each study. Two separate analyses were performed and the data were combined. In the first, a variety papers from scholarly publications released between 2003 and 2006 that reported on research involving at most five distinct games were chosen. In the second papers from 2006 that had missed the cut-off date from the first analysis through papers published in 2008 were included. Both sets of papers were examined to discover which games were chosen and whether an explanation or justification of the game choice was included. Studies featuring games like DDR (Dance, Dance, Revolution) were not included in the first analysis since there were so few commercial kinetic games available at the time, the rationale is understood UNLESS the study was looking at some aspect of the game other than its interface. Similarly, studies that focused on a characteristic aspect of a specific game (such as effectiveness of recruitment in America’s Army) were also excluded. A distinction was made in the meta-analysis between the description of the game (including gameplay and any noteworthy features of the game) and a rationale for the choice of the game. Virtually all studies described the games that were being used but these descriptions were rarely connected with the reasons for choosing that game.
In order to be able to classify studies in a manner that would permit some kind of comparison it was necessary to group the studies and the following classification was devised for this purpose. All papers were examined to determine the purpose of the study and five groups were identified:
- Specific studies were ones where a specific game is used that had no identifiable substitute. In other words, if the questions were asked, “Could some other game have been used?” the answer would have been no. An example of this is Squire’s early study using Civ. III (Squire, 2003). It could be argued that at the time, there really was no other game that could have been used in this study.
- Typical studies were ones where the game was claimed to be a representative example, such as an MMO, or that it supported an in-game economy that has a real world value. In some cases, the reason given was far more nebulous, such as that it was “interesting”.
- Apparatus studies were studies where the game was used as an apparatus rather than the focus and the object of the study was something else, such as the game was being used as the basis of a writing assignment, and it was the writing assignment rather than the game that was being studied.
- Mod studies were ones that made use of some commercial game or engine but where the study focused on the mod rather than the original game.
- Other: this was the ‘catch-all’ category for studies that could not be placed into one of the other groups.
The meta-analysis included 89 papers that were examined in detail. 131 games were identified comprising 93 distinct titles (some studies used more than one game but numerous studies used the same games such as World of Warcraft). Three quarters of the studies reviewed indicated that the game(s) were in some way representative. While most authors made some attempt to explain their choice of game, most of those explanations were effectively unsubstantiated opinions such as claiming that the game is highly successful or popular. The claims are rarely supported with other data. Only one paper out of the 89 examined reported having applied some systematic technique to identifying candidate games for study. Most of the papers examined included a synopsis of the game being studied, but only about 30% offered an explanation for how or why this game met the needs of the study, and fewer still (9%) supported that explanation with citations. For example, popularity is given as a rationale in 3% of reports, but none explain how the game’s popularity is germane to the study. Some included explanations that are either difficult to verify or substantiate in any objective way, such as that the game is interesting. Obvious rationales such as “we needed an MMO and this game is one” still beg the question, “Why THAT MMO?” In a third of the studies the explanation for the choice of game included a statement of a requirement for the study. For example, the game needed a strong story-telling component or an ability to play the game from multiple perspectives. In most cases nothing is offered to support claims that the chosen game meets the specified criteria other than the authors’ assertion that it does.
While there may not be much controversy over claiming that Midtown Races is a driving game and that a driving game is needed for a study on fear of driving after an automobile accident (Walshe, Lewis, Kim, O’Sullivan, & Wiederhold, 2003) it may still be appropriate to ask in what way this particular driving game ft the need. Other claims, such as choosing Doom because it is easily “mistaken” for violent (Molesworth, 2007) deserve objective support. Only one study described a rationale for the exclusion of one or more games from study (Warnes, 2005) and one other report actually explained the methodology used to select the game for the study. Henderson (Henderson, 2005) allowed the study participants to vote on a game, citing prior research that suggested participant interest was an important factor in the study’s success. It is suspected that for many several of the studies many game choices were, at least in part opportunistic, as the researchers had access to or were already playing this game. Only one researcher actually stated that they were already playing the game as their explanation for choosing it (Chen, 2005). In three other cases, the researcher states that they have prior experience with the game but it is not made clear whether the study began before or after that individual began to play that game, nor how much influence the author’s own game playing preferences had on the choice. Comments such as, “I’ve been playing this game for years” places game studies in a somewhat unique position as both casual and avid gamers draw on their own playing experiences to help inform their studies. This kind of connection places many game studies in the realm of what Glesne has called “Backyard Research” which can make separating researcher roles from pre-existing ones complicated and difficult (Glesne, 1999). Support for claims about the game’s popularity does not necessarily speak to a game’s fitness for purpose, and yet one in ten of the studies used this as their primary justification for choosing the game. In other words this MMO was chosen because it is popular when the focus of the study itself was something other than popularity.
If the study is looking at a game’s economy or some aspect of its interface design, then the fact that the game was popular does not necessarily imply that it is a representative choice. In the author’s own work examining player learning support in highly successful commercial games, several of the top choices had to be eliminated because they did not offer sufficient support (Becker, 2008). In other words justifying the choice based on the notion that the game’s popularity implies some requisite level of quality does not always work.
One of the things that many aspiring researchers learn early in their careers is that they must support the claims they make in their writings. A common note written by supervisors in the margins of almost every thesis draft states something to the effect of: ”Says Who?”, or “Where does it say that?” We are often told that if we cannot support a claim then we should not make it. Should we expect less of studies that use games? The results of the combined meta-analysis indicate that a very small minority of game researchers currently report on the methodology used for the choice of a game in a study, or use examples of excluded games to support their choices. Very few explain how or why their stated game requirements support the goal of the study. While some cite references to support at least some of their claims about why this kind of game is needed for this study, almost none cite any references supporting their claim that the chosen game actually meets those requirements.
If a claim is made that a particular game was chosen because of its open-ended gameplay, it would be helpful to cite other sources that agree with this claim. By far the most common attribute supported by other references is the claim about the game’s popularity and the most common outside reference is to sales figures. Simply stating that a first person shooter was needed for this study is no longer sufficient to justify the choice of genre – it should be possible to explain what qualities of this genre are important. Was the particular game chosen because it was a representative example or because it was the best or worst example? What evidence is there to support these claims? Fitness for purpose “equates quality with the fulfillment of a specification or stated outcomes” (Harvey, 2004). If a researcher claims that a particular game is an appropriate choice for a particular study then it is appropriate to offer justification for that claim. Given the great number of games available, it is no longer sufficient to claim that a particular game meets certain criteria without supporting that claim in a verifiable way.
Even though critical and commercial success are both recognizable and accepted measures of a game’s popularity, and popularity in turn gives some indication of that game’s perceived quality as judged by players, developers, and game critics, these are also highly subjective measures.
Katrin Becker, PhD, Adjunct, School of Interactive Arts and Technology Simon Fraser University, Surrey BC
Becker, K. (2008). The Invention of Good Games: Understanding Learning Design in Commercial Video Games. Unpublished Ph.D. Thesis, University of Calgary, Calgary, Canada.
Chen, M. (2005, June 16-20, 2005). Addressing Social Dilemmas and Fostering Cooperation through Computer Games. Paper presented at the DiGRA 2005 2nd International Conference, “Changing Views: Worlds in Play”, Vancouver, B.C.
Delgado-Rodríguez, M. (2001). Glossary on meta-analysis. Journal of Epidemiology and Community Health, 55, 534-536.
Dill, K., & Dill, J. (1998). Video game violence: A review of the empirical literature. Aggression and Violent Behavior, 3, 407-428.
Freedman, J. L. (2001). Evaluating the Research on Violent Video Games, Playing Games By the Rules The Cultural Policy Challenges of Video Games. Chicago IL.
Glesne, C. (1999). Becoming qualitative researchers : an introduction (2nd ed.). New York: Longman.
Harvey, L. (2004). Analytic Quality Glossary. Quality Research International. Retrieved from http://www.qualityresearchinternational.com/glossary/
Henderson, L. (2005). Video Games: A Signifcant Cognitive Artifact of Contemporary Youth Culture. Paper presented at the Digital Games Research Association Conference (DiGRA 2005), Vancouver, BC.
Hopson, J. (2006). We’re Not Listening: An Open Letter to Academic Game Researchers Gamasutra. Retrieved from http://gamasutra.com/features/20061110/hopson_01.shtml
Molesworth, M. (2007). Monsters and the Mall: Videogames and the Scopic Regimes of Shopping. Paper presented at The Third Digital Games Research Association International Conference (DiGRA 2007) “Situated Play”, Tokyo, Japan.
Squire, K. (2003). Replaying History: Learning World History through playing Civilization III. Thesis, Indiana University.
Stake, R. E. (1995). The art of case study research. Thousand Oaks: Sage Publications.
Walshe, D. G., Lewis, E. J., Kim, S. I., O’Sullivan, K., & Wiederhold, B. K. (2003). Exploring the Use of Computer Games and Virtual Reality in Exposure Therapy for Fear of Driving Following a Motor Vehicle Accident. [Article]. CyberPsychology & Behavior, 6(3), 329-334.
Warnes, C. (2005, June 16-20, 2005). Baldur’s Gate and History: Race and Alignment in Digital Role Playing Games. Paper presented at the DiGRA 2005 2nd International Conference, “Changing Views: Worlds in Play”, Vancouver, B.C.
Williams, D. (2006). A (Brief) Social History of Video Games. In P. Vorderer & J. Bryant (Eds.), Playing Computer Games: Motives, Responses, and Consequences. Mahwah, NJ: Lawrence Erlbaum.
Williams, D., & Skoric, M. (2005). Internet Fantasy Violence: A Test of Aggression in an Online Game. Communication Monographs, 72(2), 217-233.
Wolf, M. J. P. (2001). The medium of the video game (1st ed.). Austin: University of Texas Press.