By David I. SchwartzThis paper describes a preliminary approach to teaching students about game design and development by making a custom game controller in a single course. RIT’s first seminar in alternative game controller design introduced students to game input mechanisms and encouraged innovation. By working with hardware, students explored physical design issues to improve their overall game design skills. The course demonstrated early success, as evidenced by the 2009 Game Developers Conference in which a banjo controller and game created by the students garnered accolades. To help other instructors develop a similar course, this paper addresses the motivation for, and components, of the course design. By combining elements of reverse engineering, interfaces, and game design, students can indeed design and successfully make custom game controllers. The paper contains a summary of one of the two final student projects: a deformable surface. The postmortem notes that despite the course success, future versions will need more time for projects. The paper concludes with a discussion of future directions.
Game Design and Development Education
Not so long ago, the idea of forming a unique academic field devoted to games seemed folly, and to some, even heresy. However, a variety of fields, programs, and courses spanning game studies, design, development, and others have blossomed . A flurry of academic papers has explored defining/refining course topics with recent works exploring connections with education, simulation, and more, e.g., [2, 3]. Theory and concepts continue to emerge, sometimes supplanting industry focus, resembling similar “splits” seen in the sciences and engineering between theory and application. With students trying to enter the competitive game industry, academic programs that focus on skill building and creativity can retain their popularity. To stand out, students need new ways to explore game design and development.
The game input device, or game controller has a fascinating history mostly documented in a variety of websites, academically it has only been examined in some papers, and very few books [4-9].
The controller occupies a paradoxical position in computer game studies. Although it is central to gameplay experience—it marks physically the difference between play with a game and merely watching a screen—it goes largely unreflected on by gamers and in gaming literature. 
But, game controllers have roots as a variety of mechanical input systems and interfaces, like knobs and buttons. Console controllers essentially evolved with ergonomic improvements and additional functionality .
With respect to game input and player control, modern controllers involve a combination of abstraction and mapping . For example, consider how games frequently assign the A-button to jump . Although press-and-release simulates a vertical action, jumping does not necessarily imply crouch (the press) and jump (the release). Thus, a button infers an abstraction, triggering a loosely connected game action. On the other hand, a mapping (or natural mapping) implies a direct analog from controller to action. For example, pushing upwards on an analog stick would directly move an avatar up on the output device—even with axis inversion, the action still maps logically, albeit in the opposite direction.
Alternative Game Controllers
Although the term alternative game controller seems relatively accepted, alternative implies that a regular game controller exists [13-16]. One could contend that the PS3/Xbox360 style controllers (and recent predecessors) are standard. Each new console seems to alter previous controller designs, sometimes starting anew, as with the Wii. In fact, a term to signify a standard seems elusive, though some suggest “pronged.” Still, a definition of standard schemes has emerged:
- Console games: directional pad, analog stick(s), shoulder buttons, face buttons.
- PC games: keyboard and mouse.
Thus, an alternative controller deviates from these common components and forms, tending to provide greater (or different) mapping fidelity. For example, numerous steering wheels and pedals have facilitated driving games. Dance pads are another early successful example, though other genres, like RPGs, can use the pads . Two modern examples have produced tremendous attention for alternative input: Guitar Hero (and successive music/rhythm games with their controllers) and motion control via Wii, Kinect, and Playstation Move.
Cost is the greatest deterrent to alternative controllers, ranging from development to manufacturing to consumer. Until recently, alternative controllers seemed a niche market, catering to specific genre, like racing. However, the advent of motion control and interest in “exergaming” has led to a variety of products, as listed above. As seen below, the question of acceptance of controller seems to tie to how it facilitates new gameplay, creating new experiences that more easily map to natural motion.
Game design and development students can take risks that industry might avoid, leading to innovation. By exploring new ideas without commercial pressure, many student projects have helped introduce very influential games, e.g., Portal and others. Likewise, motion control has spurred on much of the recent interest in alternative controllers. For example, numerous “cracks” of the Wii Remote have shown that many students and people outside of academia have already tackled the challenge of adapting motion-based game controllers [18, 19]. In fact, Jonny Lee’s WiiMote Head Tracking project predates recent advances in new motion controllers and seems to continue to inspire students .
Given the recent popularity in alternative game controllers, game companies continue to innovate their controllers. Many research projects explore design innovations, often involving areas of tangible interaction and haptics. For example, one attempt to recreate the EyeToy for PC  foreshadows later development with Microsoft’s Kinect. In fact, one of the student projects in the experimental course used an EyeToy (Section 3). Other innovations include touch, especially for magnification  and body movement —the innovations are ongoing.
Alternative controllers offer excellent opportunities to explore game design innovation. Students can either refine interfaces to facilitate established genres or discover new kinds of games. Addressing a common complaint of limited appeal, students might expand the range of games that the alternative controllers serve. Ultimately, students can gain greater understanding of the user experience by designing for players.
Providing a hardware experience can also expose software-oriented students to more low-level techniques, “looking under the hood,” so to speak. Greater experience with drivers, C and assembly programming, signal processing, USB, networking, and other technology can enhance a student’s resume, especially those looking for development positions. For those looking outside of the game industry, knowing how to analyze and design a controller will provide even greater opportunities, e.g., consumer electronics, medical, manufacturing, robotics, and military.
This work can also provide excellent multidisciplinary opportunities. Apart from game design and development fields, entertainment engineering and entertainment technology continue to grow . What better way to gather a variety of students than with a game controller, which embodies a mechanical, electrical, software, design, and business product? Since the game industry now uses a variety of controllers, all of these students can greatly benefit from learning to work with each other, mirroring industry practice. With excellent technical abilities, inspiring engineers to consider games in later courses could boost the variety of prospective graduate students.
2. Course Design
In RIT’s Winter quarter (December 2008-March 2009), the course ran as a preliminary, experimental seminar with eleven students. The exploration posed these questions:
- What kind of new game controller would facilitate a new game experience for which that controller most logically abstracts and maps player input?
- Which game controllers can students most easily dissect, analyze, modify, and use?
- Which sensing technologies (e.g., accelerometers, gyroscopes, and other devices for embedding computing and sensing) most easily connect to game controllers?
- Can teams of students appropriately scope their ideas?
The interdependent problem of game and controller design defined the core challenge for students to overcome.
Related Work and Materials
Engineering courses have long since explored notions of embedded systems, mechanisms, especially for robotics, manufacturing, and ergonomics, which frequently involve game systems and controls as educational examples [25-27]. Other relevant areas focus on human factors and ergonomics, which especially connect to how people use devices and interact with physical environments . Game controller innovation also occurs outside of academia, tapping into an example of citizen science . An active community of independent developers (Makers) makes custom game consoles and controllers [30-33]. Sources, like Instructables  and Make , collect and distribute many community-driven projects.
The course drew inspiration from engineering labs, requiring physical supplies as with reverse engineering courses [35-37]. In these courses, students disassemble and reassemble products to learn how they work and improve upon the original designs. For example, the ActionXL motion controller provided early inspiration to explore accelerometers . In this spirit, “hacking” references (e.g., [39, 40] inspire similar reverse engineering in games and game technology. Consequently, older/used controllers, like the NES, proved especially popular with the students. For example, students investigated pin layouts and input signals [31, 32], to gain the necessary background.
To address the course outcomes, the project presented a three-fold design challenge to all of the students:
- Game: Are there game mechanics that a current controller does not naturally facilitate?
- Controller: What unique input device would serve those mechanics?
- Interface: How will the controller work?
All three design components involved interdependent thinking–a deliberately challenging problem requiring significant cycles of iteration. At a minimum, students needed to adhere to the following constraints for the controller:
- Uniqueness: It must not already exist;
- Interaction: It must be good for a particular type of game/primary mechanic;
- Reality: It could be constructed of off-the-shelf parts, including other controllers;
- Platform: It must connect to a PC;
- Durability: It must last during an all-day institute showcase;
and constraints for the game:
- Gameplay: It must exhibit as much interaction as an early game prototype;
- Length: It must have at least one level;
- Interaction: It must exhibit a genre/primary mechanic that the controller facilitates;
- Platform: It must play on a PC;
- Elements: It did not need to have narrative, plot, or other elements except when required to demonstrate the controller.
For the interface, the students needed to ensure deliberate and useful abstractions and mappings to the controller. Due to existing game design courses and experiences, the students could start with controller design, though the game required roughly parallel development. Ultimately, the project still involved providing compelling gameplay, which emphasized that the controller design must somehow serve the game design.
Thus, the class split roughly into two groups around specific projects: a banjo controller  and a surface controller. Student final reports, press releases, and pictures are all available . This section describes the “spandex garbage can”: a tactile, multiplayer, force-feedback, 3-D surface controller, as shown in Figure 1.
Figure 1. The “garbage can” controller
What’s inside? Starting from the notion of interacting with a visual environment and faced with the need to make something physical, the students suggested a fabric-based controller. Through a series of experiments and more brainstorming, the students used an intriguing idea: place a light source inside a dark, opaque rubbed trash can with a camera–an EyeToy, in fact. After melting the first barrel with holiday lights, the students opted for a “strip light” and a piece of diffuser used in ceilings, as shown in Figure 2.
Figure 2. Light source in garbage can
How does the controller actually work? Wherever a player touches the spandex, the light underneath becomes essentially more intense, creating “bumps” that the camera registers. The GPU can very quickly render the height of each section of the surface. As noted by the students, although the edges do distort, the majority of the surface response is quite accurate.
Figure 3. Rendering the heightmap
The team essentially split into controller and game sub-teams to maximize the dwindling project time. The game team opted to exhibit the multiplayer and 3-D aspects via Sand Havens, a casual RTS game (Figure 2). Players push down on the surface to raise land so that bad creatures avoid eating good creatures. But since the game conserves mass, raising land simultaneously lowers other areas. Lands that sink below sea level fill with water and threaten the good creatures.
Figure 4. Sand Havens, played on the garbage can controller
What Went Right
A number of factors in the course design and a certain degree of luck of student skill sets greatly contributed to the course success:
- Early analysis: By requiring students to do a quick prototype of a game involving an old controller, the class generated a number of ideas and introduced ways to innovate.
- Individual pitches: With such a large, interdependent design “space” (games, controllers, interfaces), students needed a way to find “common ground.”
- Guest lectures: Bringing in an electrical engineering professor greatly helped explain a variety of electronics, especially the mouse, shows the detail in what students may consider mundane.
- Group work: Game design and development programs have relied on team building and group dynamics. With an extremely complex design problem, students needed support. Besides splitting work, the mix of students across fields helped to create mentoring.
- Interdependency: Students can resolve an interdependent hardware/software/interface design problem in a variety of approaches. Once they resolved either the core gameplay (“play Banjo Hero”) or interaction (“touch a 3-D surface”), the teams could then more quickly iterate on the remaining aspects.
- Hardware: As originally planned, by providing a hardware experience, students learned about an entirely new aspect of game design. Students found NES, Atari, Sega Genesis, and other older game controllers relatively easy to dissect and study.
- Cost: Each controller ran the student teams about $60, though the banjo ultimately got a nice strap (and still needs a case).
- Motivation: The students insisted on building an actual controller despite the choice to simulate inputs.
- Feasibility: Time and resources greatly worried everyone. However, both teams successfully created games and controllers within seven weeks due to some students having prior hardware experience.
What Went Wrong
However, as with any first run of a course, there were various surprises, synergies, and problems:
Time: The final project should have started from the beginning, especially to focus the design of the game and controller.
Faculty: More multidisciplinary support would have solidified the topic flow.
Space: Via multidisciplinary effort, faculty connections might help with lab issues.
Milestones: Because any aspect of design could become entangled or hang, not all groups had sufficient balancing.
Bulk/Quantity: Without copies or smaller prototypes, choosing who should own the final controller created problems. Ideally, each student should have something to demonstrate.
5. Future Work
The initial run of the course showed promise and demonstrated that students can indeed make original and interesting alternative game controllers. However, given the frenetic pace and design complexity, future versions require more time, perhaps with one or two weeks of starting the term. A true multidisciplinary effort would greatly broaden the scope, help flesh out missing material, and improve student mentoring. Connecting with researchers in HCI, physical computing, haptics, and a variety of related fields would also improve the educational direction–and even produce students for their research. Such work has already commented with an Augmented Reality Golf project .
Another open area involves the availability of peer-reviewed technical schematics and explanations to formalize the numerous websites created by Makers who seem to have adapted their own system of peer review. Although the problem is more about pedagogy instead of theoretical research, educators who synthesize this material would provide a tremendous resource (and oversight) for a multitude of students. Because of the close interaction with a variety of engineering, computation, and design fields, a properly evaluated source would serve many courses.
Subsequent courses can build upon a common framework of controller design, perhaps forming an accepted track within many game development programs. Future efforts could include social networking and augmented reality, tapping into networked and multiplayer applications. Many avenues await innovation in game controllers.
I am deeply grateful for all the hard work, initiative, and motivation of all my alternative game controller students: Dominck D’Aniello, Sela Davis, Ben DeLillo, Mike Ey, Alex Lifschitz, Aury McClain, Tom Merrill, Joe Pietruch, Anthony Reese, Michael Tangolics, and Andy Zickler. Jay O’Leary and Ben Kalb ActionXL were extremely generous in donating many controllers (and other components) for this class and numerous other projects. My former department of Information Technology at RIT provided faculty development funds, which paid for all of the used controllers we disassembled. I am also grateful for three summers’ worth of visiting research faculty fellowships at the Air Force Research Lab in Rome, NY (AFLR/RI) with Dave Ross and Alex Sarnacki–as part of that work, we investigated aspects of citizen science, which I applied to this course work. I also appreciate “OCAL’s” public-domain joystick from http://www.clker.com/clipart-13249.html. Finally, I would like to thank the reviewers for providing great suggestions for improving this paper.
1. Game Career Guide, http://www.gamecareerguide.com, accessed December 1, 2010.
2. Foundations of Digital Games, http://www.foundationsofdigitalgames.org, accessed December 1, 2010.
3. SIGGRAPH: Sandbox Symposium, http://www.siggraph.org/s2009/galleries_experiences/sandbox, accessed December 1, 2010.
4. Adams, E. A. (2010). Fundamentals of Game Design, Second Edition, New Riders, Ch. 8.
5. Sock Master's Game Console Controller Family Tree, http://www.axess.com/twilight/console, accessed December 1, 2010.
6. Cummings, A. H. (2007). The Evolution of Game Controllers and Control Schemes and their Effect on their Games, in The 17th Annual University of Southhampton Multimedia Systems Conference, http://users.ecs.soton.ac.uk/ahc08r/mms.pdf, accessed December 1, 2010.
7. Freundorfer, S. and W, Forster (2003). Joysticks: Eine illustrierte Geschichte der Game-Controller 1972-2004, GAMEPlan. See http://www.vesalia.de/e_joysticks.htm (currently available only in German).
8. The 11 Most Groundbreaking Controllers of All Time, http://www.gamepro.com/article/features/65948/the-11-most-groundbreaking..., accessed December 1, 2010.
9. Marshall, D., T. Ward, S. McLoone (2006 ). From Chasing Dots to Reading Minds: The Past, Present, and Future of Video Game Interaction, ACM Crossroads, Winter 2006-13.2, http://www.acm.org/crossroads/xrds13-2/interaction.html, accessed December 1, 2010.
10. Kirkpatrick, G. (2008), Controller, Hand, Screen: Aesthetic Form in the Computer Game, Games and Culture 2009, 4 (127), DOI: 10.1177/1555412008325484, http://gac.sagepub. com/cgi/content/abstract/4/2/127.
11. Lander, R., A. Reid, M. Khasawneh (2006). Impact of ergonomics on game-pad design: Qualitative and quantitative analysis, in 2006 IIE Annual Conference and Exposition.
12. Smith, T. (2009). Press A To Jump, Game Developer, 16 (9).
13. Blaine, T. (2005). The convergence of alternate controllers and musical interfaces in interactive entertainment, Proceedings of the 2005 conference on new interfaces for musical expression, National University of Singapore, pp. 27–33.
14. Sall, A. & R. E. Grinter (2007). Let’s Get Physical! In, Out and Around the Gaming Circle of Physical Gaming at Home, Computer Supported Cooperative Work, Kluwer Academic Publishers, pp. 199–229.
15. Alternative Game Controllers, http://www.gamasutra.com/features/20040618/ lloyd_01. shtml, accessed December 1, 2010.
16. Funny Suggestions for Alternative Game Controllers, http://www.boingboing.net/2006/07/05/funny-suggestions-fo.html, accessed December 1, 2010.
17. Winds of Orbis, http://www.etc.cmu.edu/projects/wiixercise, accessed December 1, 2010.
18. Wiimote Project, http://www.wiimoteproject.com, accessed December 1, 2010.
19. Wong, E. K., W. Y. F. Yuen, C. S. T. Choy (2008). Designing Wii Controller - A Powerful Musical Instrument In An Interactive Music Performance System, International Conference on Mobile Computing and Multimedia, Proceedings of the 6th International Conference on Advances in Mobile Computing and Multimedia.
20. Johnny Chung Lee > Projects > Wii, http://johnnylee.net/projects/wii, accessed December 1, 2010.
21. Punchihewa, A., Tomlines, S., De Silva, L. (2009). Vision Based Game Controller Interface,
Tactful Interaction, Proceedings of the 4th International Conference on Autonomous Robots and Agents, IEEE, pp. 261-266.
22. Kolaji, R., et al. (2009). Magnifying Glass-Like User Interface for Game Control, Games Innovations Conference, ICE-GIC 2009.
23. Bianchi-Berhouze, N., W. W. Kim, D. Patel (2007). Does Body Movement Engage You More in Digital Game Play? and Why?, ACII '07: Proceedings of the 2nd international conference on Affective Computing and Intelligent Interaction.
24. Grose, T. K. (2005). The Science of Fun, PRISM, 14 (5), ASEE, http://www.prism-magazine.org/jan05/tt_science.cfm.
25. Cornell University, ECE 4760: Designing with Microcontrollers, Final Projects,
http://instruct1.cit.cornell.edu/courses/ee476/FinalProjects, accessed December 1, 2010.
26. Koopman, et al. (2005). Undergraduate embedded system education at Carnegie Mellon, ACM Transactions on Embedded Computing Systems, 4 (3), ACM, pp. 500–528.
27. Mechatronics Courses, http://education-portal.com/mechatronics_courses.html , accessed December 1, 2010.
28. Jørgensen, A. H. (2003). Marrying HCI/Usability and computer games: a preliminary look, ACM Proceedings of the third Nordic conference on Human-computer interaction, ACM, pp. 393–396.
29. Citizen Science, http://en.wikipedia.org/wiki/Citizen_science, accessed December 1, 2010.
30. Technology Blog: Hacking game controllers and 3D Wii, http://www.guardian.co.uk/technology/blog/2008/may/06/hackinggamecontrol..., accessed December 1, 2010.
31. Random Function: Games Controller Hacking Guide, http://www.randomfunction.com/?q=node/66, accessed December 1, 2010.
32. Controller Theory, http://www.gamesx.com/misctech/controltech.htm, accessed December 1, 2010.
33. Instructables, http://www.instructables.com, accessed December 1, 2010.
34. Make: technology on your own time, http://blog.makezine.com, accessed December 1, 2010.
35. Raja, V. and K. J. Fernandes (eds) (2008). Reverse Engineering: an Industrial Perspective, Springer, Ch. 1.
36. Wu, C. (2008). Some Disassembly Required, PRISM, Oct. 2008, ASEE, http://www.prism-magazine.org/oct08/tt_01.cfm.
37. Schwartz, D. I. and J. D. Bayliss (2010). The Ethics of Reverse Engineering of Game
Technology, Ethics and Game Design: Volume Two, K. Schrier (ed), IGI Global.
38. Motion Controller, http://www.actionxl.com/products.html, accessed December 1, 2010.
39. Heckendon, B. (2005). Hacking Video Game Consoles: Turn your old video game systems into awesome new portables, Wiley.
40. Carless, S. (2004). Gaming Hacks: 100 Industrial-Strength Tips & Tools, O’Reilly.
41. Ey, M., Pietruch, J., an D. I. Schwartz (2010). “Oh-No! Banjo”–A Case Student in Alternative
Game Controllers, in Proceedings of Future Play 2010, ACM.
42. David I. Schwartz, Alternative Game Controllers seminar reports, articles, and pictures, http://igm.rit.edu/~dis, accessed December 1, 2010.