Augmented Reality Serious Games

Article, Katelyn Procci 2 Comments

Katelyn Procci | April 11, 2012

Augmented reality (AR) involves somehow supplementing the real world, often by overlaying virtually-generated information and animations onto real-time video of the actual environment. AR using head-mounted displays has long-been implemented1. In recent years, however, the technology required for augmented reality (cameras, processing power, video cards, and displays) has become less expensive, less cumbersome, and more widespread. Because of this, we’ve been seeing more augmented reality in commercial products, like games and toys2. Let’s face it, AR is pretty cool. Whether or not it is just a fad has yet to be seen, but the sheer number of applications for augmented reality is astounding (and exciting), and without a doubt AR has the potential to improve applications in several domains, such as for the Department of Defense, in training and education, and for entertainment.

Google recently revealed a new prototype video for Project Glass. Basically, this project focuses on wearable computing and augmented reality. An individual who wears these glasses can access messages, email, check into locations, take pictures, and use google for pretty much anything (general googling, google maps for directions, etc.).

the Project Glass experience

More than ever, gaming hardware supports augmented reality. The PlayStation utilizes a camera for their augmented reality games (Eye of Judgement, EyePet) as does the Kinect for Xbox 360 (Fantastic Pets, to follow the virtual pet theme). The Nintendo 3DSi, which uses two screens to create a stereoscopic 3D effect and multiple cameras to support augmented reality gaming, comes packaged with AR games and cards. The PlayStation Vita also features cameras and has several free AR apps that can be downloaded to the system. Smartphones, which now make up about half of all mobile phones in the US, also have the hardware support needed to create augmented reality experiences — check out these interesting smartphone apps and games that use AR.

In very broad strokes, AR works by first scanning in some image. Mostly cards with special codes are used, but some AR platforms are capable of scanning entire objects. Next, that image is recognized and matched to entry in a database. Then, based on the programming, something happens: Text, images, video, or whatever assets are linked to that image pop-up on the screen, overlaid on the actual environment as captured by the cameras. Depending on the software and hardware, the individual may be able to interact with those assets.

Given their widespread availability and (relatively) inexpensive price tags, handheld gaming devices and mobile phones are popular mediums for serious games3,4,5, and we know that they are now capable of supporting augmented reality. Squire & Klopfer6 list some reasons as to why augmented reality is extremely beneficial and has the potential to be extremely effective when implemented in serious games:

  • AR is portable and accessible, which is especially relevant seeing as how most gaming hardware supports AR and the prevalence of smartphone use
  • AR is social and encourages active participation
  • AR allows for situated learning, which is where students learn in the relevant context and see how changes actually affect the system of interest, which is extremely important for learning science

For the serious game designer interested in creating AR serious games, it’s important to remember that we usually have to develop on the cheap. So what tools exist? You can look into the Kinect SDK, or look at Unity, which is a flexible development platform that will let you compile your game for consoles (if you have the right license). Your best bet, as a small serious games development team on a tight budget, is developing small app-based games for smartphones.

If you want to get started making apps, you should first check out the MIT App Inventor. It is a WYSIWYG, browser-based app development platform for Android phones. Originally Google’s project, this allows you to begin playing around with app creation. It is still in Beta, but it is definitely worth a look.

More specific to AR, there is Vuforia, Qualcomm’s augmented reality platform. It is used by smartphone and tablet app developers to create AR experiences. The SDK is available for download, so if you are interested in learning how to develop AR games for smartphones, this is a great place to start. Check out the video below to see an example of Vuforia in action.

Vuforia: Sesame Street AR Dolls

Another exciting platform is Aurasma, which is a free app that you can download. It scans in entire images and then launches assets tied to that trigger. You, as a user, can tag images in the environment (creating auras) and add your own content. Aurasma lets you put in video, images, 3D objects, and, important to us, allows you to use objects in the environment for games.

From CNET: Aurasma

These are advanced features created using the Aurasma Developer Studio, which does cost money, but if a serious game designer is interested in, say, making a museum display more engaging and interactive, using the free version, Aurasma Lite, can be a very helpful tool.

Since we’re also scientists over here, there are, of course, many human factors and ergonomics design considerations for using AR games, which I will only touch on briefly: there are ergonomics issues related to weight and extended use as these devices, given their hardware and power needs, may be quite heavy; general design issues related to multi-modal displays, attention, and distraction; the effect of 3D on developing visual systems; guidelines for designing near-to-eye games; issues related to simulator sickness. Future research areas could touch on AR and engagement, AR and training and education effectiveness, and ROI analyses to determine exactly how beneficial AR really is for serious games and training purposes.


This blog post comes from the following conference presentation that won best graduate paper at the 2012 Human Factors & Applied Psychology student conference: Procci, K. (2012, April). Augmented reality serious games. Paper presented at the 2012 Florida Student Conference for Human Factors and Applied Psychology at Embry-Riddle Aeronautical University, Daytona Beach, FL.

1. Szalavári, Z., Eckstein, E., & Gervautz, M. (1998). Collaborate gaming in augmented reality. VRST’98 (pp. 195-204). ACM.

2. Liarokapis, F. (2006). An exploration from virtual to augmented reality gaming. Simulation & Gaming, 37(4), 507-533.

3. Aoki, N., Ohta, S., Masuda, H., Naito, T., Sawai, T., Nishida, K., Okada, T., Oishi, M., Iwasawa, Y., Toyomasu, K., Hira, K., & Fukui, T. (2004). Edutainment tools for initial education of type-1 diabetes mellitus: Initial diabetes education with fun. Proceedings of the 11th World Congress on Medical Informatics (pp. 855–859). Amsterdam: IOS Press.

4. Crandall, R. W., & Sidak, J. G. (2008). Video games: Serious business for America’s economy. Entertainment Software Association.

5. Klopfer, E. (2008). Augmented learning: Research and design of mobile educational games. Cambridge, MA: MIT.

6. Squire, K., & Klopfer, E. (2007). Augmented reality simulations on handheld computers. Journal of the Learning Sciences, 16(3), 371-413.

2 Responses to “Augmented Reality Serious Games”

  1. Albert says:

    Great article.
    Do you know the serious game is a very complex game with augmented reality to sensitize about environments.
    It has 15 markers that can interact simultaneously and has a lot of effects, a virtual city with traffic, smoke, industry, a forest…

    It was made with Unity3d (That you mentioned in your article) and is a cheaper game… it’s totally free ;)

    It also has a didactic guide (Although it is only in Spanish by the moment)…

Leave a Reply

six × 1 =