The Office of Naval Research takes great pride and interest in their sailors in many ways such as funding a massive training movement called Capable Manpower. UCF RETRO Lab has been subcontracted by Raytheon BBN Technologies to test, evaluate, and provide quantified research findings about the effectiveness of game-based training. RETRO was involved with both research and development as part of this multi-year effort.
As a part of the Virtual Environments for Ship and Shore Experiential Learning (VESSEL) project, BBN and Alion developed the Damage Control Trainer (DCT), which is an award-winning serious game used to train US Navy recruits on communication protocols, shipboard navigation, and damage control procedures. This game is currently implemented at Recruit Training Command. The DCT is a story-driven training experience. The narrative is constructed using cinematics, in-game missions, and interactions with non-player characters (NPCs). Our 3D artist Julian Orrego created a number of cinematics for the start of each chapter to bridge in-game events between missions.
|Opening Cinematic for the DCT|
In addition to the cinematics, RETRO also developed Flash minigames to augment training, including Insignia Searcher and the Bullseye Trainer. Insignia Searcher is a quick game that helps recruits to recognize the different US Navy officer rank insignia, which provides information about rank and officer specialties. The Bullseye Trainer is used to help recruits understand the shipboard navigation system. Compartments in US Navy ships all have specific identifiers that are used to describe both the location and purpose of that space, sometimes known as bullseyes. Each ship conforms to the same system and the Bullseye Trainer is a drill exercise to help recruits memorize and understand the different parts of the bullseye as it relates to the ship itself.
|Note: Games launch in a new window.|
Modding games is a cheap and incredibly effective way to create serious games using existing game engines. Several commercial games are packaged with game-editing software and active modding communities often form. When discussing ways to improve our shipboard navigation training, we decided to create a 3D virtual training environment by modding Bethesda’s Fallout 3 (2008). One of the in-game locations is Rivet City, an aircraft carrier modeled after the USS Oriskany. Our programmer Daniel Brown modded Rivet City to have more levels and added to every room the appropriate compartment identifier. He then built a quest where players had to interact with Damage Control Central and check different compartments in the ship. This became both a testbed for shipboard navigation as well as a training environment.
|Modded Rivet City Navigation Quest|
RETRO Development Team
|Lab Directors||Drs. Clint Bowers & Jan Cannon-Bowers|
|3D Artist for DCT Cinematics||Julian Orrego|
|IS Production Manger||Rachel Joyce|
|IS Designer||Peter Smith|
|IS Programmers||Jorge Salazar, Brian Stabile|
|IS Artists||Brent Hermanski, Jorge Salazar|
|Bullseye Production Manager||Rachel Joyce|
|Bullseye Designer||Lucas Blair|
|Bullseye Programmer||Stephen Nedley|
|Bullseye Artist||Julian Orrego|
|Rivet City Mod Manager||Katelyn Procci|
|Rivet City Mod Designers||Daniel Brown, Katelyn Procci|
|Rivet City Mod Programmer||Daniel Brown|
|Rivet City Mod Artist||Daniel Brown|
RETRO was responsible for conducting validation studies on the DCT, usability studies on the mission editing software VITAL, and a number of other studies examining different aspects of games and how they could be used to improve training.
To support development efforts, several usability studies were undertaken. We conducted a usability study on the DCT in a sample of US Navy recruits. We found that brief, extremely inexpensive training (i.e., supplementary paper-based information about the compartment marking system) in conjunction with the game improved performance without making any edits to the software.
We also conducted usability studies for both Insignia Searcher and the Bullseye Trainer. Usability issues were identified and corrected. Additionally, we conducted a usability study on the VITAL Authoring Tool for the DCT. The tool was found to be useable, even by those with limited experience with computers. Suggestions for potential modifications were generated.
Several validation studies were also conducted on the DCT, Insignia Searcher, and the Bullseye Trainer.
To validate the Damage Control Trainer game, we initially conducted a pilot study using US Navy recruits in November 2008 which was followed-up with a complete study in April 2009. We also conducted a team validation study in late 2009. We found that those in the game condition consistently outperformed those in the no-game condition with respect to time to navigate to compartments and errors made. Decision making errors were reduced by 50%, communication errors were reduced by up to 80%, and situational awareness and navigation skills were improved by 50%.
In early 2008, we conducted a validation study on Insignia Searcher comparing groups of undergraduate and JROTC students who either played the game or viewed a PowerPoint. We found the game to be as effective as traditional methods and that it was more emotionally engaging.
In 2010, we conducted a validation study on the Bullseye Trainer. We were unable to improve our participants’ abilities to find bullseyes within the DCT with our 2D training intervention. We did find that gaming familiarity was the strongest predictor of performance.
In order to further the science of serious games and articulate a set of guidelines for training game development, we conducted several experiments based on several different theoretical foundations to uncover why and how serious games are so effective. These studies included:
- Brain lateralization as a predictor of team performance
- The effectiveness of written narratives as pre-experiences for improving game-based training
- The role of opening cinematics in improving game-based training
- The use of avatars for training interpersonal skills
- Engagement and flow as a construct and the effect these states have on training effectiveness
- Worked examples
- Using synthetic voices for training communication skills
- Avatars and personalization
- Predictors of positive game-based training outcomes
- Gaming preferences
For more details about these results, please check out our Publications.