Evaluation of “Cultural Awareness – Afghanistan Pre-deployment”: A User Study

Abstract

Attention to human factor in conducting any type of military operations is an objective of utmost importance in the context of troop training within NATO. To this purpose, a number of serious games were sponsored by NATO and the Ministries of Defence (MoDs) of various nations, addressing different aspects of cultural awareness in approaching human beings during a mission. A significant example is “Cultural Awareness–Afghanistan Pre-deployment”, a serious game distributed by UK MoD and widely adopted in Defence Schools to support pre-deployment training of troops in Afghanistan. In the context of GALA, the European Network of Excellence on Serious Games, the Romanian National Defence University and NATO-STO CMRE conducted a user study focused on the learning impact of this serious game. The study was first run in parallel on different typologies of players in order to get different perspectives of evaluation; respective results were then merged and compared to achieve a more complete view, and, hence a better understanding of the advantages and limitations of this type of learning tool.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Awareness of the importance of human factor in conducting military/security operations has strongly increased within NATO in the latest years. The revolution in military affairs, dating from the end of the Cold War, and the technology breakthroughs achieved in a number of different fields, dramatically changed the way NATO considers the role of the human operator in military activities. This is particularly true when military staff is involved in complex socio-technical systems, with their difficult multi-language, multisystem and multicultural contexts, in uncertain and ambiguous environmental and conflictual situations. It is considered that new approaches to military situations, with a focus on human factor, mostly benefit multinational operations and - in particular - Operations Other Than War (OOTW) [1].

Part of the effort of NATO ACT and MoDs of various NATO Nations consists in sponsoring and, in some cases, contributing to develop serious games [2], seen as learning environments addressing various aspects of cultural awareness and human behavior correctness during various kinds of security missions (e.g., against terrorism, illegal trade of weapons, etc.).

One of selected best practices of game-based training application on this topic is “Cultural Awareness Training - Afghanistan” [3] a serious game sponsored and distributed by UK MoD and widely adopted in Defense Schools to support pre-deployment training of troops in Afghanistan within International Security Assistance Force (ISAF). ISAF’s main objective in Afghanistan is to conduct security and stability operations throughout the country together with the Afghan National Security Forces (ANSF). The “Cultural Awareness Training - Afghanistan” game provides a self-paced training environment that generates and develops the cultural competency of trainees, helping them to perform successfully in civil-military operations based on relevant cultural information. Serious Games, unlike entertainment games, necessitates the implementation of specific educational and technical requirements related to pedagogical constructs, learners’ assessment and standardization [4, 5]. To this aim MAN has carried out the implementation of the “Cultural Awareness Training - Afghanistan” on the ILIAS (Integrated Learning, Information and Work Cooperation System) platform of the “Carol I” National University of Defense in Romania [6]. ILIAS is an open source web-based Learning Management System (LMS) which supports learning content management (including SCORM 2004 compliance) and tools for collaboration, communication, evaluation and assessment.

In the context of GALA, the European Network of Excellence on Serious Games [7], the Romanian National Defense University and NATO CMRE conducted a user study particularly focused on the learning impact of this game. Thanks to the interoperability feature implemented by MAN, the user group study could be conducted remotely, in parallel on different kinds of user groups (soldier trainees at MAN, and civilian/military staff at CMRE) in order to get different perspectives in the game test and evaluation. Responses were then statistically analyzed and compared, in order to achieve a more complete and wider view.

This work aims at describing how the user studies were conducted and at presenting and discussing the feedback received by the users through a post-briefing questionnaire.

2 “Cultural Awareness – Afghanistan Pre-deployment” Serious Game

“Cultural Awareness - Afghanistan” is part of a pre-deployment course, the main objectives of which are to train troops (a) to familiarize with the specificities of a mission (in terms of techniques, tactics and risks); (b) to familiarize with the geography of the area; and, above all, (c) to learn the best approach to keep with local people. It was developed in 2007 by LINE Communication, the UK corporate which works very close with the UK MoD for training and education programme. LINE worked with the MoD in identifying authentic scenarios the troops could face, possible cultural factors that could influence them and ways in which service personnel could react. Culture is a subjective issue and our opportunity was to raise awareness and facilitate a behavioural shift rather than teach individual elements of Afghan culture. The course needed to be adaptable; it needed to be available as a pre-deployment desktop resource for classroom based training and for commanders to use with their troops in any environment once deployment had taken place.

In this context the “Cultural Awareness” serious game is intended to help the users to develop and reinforce skills in identifying relevant cultural information, generating decision criteria and selecting alternatives. The skills which can be achieved through the game mainly consist in:

• basic knowledge of Afghan culture, and, more importantly,

• capability of making the best decisions in various scenarios and situations, by using available resources.

The game consists on a first introduction (aimed at providing with background information on Afghanistan geography, population and culture), followed by a set of game scenarios where the players can verify and improve the cultural awareness achieved in the first part.

It integrates audio and visual assistance able to provide the user with useful information and, hence, support feedback and performance assessment. The development concept revolves around adaptable decision-making scenarios and real-time assessment of actions. The student can choose among different scenarios and his/her performance is assessed along the game by showing escalated/decreased state of conflict, depending on his/her personal choices and answers.

It is clear that the first target users for which the game has been conceived belong to military personnel who will perform missions in Afghanistan theatre of operation. However, such training can be extremely useful also for those civilians who have to participate to humanitarian missions with administrative, logistic, or support tasks. Indeed, some of the scenarios included in the game address such kinds of applications.

The major characteristics of the game as a learning environment follow.

• The primary purpose of the game is to help players learn something real about the reality of battlefield, not simply to entertain them.

• In the military field, learning the games rules (which means to accomplish the training task) can be the difference between being dead or alive, to survive and carry out the mission on the battlefield.

• Players are autonomous. They need to be free to make decisions. Only a very short pre-briefing on how to access the game through the web and to use the game’s man-machine interface is provided at the beginning.

• There are different game scenarios classified according to various types of missions. The scenarios propose possible situations, and solutions to the problems encountered can be found among the theoretical inputs provided in an introduction. The difficulty level increases from scenario to scenario: they need to apply, evaluate or adapt the knowledge/experience background provided in the introduction, depending on the difficulty level of the specific problem [8, 9]. This means that in the most complicated cases the user cannot directly find the solution in his background knowledge, but needs to associate various factors and, hence, extrapolate/adapt the solution from the whole fund of cultural awareness gradually achieved during the game.

• Users can repeat an action as many times as they want, until awareness is complete.

• Users have the possibility to access available resources (pop-up windows with information in the form of text) while playing, if they have doubts or do not remember some knowledge element.

• The scenarios, situations and tests are presented through a combination of photos, text, and videos with interview to military experts.

• A game characteristic is immediate feedback. Feedback motivates and stimulates action, even when it is negative. Players see the consequences of their actions and receive an instant response if they acted well or not.

• Assessment is given during the game by displaying both the correct answers/solutions, and a conflict indicator:

  • feedback is provided via virtual human presence embedded in the game (videos with short interviews or statements by military personnel actually involved in operations and with profound, direct experience on the field);

  • if the player fails, the culture risk indicator clearly indicates this, and a video explains the related risk.

3 Organization of User Studies of “Cultural Awareness – Afghanistan” at MAN and at CMRE

The study was first conducted in parallel on different typologies of players (soldier trainees at MAN; and volunteer players - civilian and military staff - at CMRE). The results of respective analyses were then fused and compared.

3.1 Characteristics of the User Groups

More than 600 MAN users played the game and 60 of them answered the post-briefing questionnaire. They are Romanian students at the National Defense University and other military personnel which will perform missions in Afghanistan.

They are highly goal-oriented. This SG is part of their pre-deployment training course, and they usually know what goal they want to attain. Their motivation is high, as players are totally aware that acquired knowledge is crucial for their future missions and will be applied on the field in near future.

Further, they know that learning objectives have to be achieved through their own effort, in order to be of real value to them.

At CMRE 40 users were involved. They are volunteers belonging to the staff. Most of them are civilian Staff, although some military staff volunteered to participate in the test, and the majority of participant are of Italian nationality. Non-Italian participants are nationals of a diversified sample of NATO nations. A They do not expect to apply what learnt immediately to real life. However, they are deeply aware of the problem and might be involved in operations on the field.

In general, they have various degrees of familiarity with military operations from different perspectives. Only a limited number of users (three people) have got first-hand experience on the field.

As an average they have medium experience as video-game players, very low/no experience as serious-game users. Most of them have mid to high technical expertise in modeling & simulation or computing (Fig. 1)

Fig. 1.

Snapshot of “Cultural Awareness Training - Afghanistan” game. The cultural risk indicator.

Some information can be retrieved from the data shown in Fig. 2 on the profile of the MAN and CMRE players as a whole group.

Fig. 2.

Description of players’ profile, in terms of (a) histograms of age, (b) professional role and (c) nationality (in percentage, over 100 users among CMRE and MAN).

3.2 Criteria of Assessment of Game Learning Impact Through User Groups

The procedure for the evaluation of the game through user groups was defined as follows:

• (1) Provide users with a quick pre-briefing (only instructions on how to access the game and to use its man-machine interface) either in a classroom or via email.

• (2) Let users play on their own.

• (3) Ask users to fill an on-line post-briefing questionnaire. Excluding some initial profiling questions aimed at drawing the layer profile and basic knowledge on the topic, all the questions are focused on assessment of the game impact as a learning tool. Most questions expect multiple answers; only a limited number allow open answers.

• (4) Collect all local answers (separately at MAN and CMRE) and make local (marginal) statistical analysis.

Share answers between MAN and CMRE, and subsequently compare and merge results. The following subsection presents a summary of most significant results of the global statistical analysis of answers.

3.3 Statistical Analysis of Evaluation Results

A statistical analysis of the most interesting and significant results of the game evaluation through user groups is presented in this paragraph.

One of the preliminary issues investigated was on previous training experience (through briefings, reading, etc.) on this topic. The great majority of CMRE users had not received any other type of training before, while about one half of MAN users had.

All users believe the information contained in the game correspond to real life situations.

Another important aspect investigated is time scale assessment of learning curve (i.e., time needed to understand how to play the game – excluding the time to go through the introductory briefing) and effective learning time (i.e., total amount of time needed to achieve the game learning objectives – again excluding the time taken to go through the introductory briefing) experienced by a user. As summarized in Fig. 3, answer statistics show that on average the game is extremely intuitive ad easy to use. The effective learning time is also extremely low: this suggests that the game allows an effective use of time and favours quick knowledge transfer.

Fig. 3.

Users’ responses on duration of their own learning curve and effective learning time (in percentage, over 100 users).

A number of game characteristics determining its usefulness as a learning tool are then addressed in the questionnaire (see Fig. 4). Attention is particulalry focused on usability, level of immersion and engagement, motivation capability, but also on interactivity level and assessment tools provided.

Fig. 4.

Users’ responses on a set of game features related to its usefulness as a training tool (in percentage, over 100 users).

Again, the game is considered on average very easy to use (which confirms previous results on average duration of learning curve), and very helpful in achieving learning objectives. Level of interactivity is medium, although information resources provided on demand are considered highly valuable. An important result is that the assessment tools provided in the game are very much appreciated; this is a feature of major importance for a serious game devoted to education and training.

Two sets of questions refer to the kind of learning goals users believe to have achieved by completing the game(cognitive learning goals, according to the revised Bloom’s taxonomy [8, 9], as well as soft skills). As presented in Fig. 5, users believe to have improved their knowledge at each of the revised Bloom’s taxonomy level, up to the top one, i.e., the adapting/creating level. It is particularly interesting that users get this awareness at the end of their game session; in fact an increasing difficulty level, leading the player to pass from mere remembering and understanding to analyzing, is implemented on purpose in the different game scenarios. As expected, evaluation and adaptation/creation learning goals are not considered to be deeply addressed by the game as the lower levels are; this is perfectly coherent with the game goals (Fig. 6).

Fig. 5.

Users’ responses on cognitive learning goals (in percentage, over 100 users), according to the revised Bloom’s taxonomy [8, 9].

Fig. 6.

Users’ responses on selected soft-skill learning goals (in percentage, over 100 users).

The same enthusiastic feedback is expressed at average in the case of soft skills: again most of users state to have experienced an improvement (from reasonable to high) in the psychological skills selected.

4 Discussion

A summary follows of most significant feedbacks/responses at the two sites. Some responses are common to the two user groups:

• Great majority of participants at both sites consider this game very useful.

• All considers the topic and content very interesting.

• Most participants believe that the game enables a good knowledge transfer, quicker than other more traditional means (such as books or slides).

At MAN the most interesting results are:

• Most of participants at both sites consider that this game is very well made.

• Most believe that the game represents a real support in terms of theoretical training for future deployment.

• Most users state that they were curious to try/test new learning tools.

• Most players state:

  • the interface is very friendly and contains catching audio-visual materials;

  • the content is attractive, with detailed features which enable a good fixation of knowledge.

At CMRE the view is partly different.

• Most judge its man-machine/graphical interface too static, basic and obsolete.

• About one half of them considers the game boring and not immersive at all, and suggests to significantly improve its graphical interface to make it more appealing.

• Feedback and assessment are considered appropriate and successful.

The evaluation is generally very positive, and the SG seems to be well received as a new and modern way for assimilating appropriate knowledge with the purpose of determining proper behavioral and attitudinal reactions. It can be seen a valuable, complementary solution to traditional education and training tools.

Major differences in responses might be related to either the level of engagement the game can provide to the two user, and to their different background: different perspectives and different experience are likely to generate different perceptions.

MAN military students are highly motivated, and their target is to get profound knowledge of the content. If the means to achieve this are a serious game instead of pages and pages of a book or a power point presentation, they are probably very happy; also, they are curious to try new learning tools (as it can be deduced by their open answers). In this way they give less importance to the entertainment/fun aspect of the serious game, usually related to 3D sophisticated and realistic animations and high level of interactivity.

CMRE users seem to have more expectations in terms of technology and entertainment than a strong will of achieving learning goals. Although they are interested in the topic, and wish to achieve some knowledge, they look at (and much weight) the fun aspect of the game as well. Hence, many consider the serious game boring and not efficient enough; these users are generally video-game experts, hence for them using arrows and mouse, possible at the same time, as in a typical 3D, technologically advanced game, is not a barrier. However, they tend to underestimate that the simplicity of its user interface and the low-mid interactivity level very much decrease the game learning curve, and make the game extremely user-friendly for any kind of user, from neophyte to video-game expert.

In conclusion, the richness of information and realism provided through video segments with interviews to people really operating in the field seems to be a very clever and successful expedient to:

• (1) make the players feel to get in touch with something real and useful for a future deployment,

• (2) make users more confident on what they are learning,

• (3) contribute to make their effective learning time short.

The simplicity and lack of refinement of its graphical interface annoys the most “technological” users, but allows to achieve a very smooth, almost instantaneous learning curve for all users, independently from their background as video game players. So, the simplicity of the user interface and of the game engine very much increases the game usability. Too refined 3D graphics (implying a certain ability with mouse and keyboard to move in the scenarios and interact with the virtual environment) may be seen as a barrier by non-expert users.

Furthermore, the success and generally positive feedback obtained from this user study applied to “Cultural Awareness – Afghanistan Pre-deployment” seems to suggest the following consideration. After all, the level of fidelity and refinement of the virtual reality created by a game of the latest generation (i.e., generally characterized by refined 3D graphics and sophisticated game engines) is certainly important for the effectiveness of a game, but more important is the content, the learning goals selected, the motivation that is transferred to the user. If the content and the information to transfer are too limited, and are not able to convince to be real, a nice 3D graphics does not help. This is particularly valid when the learning objectives are mainly of cognitive kind.

This is an important lesson learnt: very simple games, but providing clear learning objectives and real-life references, may be very successful even nowadays, even in a field such as security, where very high-tech solutions and extremely refined (but often quite complicated) tools are often pushed and advertised as fundamental features.