SCUDHunt and Shared Situation Awareness
-- includes excerpts from the CNA publication Gaming and Shared Situation Awareness
Overview
ThoughtLink, together with the Center for Naval Analyses (CNA), conducted an experiment that measured the effect of different modes of communication and visualization on a distributed team's shared situational awareness (SSA). The study was done for the Defense Advanced Research Projects Agency (DARPA) in support of the Wargaming the Asymmetric Environment (WAE) program. The approach we took was to develop an experiment in which teams would play an online game. The game we developed, SCUDHunt, was designed with the following factors in mind:
- Team members had to share information to do well
- Their decisions could be directly and easily recorded
- The measure of their decisions would describe the degree of shared situational awareness of the team.
The experiment included 6 teams composed of 4 people on each team. Each team played 6 different versions of the game - each version used a different combination of communication and visualization tools. The results of this study showed that communication and visualization play an important role in how distributed teams build their shared situational awareness. It also exposed some interesting facets associated with distributed teams, their processes, and social dynamics and relationships.
In addition to these findings, the experiment showed that the use of simple games, designed to target specific experiment goals, is a promising technique for continued research in this field. We hope to continue to explore the relationship between shared awareness and information superiority concepts, such as Network Centric Warfare.
This is a complex and rich area for research. Further studies in this area will contribute to our understanding of what helps and hinders distributed teams' SSA; what parameters we can, and should, measure; how to measure the effect of those parameters; and how to apply lessons learned to larger more complex situations (e.g., Coalition Operations) and to less understood adversaries (e.g., Asymmetric Warfare).
 Back to the top of the page
Shared situation awareness (SSA)
There is no agreement yet on the definition of situation awareness or shared situation awareness (SSA); different communities think of it in different ways. We spent quite a long time reviewing SSA research and formulating our own definition. For an extensive discussion of situation awareness - both individual and shared - see Defining and Measuring Shared Situational Awareness by CNA team member Albert Nofi.
We characterize situational awareness as a dynamic mental model of our operating environment and our place in it. We build this model through a process we call situation assessment, which consists of four interwoven subprocesses: perception, comprehension, projection, and prediction.
For our purposes, the best description of the process of situation assessment is the one described by Mica Endsley in a 1995 paper as:
the perception of the elements in the environment within a volume of space and time, the comprehension of their meaning, the projection of their status into the near future, and the prediction of how various actions will affect the fulfillment of one's goals.
Note: In the paper, this is Endsley's definition of situational awareness. In a later email, she then refers to it as situation assessment.
So the critical factors in the process of situation assessment are:
- Perception-acquiring the available facts
- Comprehension-understanding the facts in relation to our own knowledge of such situations
- Projection-envisioning how the situation is likely to develop in the future, provided it is not acted upon by any outside force
- Prediction-evaluating how outside forces may act upon the situation to affect our projections.
It's important to note that these four stages form a dynamic tapestry of interwoven threads rather than a static sequence followed like a flow chart. In developing our situation assessment, we don't necessarily follow the neat flow from perception, through comprehension, then projection, and finally prediction. These stages occur virutally simultaneously, given the speed with which our minds work. As we perceive the information, we are already processing it for comprehension and its implications for our purposes. And this process goes on continuously, so that our situational awareness evolves continuously as well.
We then use the results of our situation assessment to develop a mental model and that mental model represents our situational awareness. A mental model is a "psychological representation of the environment and its expected behavior." The purpose of a mental model is "to provide conceptual framework for describing, explaining, and predicting future system states." The mental model is inherently subjective, based on integrating acquired information with our own personal structural and situational factors. Structural factors include: training, experience, culture, personality, interests, and skill level. Situational factors include things such as mood, fatigue, stress, time pressure, and the complexity and ambiguity of the situation.
The quality of our situational awareness may be characterized by the degree to which our mental model- our situational awareness-"accurately" reflects objective reality. Measuring the "goodness of fit" between reality and SA is not an easy task, however, we were able to neatly capture this in the game construct, described below.
So what is shared situational awareness? Is it that we understand we are in a common, or shared, situation? Or does it mean that we have a common understanding of a particular situation?
For our purposes, we defined shared situational awareness the degree of overlap in the situational awareness of team members. There are three elements in the development of a team's shared situational awareness.
- Build individual situational awareness.
- Share individual situational awareness. This is probably the most critical factor in creating shared awareness. It depends on effectively communicating each person's awareness, in order to build a shared mental model from the individual mental models.
- Develop the group's shared situational awareness. This is the integration of the different individual mental models of the situation. Note that there need not be a single "team mental model." Multiple mental models can exist among team members but the models must overlap sufficiently to make it possible to perform the mission.
Back to the top of the page
Game Description
SCUDHunt is a simple, short, abstract game of command, control, communications, intelligence, surveillance and reconnaissance (C3ISR) played by a team of 4 to 7 players. The game requires group decision-making and allocation of scarce resources under conditions of time pressure and uncertainty.
The goal of the game is for team members to correctly determine (within a specified timeframe) where three Scud launchers are located, on a 5 by 5 grid.
Each team member controls a different sensor, and team members must share their sensor results in order to identify the launcher locations. In each turn, team members decide, typically in a collaborative process, where to locate each sensor. The results for a sensor are returned to the team member controlling it. Results include: X - launcher found; O - no launcher in the square; ? - not sure. Some sensors can also be killed or temporarily disabled on a turn.
At the end of each turn, based on the search results to date, each team member nominates at least three grid squares in which the launchers might be located. The overlap among the nominations reflects the team's shared situational awareness. If there is no overlap, every team member will vote for a different set of grid squares. If there is complete shared situational awareness, each member will vote for the same set of grid squares.
We measured SSA as: the total number of nominated squares divided by the number of unique grid squares nominated. Team SSA scores vary from 1 to the number of team members. As an example, given 4 team members each of whom nominates three squares, if there is minimal SSA, each member will nominate different sets of squares. Then the total number of nominated squares is 12 (4 members x 3 squares/member) and the number of unique grid squares is also 12 (there is no overlap), and the team's SSA score is 1. If the same team has complete SSA and each member votes for the same set of grid squares, then their team score will be 4: 12 squares nominated overall divided by 3 unique nominations.
Two important properties of this measurement of SSA are 1) it directly measures SSA and 2) it does so in the natural context of the game, thus it does not depend on the person's subjective assessment or description of their mental model. More information on SCUDHunt is available at http://www.scudhunt.com.
Click to enlarge
Figure 1. Sample SCUDHunt Game Board, with COMINT position on left and shared visualization on right, displaying results returned from all sensor assets.
Back to the top of the page
How SCUDHunt Can Be Used
In 2001, the University of Arizona Management Information Systems Department and the Center for the Management of Information are using SCUDHunt in an experiment exploring leadership, trust, and situational awareness for ARI. Another potential educational application of SCUDHunt might be to teach the importance of sharing information and developing team processes when working in a distributed environment.
Additional research areas include: information sharing, information quality, new warfighting concepts, decision making, and new command and control strategies. The game can be made more or less complex depending on the research agenda. For instance, in the version of SCUDHunt used in the DARPA experiment, the enemy was stationary. In future versions of the game, the enemy could move, the game could include decoys, and the sensor assets' reliability could change over time (e.g., the spy could be turned and give faulty reports).
The game can also be used for training applications. Some ideas for training applications include: training the importance of information sharing or highlighting problem areas found in virtual teams. If you are interested in finding out more about how you might use SCUDHunt, contact us at scudhunt@thoughtlink.com Back to the top of the page
SSA Experiment and Results
The experiment was designed to evaluate the effects of different modes of communications and visualization on a distributed team's ability to develop shared situational awareness.
The 3 communications modes were:
| |
 |
none |
| |
|
text chat |
| |
|
voice teleconference. |
The 2 visualization modes were:
| |
|
none |
| |
 |
shared view of the results of each turn (i.e., for each sensor, the result returned was shown: 0, X, or ?. However, team members could not see from the visualization which asset had generated which result.) |
We used a Latin Square design, with six teams playing six games each. Each team played all six combinations of the communications and visualization modes; each team played those games in a different order.
There were four players on each team, so we recruited 24 players altogether for the experiment. During game play, the players were distributed, playing over the Internet from different physical locations. In two of the six teams, players knew each other beforehand; on the other four teams, players did not know each other before the experiment began. Data collected in the experiment included:
| |
|
online game data: each player's moves for every turn for each game and their estimates of target locations at the end of each turn |
| |
|
recordings of all teleconferences |
| |
|
archives of the text chat sessions |
| |
|
pre-experiment questionnaires about each player's background in computers and games |
| |
|
post-game questionnaires, filled out by most players. |
We used standard analysis of variance techniques to determine whether the SSA results showed statistically significant differences:
| |
|
Among the teams |
| |
|
Resulting from the sequence in which the teams played the games |
| |
|
As a result of the different combinations of communications and visualization modes. |
The full results are available in the report "Gaming and Shared Situation Awareness"; in a simplified form, the results of our hypothesis tests are:
| |
|
Do communications and shared visualization affect building a shared picture? YES |
| |
|
Do communications matter? YES |
| |
|
Does mode of communications matter? NO |
| |
|
Does shared visualization matter? YES |
| |
|
Is there interaction between communications and shared visualization? PROBABLY |
The least anticipated result was that the mode of communications, text chat or voice teleconference, was not associated with statistically significant differences in team shared situational awareness. We had thought that, because voice communications are richer in contextual information, games with teleconferences would have higher SSA results than games with text chat. This was not the case.
At least two and normally three project members observed each of the 24 games and we collected some interesting observations about game play. These are discussed further in the final report Gaming and Shared Situation Awareness.
| |
|
Playing a game appeared to promote bonding and trust. Team members bonded quickly and tightly. |
| |
|
Teams that established a specific process for using their assets and covering the game board appeared to have better shared awareness. This could compensate, to some degree, for degraded communications or visualization. |
| |
|
Teams that started with the no-communications game seemed to take longer to achieve higher scores in subsequent games. |
| |
|
Some female team members appeared to be more concerned about achieving consensus than other team members. |
Back to the top of the page
Related Publications
Back to the top of the page |
