The Role of a
Distributed Interactive Collaboration Environment (DICE) for
Interagency/Military Training
Julia Loughran
Marcy Stahl
ThoughtLink, Inc.
703/281-5694
loughran@thoughtlink.com
mstahl@thoughtlink.com
Abstract
This paper describes ThoughtLink’s research into the
use of low-cost gaming and web-based collaboration technologies to enhance
training, coordination, and communication within the interagency community,
particularly training for complex contingency operations[1]
(CCOs). This research, which began in
1998, is being funded by the Department of Defense (DoD)
C4ISR Cooperative Research Program (CCRP) and the Defense Advanced Research
Projects Agency (DARPA). The research will culminate at the end of this year
with an experiment examining how a Distributed Interactive Collaboration
Environment (DICE) can be used to enhance interagency training. An annotated briefing documenting the results
of this experiment will be available at the end of the calendar year.
The
experiment will evaluate how DICE, a collection of commercial off-the-shelf
(COTS) and government off-the-shelf (GOTS) products tailored with appropriate
content, can support the interagency community as they train on the development
of a political-military (pol-mil) plan for a hypothetical CCO. Although the experiment will focus on
training the interagency community in the pol-mil development process, DICE
might also be used as an operational planning and rehearsal environment for
real-world contingencies. The DICE
approach, using collaboration tools and applying them to specific domains, can
be applied to many real-world issues and problems.
1. Overview
The focus of this research is not about the
development of new collaboration technologies.
There is already a multitude of COTS and GOTS products available
today. The focus, rather, is on how
these technologies can be used to enhance training, communication, and
information sharing. The experiment, planned for this fall, will explore how
these technologies can be applied to interagency training for CCOs and will
identify the potential benefits and pitfalls of their use.
Section 1 of this paper describes the project’s
history and the scope of the research in fiscal year (FY) 1999; Section 2
describes the technologies and components of DICE; Section 3 provides the
experiment details; and Section 4 summarizes the proposed benefits of DICE as
well as its potential use for training subsets of a JTF staff.
1.1
FY98 Work
The DICE concept is an outgrowth of ThoughtLink’s FY98 task for DARPA, in which we explored the applicability of low-cost collaboration and computer game technologies to Joint Task Force[2] (JTF) training in the area of operations other than war (OOTW)[3]. The DARPA sponsor was Mr. Dell Lunceford, Program Manager for the Advanced Simulation Technology Thrust (ASTT) program. The ASTT program does advanced research in high payoff areas for the Joint Simulation Systems (JSIMS) program and the Joint Warfare Simulation (JWARS).
There were 3
phases to our FY98 work: understanding today’s JTF staff training, particularly
for OOTW-related training, reviewing collaboration and computer game
technologies, and defining a new training environment to augment, not replace,
current JTF training methods.
An early
observation in our review of today’s JTF training was that training occurred at
two ends of the training spectrum, with a void in the middle. At one end of the spectrum are low-cost,
low-tech alternatives: academic training and seminar games. These methods are used for small groups
(2-200); have a narrow focus; and are relatively static (once developed, the
content doesn’t radically change with each new training audience). At the opposite end of the spectrum is the
simulation-supported command post exercise (CPX). This method is very expensive (typically
$1M+) and involves a large training audience (100-1,000) supported by almost
equally large training support organization.
It appeared that there is utility in developing computer-based training methods to fit in the middle of the spectrum: medium-cost and medium-fidelity. Relevant research areas included web-based gaming environments, multiple user role playing games, strategic thinking games, and synchronous and asynchronous collaboration tools, see Figure 1.
The product of our FY98 work
was a report – “Applying Commercial
Gaming and Collaboration Technologies to JTF Staff Training” [Loughran et
al., 1999]. The report proposes a new
training environment to augment the current JTF staff training methods. This new environment would use collaboration
technologies, together with a team of role players and facilitators, to help
multiple geographically distributed participants work together. The report is available on our web site,
http://www.thoughtlink.com.
Figure 1. Spectrum of JTF Training and Areas of Evaluation
1.2 FY99 Work
In FY99, DARPA,
under ASTT Program Manager Mr. Larry Willis, and CCRP, under Chairman Dr. David
Alberts, are funding ThoughtLink to integrate COTS and GOTS collaboration
tools, add appropriate content, and employ the integrated products and content,
referred to as DICE, in an experiment in the fall. This research will be conducted with an
operational user.
The operational users are three organizations charged
with training US government officials to respond to complex contingencies under
Presidential Decision Directive (PDD) 56[4]. These organizations are: the National Defense
University (NDU), the
NDU, NFATC, and
the USA PKI train government officials from multiple agencies in how to develop
a cohesive political-military plan for the
Using DICE, we
propose that the training audience will meet in person for less time and do
part of the training while still at their office. The focus of the fall experiment will be to
identify the benefits to this form of training.
The assessment of DICE will occur in a two-phases. Phase 1 was a March 1999 PDD-56 training
exercise, described in Section 3.2. No
collaboration technology was used for either planning or training, no gaming
strategies were used to provide a time-phased scenario, and there was minimal
use of role-players. During the March
training event, data were collected (both quantitative and subjective) to serve
as a baseline, describing the training process without the benefit of
technology. Phase 2, to be conducted in
the fall of 1999. It will include some
limited face-to-face meetings (less than occurred in Phase 1) and it will also
include the use of DICE. In Phase 2, the
training audience will be instructed in the use of DICE and asked to perform a
subset of the tasks from Phase 1. Data
will be collected and compared, where feasible, to the Phase 1 data.
In support of this effort, employees from the
Major milestones for the project are:
- March
1999. Observe PDD-56 Brave Knight
training exercise.
- Spring
1999. Develop and collect requirements
for DICE technology; define the PDD 56 training objectives and approach.
- Summer 1999. Integrate COTS/GOTS products with PDD-56
training content to produce the DICE prototype.
- Fall
1999. Use DICE in a training experiment.
- December
1999. Document the results of the fall
experiment.
2. DICE Overview
DICE is not a specific product. It is a concept for integrating a collection
of collaboration tools, enhancing those tools with domain specific content, and
employing the tools and content using distributed gaming and role playing
techniques. The choice of tools can vary
depending on the operational user’s requirements. Some organizations already use collaboration
tools and may be committed to a specific set of tools, such as Lotus
Notes. Other users may have varying
requirements, depending on the type of collaboration and information sharing
they want to do, and their purposes for using the tools might vary (e.g.,
training, planning, and knowledge management).
Other requirements to be considered for each different potential user
community include bandwidth availability, security issues, and cost.
Despite the variability that might exist across
different users’ requirements, there are certain common features. The key features of DICE follow.
·
It runs over a network (WAN/LAN);
·
It supports collaboration between distributed participants. Users can share information, work together on
projects, ask questions, and access outside experts or trainers;
·
It provides a persistent environment (archiving capability), meaning
that documents are stored and available for retrieval and sharing amongst
participants;
·
It is platform-independent, thus it can be used on multiple kinds of
computers.
A desired feature for DICE is also a low-fidelity
simulation for stimulating the training audience and for conducting what-if
analyses in an operational environment.
A computer-based simulation will not be part of the fall experiment, but
instead, human role players will provide the simulation.
One of the reasons that this experiment’s focus is
on the process of applying technologies versus the technologies themselves is
that, although collaboration technologies have been available for many years,
knowing how to combine and apply these technologies is not widespread in the
DoD or other government agencies. Just as we had to learn to incorporate e-mail
into our daily work schedule, we need to explore how relatively low-cost
collaboration tools and gaming techniques can be used for both operational and
training applications. The DICE
experiment will identify the potential benefits and pitfalls of using collaboration
technologies to support the interagency community in training and planning for
CCOs.
Collaboration tools come in two flavors.
Asynchronous collaboration occurs when people communicate with each other at
different times, and synchronous collaboration is when people are collaborating
at the same time. Some examples of
asynchronous collaboration tools include e-mail, newsgroups, and bulletin
boards. Synchronous collaboration tools are not used as commonly as the
asynchronous collaboration tools. One
example of a synchronous collaboration tool that is growing in popularity is
Other synchronous collaboration capabilities such as
video teleconferencing (VTC) are used less frequently, but many tools exist and
many are free of charge. For instance, Netscape’s SuiteSpot includes the
Collabra Server which allows users to share information through “virtual”
discussion rooms, and Microsoft has a free product called NetMeeting that
supports synchronous collaboration and runs with their Internet Explorer
browser. NetMeeting is probably one of the
most popular free synchronous collaboration tools. It is being used in many
government projects to support synchronous collaboration. In addition, there are many commercial web
sites that provide a text chat capability for free. The use of this capability requires that you
have software to support the chat session and generally requires that you
register with the site.
In addition to these commercial tools, there are a variety of government owned tools under development to
support collaboration. These tools are
described in more detail in Section 2.2.
Collaboration in general, and DICE more specifically, seems like a good fit for distributed training applications. In PDD-56 training, the training audience includes disparate groups from many agencies, they’re short on time, and they could benefit from a reach-back capability to their offices, to provide access to experts who are not present at the training.
2.1 Capabilities
As was mentioned earlier, the actual implementation of DICE can vary
depending on the user’s requirements. In
this section, we will discuss the DICE capabilities important for PDD-56
training. Since we are still gathering
requirements and evaluating the COTS/GOTS tools that might be used, these
capabilities may or may not be incorporated into the fall experiment.
The DICE being
developed for PDD-56 training will most likely be a collection of software
products that operate in conjunction with an Internet connection. Because the training audience needs to have
the capability to access training materials prior to the training there will
most likely be a web site established to provide a persistent workspace. This workspace will be shared by all users
and contain electronic documents, images, audio clips, and videos related to
PDD-56 and the training scenario for the fall experiment. The site can also contain the details about
the training, including when and where the face-to-face portion of the training
will take place.
This new PDD-56
web site will also include a variety of collaboration tools. For instance, a bulletin board capability
will be provided so the training audience can direct questions to the trainers
or to other members of the training audience should they have a question about
a particular agency. They will also have the capability to send e-mail to the
trainers and/or other members of the training audience. Since e-mail is
generally not captured and archived, the PDD-56 DICE application might take
advantage of an e-mail archiving capability.
For instance, a tool called Hypermail captures and translates e-mail
between a group of people and displays it on a web
site. These archived e-mails can be
reviewed by members of the group and might contain information to help clarify
something, or be used to evaluate the types of communication conducted during
the training event.
Some of the
synchronous collaboration tools that will most likely be available include VTC,
audio, text chat, and shared white boards.
These collaboration tools will allow the training audience to
participate from their home stations and can potentially increase understanding
prior to the face-to-face portion of the training.
One of the requirements for this particular
application of DICE is that it must be very easy to use. One of the features that some of the
COTS/GOTS products include that make them easier to use is a “room-based”
navigation scheme. With this scheme,
users can navigate to different areas of the collaboration space by selecting
from a 2D or 3D map display. The 2D map
may have different rooms for each of the agencies. By selecting a particular agency room, the
user has access that agency’s information, a list of people to contact for more
information, a current listing of other people who are in that room at that
particular time, and any documents, audio, or video that might persist in that
room. When two people are in one of
these “virtual” rooms at the same time, they can have a conversation via the
text chat tool or VTC if they have a sound card and digital camera on their
computer. Using this room-based navigation
scheme, another “virtual” room might be related to the hypothetical scenario or
there could be individual rooms for each of the functional working groups
writing portions of the pol-mil plan.
The PDD-56 DICE training application will feature a
great deal of document sharing capabilities since distributed participants will
be working on different components of the pol-mil plan. Therefore, this DICE application must include
a version control capability and it must be easy for users to add documents from
their computers to DICE. In some of the
COTS/GOTS products, users can “drag-and-drop” a document from their desktop to
the virtual environment. In other tools,
users select a Browse button and, if using a PC environment, it opens the
standard Windows file navigation window.
For the purposes of the fall experiment, we will
need to capture data about the events that occur in DICE. Therefore there is a requirement for tools
that create user logs or that can be instrumented so we can collect the data pertinent
to the analysis of the training event.
Many of the COTS/GOTS products available do not provide this capability
and we will most likely have to develop the programs for gathering this data.
2.2 Collaboration Tools Under Consideration
There are a number of relatively low-cost or free COTS and GOTS products that may be used in our experiment. We are currently mapping the training requirements to the features incorporated into these products to identify the best tool or combination of tools. This process will be completed by the end of May. Some of these tools feature only a synchronous or asynchronous collaboration capability while others are fully integrated products that provide the complete range of support. A subset of the available tools is described below to provide an overview of their functionality.
The two most popular forms of synchronous tools (video
and audio teleconferencing and text chat) are probably Microsoft’s NetMeeting
and CU-See-Me by WhitePine Software Inc.
Both tools have a whiteboard and application sharing capability. NetMeeting is available for free from
Microsoft’s web site (http://www.microsoft.com)
and CU-See-Me has both a free and a commercial version. The free version is available from
Some of the COTS candidates for asynchronous capability include Instinctive Technology’s eRoom, Involv’s Teamspace product, TeamWave Software Ltd.’s TeamWave workplace, and PlaceWare. All of these products are web-based and provide a persistent workspace for multiple document types and URLs, version control for documents, a bulletin board capability, a polling capability allowing users to vote on issues, and text chat in an easy-to-use interface. Most of these groupware products operate with the use of Internet browser software along with the individual client software associated with each product.
Some of the collaboration products being developed and used by the government include Mitre’s Collaborative Virtual Workstation (CVW), GTE’s Intelligent WorkStation (IWS), and SPAWAR’s Odyssey tool. Both CVW and Odyssey use the room metaphor described earlier for navigation and were developed by making enhancements to LambdaMOO, a popular and freely available multi-user domain (MUD) software originally developed at Xerox PARC.
CVW supports both synchronous and asynchronous collaboration. As you navigate through the different virtual rooms, you can see the people and objects in each room. You can make copies of these objects and/or collaborate with someone about them. For instance, 3 people in a room can be looking at the same document and making comments about it using the text chat tool. Different people can have control over the document and make the agreed-upon changes. After the document has been revised, the new document will be in this room for other users to collect and copy to their desktop, or they can post questions and/or comments for others to review at a later time. CVW has been used extensively in the intelligence community and it is also the collaboration tool incorporated into COMPASS, the Common Operational Modeling, Planning, and Simulation System. COMPASS is middleware used to share Global Command and Control System (GCCS) data across multiple users.
IWS is a web browser-based tool that has similar capabilities to CVW with a slightly different development approach and interface. It supports both synchronous and asynchronous collaboration. It was developed using PlaceWare, a COTS collaboration tool, as a base. IWS is the collaboration component for the Joint Intelligence Virtual Architecture (JIVA). The focus of JIVA is the “modernization of intelligence analytical processes and methodologies.” (http://www.mews.org/jto/jivva.html)
The Space and Naval Warfare Systems Command (SPAWAR) has developed a collaboration tool called Odyssey. It is built upon the COTS tool called Facilitate.com. Odyssey is being used in the Adaptive Courses of Action (ACOA) Advanced Concept Technology Demonstration (ACTD). ACOA is funded by DARPA and the Advanced Information Technical Services Joint Program Office (AITS JPO) and is intended to revolutionize the joint planning process through the use of collaboration tools and other emerging information technologies. Odyssey, like CVW and IWS, supports synchronous and asynchronous collaboration through a variety of integrated tools.
All three of these products are being incorporated and supported in the Defense Information Infrastructure Common Operating Environment and are being used in fairly large experiments.
3. FY99 Work and Experiment
3.1
PDD-56 Background
”The PDD
defines “complex contingency operations” as peace operations
. . . and
unless otherwise directed, this PDD does not apply to domestic disaster relief
or to relatively routine or small-scale operations, nor to military operations
conducted in defense of
The motivation for PDD-56 stems from problems
experienced by the
NSC interest in defining PDD-56 arose from
experience planning for US and UN missions in
Although this PDD was signed in May 1997 and the
When PDD-56 is invoked, the Deputies Committee of
the National Security Council creates an Executive Committee (EXCOM), composed
primarily of Deputy Assistant Secretaries (DAS) from all participating
agencies, to oversee day-to-day management of
PDD-56 charges three organizations to develop and
conduct an interagency training program to be held annually. These organizations, NDU, NFATC, and PKI,
have held one training exercise per year since 1997.
The major features of PDD-56 to be trained are the
creation of the pol-mil plan and the interagency rehearsal of the pol-mil
plan. Other training objectives include
learning about other agencies: their culture, capabilities and limitations, and
providing an opportunity for networking for the training audience.
3.2 March 1999 PDD-56 Training
Exercise
The PDD-56 training audience
is composed of three levels of US government (USG) officials: desk/action
officers, directors, and DAS. The desk
officers and directors participate in functional working groups to develop a
pol-mil plan, then the DAS are briefed on the plan and role-play in the EXCOM
rehearsal of the plan.
The March 1999 training audience included about 45
people from 16 different agencies and organizations. They were from functional offices; no
regional specialists from any agency attended the exercise.
The scenario was set in
·
A large and sudden influx of refugees from
·
An economic slowdown due to
·
The potential deterioration of Soviet-era infrastructure, including the
transportation system, water and power grids, and nuclear power plants;
·
The rise of organized crime, in drugs, money laundering, and smuggling,
including smuggling of fissile material.
The scenario also posed specific requests for aid
and assistance from the Polish government that would need review by the
training audience, including a review of customs procedures, police training,
advice on economic restructuring of the agricultural sector, and a public
information campaign.
The training event took place over three days: March
24-26, 1999 at NFATC in
3.3 Data Collection and Results
We had 7 observers at the
exercise, from ThoughtLink, Greenleaf Point LLC, and NAWC-TSD. We sat in on all
of the briefings and the agency and functional groups. We took notes and informally interviewed some
of the participants. We also distributed
questionnaires at the end of each day to the training audience. The training audience was asked about the
following topics: their work background, level of computer access and
expertise, interest in using technology for future PDD-56 training, level of
satisfaction with the briefings and panels on each day, knowledge of PDD-56
after the training, and the degree to which they networked with other participants. The number of participants each day varied,
decreasing each day, and not all participants on a given day filled out
questionnaires. Figure 2 shows, for each
day, the number of participants in the training audience and the number who
filled out surveys. There are no data
for the total number of participants on Day 3, but we estimate that number at
about 25.
Figure 2. Number of Participants and Survey Respondents by Day
Most participants (27 of 29) were accustomed to
regularly coordinating/cooperating with other agencies in their work and had
been involved in several real world events requiring interagency
coordination. On average, they were not
very familiar with PDD-56: an average of 2.6 on a scale of 1-5 (1 is not
familiar, 5 is very familiar). Almost
all had desktop access to e-mail and the Internet and most rated themselves as
intermediate or expert in their level of computer knowledge.
Overall, they learned something new during the
exercise. On Day 1, 27 of 29 said they
learned something new about PDD-56. One comment from a member of the training
audience was:
“Good experience.
I know a lot about PDD-56, but at every event, I learn something new.”
Twenty-two of 24 respondents said they knew more about the interagency process after Day 2. They had a high level of satisfaction with the briefings and panels. Table 1. lists the training objectives and the average rating among participants on how well they felt those training objectives were met on a 1-5 scale.
|
Training Objective |
Average Rating |
|
Develop understanding for intent behind
PDD-56 |
4.2 |
|
Develop
appreciation for diversity of agencies |
4.1 |
|
Learn
other agencies’ capabilities and limitations |
3.6 |
|
Develop
personal contact with other agencies |
3.6 |
|
Learn
structure of pol-mil plan |
3.5 |
|
Learn
how other agencies function |
2.8 |
Table 1. Participant Ratings on Achievement of Training Objectives
The following results are of particular interest to
us as we plan for the fall experiment using DICE:
·
Time. For most participants, time
was short and they had difficulty getting 3 days away from the office. In fact, only 8 of the approximately 45 training audience
members attended all three days. In
later surveys given to a subset of the audience, an exercise length of 1 to 2
days was preferred.
·
Ratio of passive to active learning. Table 2 shows that most of Day 1
and 2 were spent listening to briefings and panels. From the observer point of
view, this quickly became tiresome and the level of interest flagged. Some comments were made about this in the
training audience questionnaires.
“I did not find agency
briefings at all useful; after two hours absorption rates fall to zero.”
“Would have liked longer
working sections with other agencies where you really learn, within the context
of this exercise, the capabilities/mandates, and limitations of the agencies.”
|
|
Day 1 |
Day 2 |
Day 3 |
Total time (minutes) |
|
Time spent listening
(minutes) |
270 |
225 |
varied from 0 to about 45 |
495 to 540 |
|
Time spent working
(minutes) |
45 |
100 |
135 |
280 |
Table 2. Allocation of Time Over Training Days
·
Interest in using web technology.
Most (21 of 27) wanted to use technology for distance learning for
future exercises. All respondents wanted to use technology to get more
information about USG agencies. Some
people wanted more detailed information about the agencies that were briefed at
the exercise; some people wanted information about the agencies for which
presentations were not given. Most
respondents (25 of 29) wanted to access administrative information about the
training online.
·
Respondents on Day 3 reported meeting 8.9 people on average. Some would have like more and one participant
suggested incorporating a social event into future exercises. This aspect of the training is an important
benefit that must be preserved when technology is introduced into the
training.
“Most useful benefit was to interact with
other agency personnel in thinking through a USG response to a scenario.”
·
Seventeen of 24 would have found it useful to have access to other
groups during development of the pol-mil plan.
In the March event, this would have required someone from one group
going over to the room of another group and asking about their work. In the future, this could be done via
collaboration tools.
·
Thirteen of 14 would have liked to learn what the outcome was of the
pol-mil plan when applied to the crisis.
In the future, we want to explore incorporating a simulation to link
participant decisions to outcomes. This
concept is discussed further in the context of JTF staff training in [Loughran
et al., 1999].
·
Poor performance on the quiz. We asked six questions about the PDD
itself and the majority of the training audience filling out the quiz failed to
answer them correctly. This information
was contained in the PDD-56 handbooks distributed at the training event. Although the questions covered very basic
material, for example, what occurs when PDD-56 is invoked, one explanation for
the poor performance might be that not much time was spent covering these areas
in the actual training event.
Observations and participant feedback from March
exercise identified the following potential improvements for future PDD-56
training events, irrespective of the use of technology:
·
Presence of a facilitator/trainer in each working group
·
Need for training aids, e.g., a checklist of items for different
functional groups when developing their group’s response to the crisis, a
glossary of terms, etc., and
·
Need for increased feedback to the training audience on their
performance.
3.4 Plan for the fall experiment
We will conduct a one to three day experiment in the
fall to explore the ways in which DICE can be used and where it is most useful
for PDD-56 training. This section
describes the fall experiment in more detail.
Note that the experiment plans will continue to evolve during the next
few months.
3.4.1 Assumptions for Using DICE
Based on our observations of the March exercise and
on feedback from the training audience, we are keeping the following precepts
in mind while planning the fall exercise.
·
Face-to-face interaction is critical for the training audience and must
be maintained at some level. This
interaction promotes greater understanding of other agencies’ cultures,
responsibilities, and limitations, and promotes networking among
participants. This means that the fall
exercise cannot be entirely distributed.
·
Shortening the total length of the exercise is a concern of many
participants. One to two days was the
preferred length, from surveys we did of a small subset of the audience after
the exercise.
·
Emphasizing active learning, which corresponds to an information pull
approach, is thought to be a more engaging and therefore better form of
training. The ratio of passive learning
(or information push) to active learning, shown earlier in Table 3.1, was high
in the March exercise.
·
Providing feedback to the training audience is important for their
learning.
·
Providing a more realistic information environment to the participants
is good. In real life, people who
develop a pol-mil plan have access to a variety of information about the
contingency, which might include intelligence reports, State Department cables,
regional expertise within an agency, and open source information, most notably
on the Internet.
3.4.2
Use of DICE
This section explores how DICE can be used to
enhance the curriculum/structure of the existing PDD-56 training program. The focus of our research is on the
benefits and/or pitfalls of DICE and not on how a new curriculum might enhance
the PDD-56 training.
The point of the fall experiment is to learn how
DICE can most effectively support interagency training. The key questions are: which parts of the
training are most effective in a face-to-face setting; which parts are best
done in a distributed way (to reduce the total time out of the office or to
speed learning); and how much of the distributed collaborative work can be done
asynchronously vs. synchronously.
The answers to these questions are not
clear-cut. Therefore the fall experiment
will be an exploration of these new techniques and will be a learning
experience for us and for the planners of PDD-56 training. We will encourage the training audience to
explore different ways of using DICE to create work products (e.g., the pol-mil
plan) and for several other parts of the training. Based on their feedback and our observations,
we’ll be able to describe with greater confidence how DICE can be used to
support future official PDD-56 training events.
DICE for
Pre-training.
The training audience will access a DICE web site to
learn the necessary background information for the training event. Information about PDD-56, the scenario, and
agency capabilities will be accessible from a web site, in advance of the
exercise. This information might be in
multiple forms: text, audio, or video.
Hyperlinks will be included to agency web sites and to sites with
additional country information for the scenario. Administrative information for the training:
the schedule, maps to the facility, etc., will also be posted on the web site.
Using the collaboration tools, the training audience
will be able to interact with the exercise planners, who are subject matter
experts in PDD-56, prior to the training.
The planners can answer questions about the PDD, the scenario, or the
training event.
Given the need to supply the training audience with
a certain amount of background information, along with the desire to shorten
the time of at least the co-located training ( if not
the total length of the exercise) using DICE for pre-training seems
useful. In previous exercises, most of
this background information is provided to the training audience in real time
during the exercise. It is probably
faster for them to read this information.
In addition, audience surveys showed that most of the training audience
wanted additional information about agencies, including more in-depth
information than was presented.
However, in practice, there are potentially two
major obstacles to pre-training. First,
the training audience members are very busy with their jobs and have almost no
time for training. Many agencies also
have a culture in which training is not valued.
Once they leave their office and come to the training event, they’re
safe from most work demands. If they have to do part of their training at their
office, it’s quite likely that their real work will have a higher priority and
that part of the training just won’t get done.
Second, the State Department employees, who are a significant block of
the training audience, do not have desktop access to the Internet. Instead, they share a computer with
unclassified Internet access with many other State Department employees. When
they want to go online, they have to leave their office, go to this machine,
and, if someone else is already using it, get in line.
Use of DICE
during the Pol-Mil Plan Development.
It is not clear how to most effectively use DICE
during the development of the pol-mil plan, so we plan to set some parameters
for the training audience, in order to ensure that they will use DICE, but we
won’t mandate how they’ll use it. We
hope that different functional groups will use DICE in different ways (e.g.,
some may collaborate intensively both synchronously and asynchronously to
develop their part of the pol-mil plan, others might work with minimal
collaboration, relying more on asynchronous communication). The variety of uses, and their success, will
help shape how the planners structure future training events.
In addition
to the functional groups using DICE to develop their own sections of the
pol-mil plan, other uses of DICE might include:
·
Reach-back to regional expertise.
When the training audience is co-located (and there will definitely be a
co-located part to their training), they might find it useful to access
regional experts at their agencies and DICE can be used for this.
·
Supporting collaboration across functional groups while they’re working
on separate sections of the plan. There
are areas of overlapping interest across the different functional groups. For example, the military/security section is
interested in the diplomatic section.
DICE should make it easy for functional groups to interact with each
other during plan development.
·
Providing visibility into the interim sections of the plan as they are
developed. The persistent workspace in
DICE will allow different participants to view any functional group’s interim
version of their part of the pol-mil plan.
·
Support exercise planning.
Currently, the planners meet about once every month to coordinate their
activities for the four months or so preceding the exercise. They divide up tasks and communicate the
results of their work by handing out Xeroxes at meetings. Planners from PKI, located in Carlisle,
Pennsylvania, do not attend all planning meetings due to the travel time
required. A distributed collaborative
environment will provide immediate access for all planners to the most
up-to-date copy of their work products (e.g., agenda, list of speakers, list of
training audience members) and will reduce the number of meetings required for
coordination.
·
Operational use. Assuming that
DICE can successfully be used for PDD-56 training, it is a natural extension to
consider its operational use. One
problem that immediately came to light when PDD-56 was invoked for Kosovo is
that most of the pol-mil plan developers had not had the training and did not
know what was expected for the pol-mil plan.
An operational web site, with information about PDD-56 as well as sample
plans from previous operations, would help those new to the process. In addition, the collaboration tools could
help reduce the number of face-to-face meetings and perhaps speed up overall
plan development.
3.4.3 Experiment
Design
The training audience for the fall experiment will
most likely be a subset of the attendees at the March exercise. The total
number of people will be small - between 10 and 15. Since this experiment will be the first
extensive use of technology for this training, it was decided that using a
small group, for a pilot experiment, would mitigate risk, as opposed to
introducing the technology into one of the annual exercises.
The scenario will be a Korean scenario used in an
earlier PDD-56 exercise. The experiment
will take place at NDU and will use workstations already present on the campus.
Given the small size and previous experience of the
training audience, as well as the unstructured nature of the participants’ use
of DICE, the fall experiment will not provide statistically significant
results. This experiment will look at the benefits or disadvantages of using
DICE and that assessment will rely on participants’ perceptions, observations
by subject matter experts, and some comparison of data between the March and
fall training exercises. However, the
comparison between exercises cannot be rigorous; the training events will have
different training audiences, different planners, and different scenarios.
Earlier in the planning process, we had considered
dividing the training audience into two parts and providing one half with the
technology for the training while the other half would not use technology. However, the training audience will be too
small for this and we will encourage them to experiment with different ways of
using DICE. We also considered measuring the participants’ knowledge before and
after the training with DICE, but because the curriculum is changing at the
same time technology is introduced, there would be no way to determine whether
learning is due to the curriculum or the technology.
3.4.4 Measures and Data Collection
Listed below are a series of questions that will be
explored in the fall experiment. We’ll repeat the surveys given to the training
audience in March, asking for their subjective assessments of the
training. We’ll also repeat the March
quiz on the substance of PDD-56. In
addition, the collaboration tools will need to be instrumented so that we can
obtain information about whom is collaborating with whom, which tools are used,
and how often are they used.
·
How do the fall experiment outcomes compare to the March outcomes? Outcomes of PDD-56 training include: the
participants’ level of satisfaction with the training, whether they’re learning
anything new, the amount of networking they do, and the quality of the pol-mil
plan they produce. We asked the March
training audience about their level of satisfaction with the panels and
speakers and whether they learned anything new about PDD-56 and the interagency
process. We gave them a quiz to measure
what they had learned about PDD-56 and we also asked them how much networking
they had done. We could repeat these
measures for the fall and compare them to the March results. Another way of
assessing “outcomes” is to have the planners subjectively evaluate the product
of the fall experiment, which is the pol-mil plan.
·
During pol-mil plan development, do students get more information from
a greater variety of sources? For the
March exercise, the only information the students had was the written scenario
and the video version. In the fall
experiment, the training audience will be able to access information on the
Internet.
·
Does DICE provide for reach-back to office? What’s the effect?
·
Does DICE provide the opportunity for pre-training (reviewing
materials, contacting planners?)? If so,
what’s the effect?
·
Does the training audience learn more?
For example, do they create more sections of the pol-mil plan than in
March or do they write the sections more thoroughly (based on our
observations)?
·
Is the training audience more engaged in the learning experience? This would be a subjective assessment they
would make.
·
How does DICE change the amount of training time? How much time is spent in pre-training, how
much is spent in face-to-face activities and how much is spent in asynchronous
distributed work? In March, we measured
the amount spent in passive and active learning and we will measure that again
in the fall.
·
Does DICE support an evolving, dynamic scenario? What’s the effect?
·
Do the functional groups interact more with each other during plan
development? If so, which tools do they
use for this interaction?
·
What is the reliability of DICE?
How often does it fail? For what
percentage of the exercise is it working?
·
What are the training audience’s subjective assessments of DICE’s
utility to training, ease of use, reliability, and applicability in their
office?
·
How much are the collaboration features used each participant?: # of
VTCs, # of e-mails, ...
·
What is the audience using them for?
e.g., pol-mil plan development, research, lunch plans...
·
How much is the persistent workspace used?: # of URLs visited, # of
documents read, ...
·
Does DICE make it easier for the planners to:
1)
Provide training materials to the audience?
2)
Provide more materials to the audience than they have traditionally provided?
3)
Find the training audience?
4)
Attract a larger audience?
5)
Provide feedback to the audience during training?
6)
Understand what the training audience is doing during training (transparency)?
4. Summary
Distributed collaborative environments hold great
potential for interagency training. The
hypothesis is that these technologies can shorten the total training time
required, provide a richer training environment, and make it possible for participants
to work collaboratively from their home offices. The fall DICE experiment will clarify which
aspects of DICE, applied to PDD-56 training, offer the highest payoff.
Assuming the anticipated benefits of DICE for PDD-56
training materialize, it is a natural extension to consider using the same
technologies and content operationally, to help the USG agencies respond to
CCOs. Indeed, the need to communicate
information about pol-mil development both quickly and easily has become
apparent with the recent invocation of PDD-56 for the Kosovo crisis.
Another application of DICE, for which it was
originally conceived in our FY98 research, is to augment current JTF staff
training. LTG Keene and others at U.S.
Atlantic Command (ACOM)[6], have identified a requirement to train partial JTF
staffs. Approximately one-third of a
designated JTF commander’s core staff will rotate out in a given year. Since CPX training occurs only every 18 to 24
months, there is a need for sustainment training in between CPXs to help train
the new arrivals. Discussions are
underway with the ACOM to define the requirements for partial staff training.
5. References
[Joint Pub 3-07] Joint Pub 3-07, Joint Doctrine for Military Operations Other
Than War, 16 June 1995.
[Loughran et al] Julia J. Loughran, Marchelle M.
Stahl, John Howard Eisenhour, Thomas J. Hammel, Edward Marks, Applying Commercial Gaming Collaboration
Technologies to JTF Staff Training, 1999.
http://www.thoughtlink.com
[Walsh and Harwood] Mark R. Walsh and Michael J.
Harwood, Complex Emergencies: Under New
Management, Parameters, US
[White House Paper on PDD-56] The