ThoughtLink Inc. - Linking People and Technology for Extraordinary Results

Remote Expert Concept

Abstract: This concept features a team of experts and one or more teams of "students" at geographically distributed sites. The students and experts interact with each other using various kinds of collaborative technology: audio, video, whiteboarding, and graphics, embedded in a 3D virtual environment. This training architecture provides an infrastructure for collecting training data that can be used as a pattern repository for lessons learned and for tracking individual student and group performance. Applications are in training and education.

Remote Expert

Overview
The Remote Expert concept uses networking, virtual reality, and collaborative technologies to create a shared 3D environment in which a team of experts and groups of students, at physically distributed sites, can interact with each other. Imagine that your PC starts making an alarming grinding noise while you're in the midst of preparing an important presentation. Using your Internet connection (which is still functional), you go to the Web site of your computer vendor and click on "Troubleshooting Hardware Problems". An "expert" in the domain of computer hardware establishes an audio connection with you and brings up a 3D model of your computer that both you (the "student" in this example) and she or he can view. The expert asks you questions about the symptoms of your problem, then, using the 3D model, highlights those parts that you need to check and demonstrates how to detach and re-connect them. Both you and the expert can use a drawing tool to draw on the 3D model or highlight certain parts of it. Part or all of your interaction, both audio and visual, can be recorded, saved, and may be viewed by other experts or students at a later date.

Key Features

The key features of the Remote Expert are:

creating the collaborative environment in 3D space. For most training and education applications of the Remote Expert, there is a physical object or environment that is being studied. The previous example used a 3D model of the defective PC. One can easily imagine a multitude of physical objects for which people receive advice on how to employ or repair them. In a subsequent example applying this concept to military training, the 3D space is a computer-generated battlefield, complete with vegetation, appropriate topography, buildings, US forces and their weapons, and opposing forces and weapons

supporting multimedia annotations by participants. Separate applications already exist that allow people to communicate by:

sharing a whiteboard and drawing on it in multiple colors;
using a chat feature, where text can by typed and shared;
sharing an audio connection; and
using video teleconferencing (VTC), which includes a video connection as well as an audio connection.

In the Remote Expert concept, these functions would be integrated into the 3D world, and either the expert or the student could create annotations, using any of the functions, and attach them to part of the 3D world. See the military training discussion for examples of this.

adding analytic information into 3D space. Instead of limiting ourselves to a strict visual representation of reality in the 3D space, we can create an "enhanced reality" by adding additional kinds of information. For example, if we used the Remote Expert concept to train people to work on electrical power lines, objects in the 3D space (power lines, transformers, resistors, etc.) could be colored to show the current and/or voltage flowing through them.

Additional Features

Given the features described above, it's easy to envision the following additional capabilities.

		multimedia feedback to students. Both the audio and visual portions of the interactive session between the expert and students can be recorded. Selected parts of this, or even all of it, could be given to the student(s) for their future review. This kind of multimedia feedback, that students could review in a vivid 3D world after their initial training, could lead to greater retention of the subject matter.
		creation of a repository of patterns. Teachers and trainers recognize that there are certain areas where students or trainees continually encounter the same set of problems. By creating a repository of patterns, these problem areas can be automatically identified by the system itself and alternative problems can be provided to help the student in that particular area. The repository can also be used by curriculum developers so that course materials are focused in the areas where students seemed to encounter the greatest number of problems.
		extension to a virtual agent. If a pattern repository were created, the role of the Expert could be replaced to some extent by a virtual agent, who could guide students through the most common kinds of problems and solutions. This could substantially lower the cost of providing the Remote Expert service.
		what-if analysis. In domains where simulations exist, and military training is a good example of this, with many combat simulations extant, software could be written to translate the recorded Remote Expert session into data to be fed into the simulation. The data would create the initial conditions for the simulation. Then other outcomes could be explored using the simulation. This is called what-if analysis.

Application to Military Training

Given the features described above, it's easy to envision the following additional capabilities.

The military performs an increasing amount of training through shared 3D spaces. (Distributed interactive simulation [DIS] is one example of a shared 3D space used for training.) In the Army, units travel to one of a small number of training sites that consist of many combat simulators and, in some cases, a team of full-time trainers, called observer/controllers or O/Cs. Training with O/Cs providing feedback has been very successful but will become more challenging in the future as experienced mentor resources in the Army become less widely available. The Army's CCTT program, providing the next generation of combat simulators, has no funding for any O/Cs.

Using the Remote Expert capability - with a team of O/Cs serving as experts and units to be trained in the student role - would allow the Army to maintain a core of highly skilled O/Cs at one (or a few) location(s) who could interact virtually with units performing training at their home stations.

As a sample of the application of the Remote Expert concept, assume a unit has been told they're going to some crisis in a foreign country. They schedule their training and contact a remote O/C (the expert). The unit trains at the nearest training site, which may be their home station, while the O/C remains at his or her home station. The remote O/C reviews their training either in real-time, or after it has been completed, using a variety of tools including: patterns from the CALL database that are similar to the unit's training; analytic tools to create measures of performance; replay of the battle in 2D or 3D; and review of the radio traffic.

The unit and the remote O/C then meet, via video teleconferencing (VTC), for the after-action review (AAR). During the AAR, the key role for the O/C is to identify (or help the unit identify) what they did well, what they did poorly and need to improve on, and what are the critical events or decisions in the battle. Information from the pattern repository may be used to help identify the critical events.

The important events/decisions are reviewed by both parties using either 2D or a 3D space. Either party can annotate the map (in 2D) or objects (in 3D). Annotation can be graphical, or can be done with voice recordings over parts of the battle. Graphical annotations would be similar conceptually to a 3D whiteboard. For example, the O/C could mark phase lines of the battle, routes that the unit should have taken, points at which particular events occurred or should have occurred, etc. An example of the use of voice recordings is to have the O/C narrate parts of the battle, describing the situation, the choices made, and other potential alternatives and outcomes. Then the unit could create another layer of voice recording by describing their perspective on the battle: why they made certain choices and what their information/situational awareness was at the time.

Multimedia feedback, referred to as a Take-Home Package (THP), is created for the unit. This could include: graphs and charts of analytic results; 2D or 3D replays; voice recordings from their training or possibly from other units' training - for example, the unit could be given part of a battle from a unit that performed really well, as a model that they might follow. Video and/or voice recordings of the AAR itself would also be part of the THP. The THP could also include enough information to initialize a simulation that the unit could use for what-if analyses, to explore the effects of different choices they might have made during their training. The THP might be accessible over the Web.

Benefits

		reduced cost. The Remote Expert has the potential to lower the cost of providing training to students, compared to face-to-face training. This is particularly applicable in cases similar to the military, where the "students" have to travel to training sites. Costs to the students include the travel, lodging, and food. Costs may be less over a period of time, but could be high at the outset when establishing the initial capability.
		richer learning environment. The combination of a reasonably realistic 3D space (the degree of fidelity required depends on the specific application) with audio instruction from the expert and persistent annotations will provide much more sensory stimulation and more detailed instruction than many current environments offer.
		richer feedback to students. Full or partial recordings of a training session, including audio and video annotations in the 3D space, will provide rich feedback, particularly in comparison to systems offering only written hard-copy feedback.
		pattern repository. Aggregating training material over time will identify those areas where students typically encounter problems or difficulties. Lessons learned contained in the repository can be used to enhance the training curriculum, provide focused feedback to the student, and support an environment where intelligent agents can provide the necessary feedback to the student.
		reuse of existing sessions for learning by others. Training sessions can be observed in realtime by other students or replayed at a later date, thus allowing one session to be used for multiple students. This may also lower costs, to the extent that it can replace a new session.