Experiences with Remote Collaborations in Fusion Research*
S. Davis,f D. Barnes,f T. Casper,b R. Fonck,g T. Fredian,d T. Gibney,f M. Greenwald,d D.Greenwood,e P. Henline,a K. Keith,a B. McHarg,a W. Meyer,b J. Moller,b P. Roney,f J. Stillerman,d and G.Wurden c
aGeneral Atomics Corporation, P. O. Box 85608, San Diego, CA 92186
Abstract The magnetic fusion research community has considerable experience in placing remote collaboration tools in the hands of real users. The ability to remotely view operations and to control selected instrumentation and analysis tasks has been demonstrated. University of Wisconsin scientists making turbulence measurements on TFTR  were provided with a remote control room from which they could operate their diagnostic, while keeping in close contact with their colleagues in Princeton. LLNL has assembled a remote control room in Livermore in support of a large, long term collaboration on the DIII-D tokamak in San Diego [2,3]. From the same control room, a joint team of MIT and LLNL scientists has conducted full functional operation of the Alcator C-Mod tokamak located 3,000 miles away in Cambridge Massachusetts . These early efforts have been highly successful, but are only the first steps needed to demonstrate the technical feasibility of a complete "facilities on line" environment. These efforts have provided a "proof of principle" for the collaboratory concept and they have also pointed out shortcomings in current generation tools and approaches. Current experiences and future directions will be discussed.
The magnetic fusion community has a history of collaboration, both nationally and internationally [1-5]. The current US experiments are used as "national facilities" overseen by advisory committees made up of experts from outside laboratories and operated with the participation of outside scientists. In recent years, the community has come to increasingly rely on participation of remotely sited individuals in order to bring together the wide range of expertise needed to study fusion. The effort to more fully enable remote collaborators to perform their research activities demands that both technical and sociological issues be addressed. The excitement of adopting new technologies into our collaborative environments must be tempered by a search for both cost effective solutions and for solutions which fit well with the way people work.
The US Fusion community is fortunate that our primary funding agency, the Department of Energy, is very supportive and forward looking in the area of collaborative technologies. A principal theme of the DOE 2000 initiative is to create "National Collaboratories" which will develop a set of tools and capabilities which permit scientists, engineers, and managers working at a number of DOE sites and facilities to collaborate on solving problems as easily as if they were in the same hall of the same building.
Another enabling feature that has resulted from being connected with the DOE, is the ability to use the Energy Sciences Network (ESNET) as a basis for our wide area electronic communications. This reliable high speed network currently provides connectivity between most major fusion sites at 45 mbs speeds (T3).
This paper reviews the history, current status, and futures of remote collaboration within the fusion research community. From an historical perspective, much of the fusion communitys current capability grew out of the initial remote collaborations of the University of Wisconsin group  on TFTR and the LLNL group on DIII-D [2,3].
II. Early TFTR Collaborations
The TFTR collaborations developed the ability of remote researchers to view a considerable amount of computerized (VAX-based) information about the TFTR experiment . It was possible, for example, to specify an arbitrary selection of plots to be displayed automatically. Updates of graphs, tabular database views, or plots from higher level data analysis programs occurred as soon as data was available. Diagnostic-specific codes could be run automatically as data became available, or, for interactive codes, notification that the data was available could be requested. Information originally noted in the Physics Operator's log books or displayed on overhead transparencies (progress of the current experimental proposal) was organized into computerized databases, and entering this data as a routine part of operations became the norm. For those diagnostics which needed to make control changes depending on experimental conditions, for instance to change sight-lines or move probes, fairly accurate knowledge of the current shot-clock time was critical. The TFTR shot clock emulator was developed to satisfy this need. This program was the first to add the use of sound. Programmable beeps accompanied major clock-cycle timing events. These were very helpful, since they made it less necessary to monitor the clock screen closely.
One of the more important collaborative improvements occurred when TFTR added the ability for remote collaborators to receive network based audio and video. This became feasible when public domain desktop software for both audio and low-resolution video, using low-cost cameras and video capture boards, became available. The software used on TFTR was CU-SeeMe, from Cornell University, and MAVEN, from the University of Illinois; both were primarily used on Macintosh platforms.
The desktop video links involved cameras and microphones at the University of Wisconsin and Los Alamos National Laboratory (LANL researchers were also early remote collaborators), and at several locations in the TFTR control room. Collaborators at any site could receive these transmissions on Macintoshes using ordinary network connections without any other special hardware. A microphone in the control room captured the public address system announcements, including the shot-clock countdown, and provided such a rich source of general information that on-site TFTR staff connected to it on their office computers.
Additional coverage was provided of the daily planning meetings and weekly Physics meetings as well as other meetings which were deemed to be of wide interest to the national fusion community.
III. Remote Operation of Alcator C-MOD
Actual operation of a tokamak from a remote site was first demonstrated when the Alcator C-Mod tokamak, located at MIT, was operated over the internet from a remote control room set up at Lawrence Livermore National Laboratory (LLNL). This control room, developed to support operations on the DIII-D tokamak, employs standards supporting wide area network distributed environments consistent with the design of the Alcator C-Mod data system [2,3]. Setup of the physics parameters, such as plasma current, density, shape, heating power, and active diagnostics was done entirely from the remote site using the same interfaces that would normally be used on site. Control of the engineering subsystems (vacuum, cooling, power supply limits, etc.) was kept under local control, so as to provide appropriate personnel and equipment safety.
The interpersonal communications during this demonstration were one of the most critical components of ensuring successful operation. In general, responsibility for running a tokamak is divided between the engineering and physics teams, with each team being lead by a chief operator. The physics and engineering operators are primarily responsible for communications between the two groups. On the physics side, a session leader is responsible for communications between the physics group and the physics operator and for ensuring that the daily run plan is implemented.
To enable these communication channels, three tools were used in various combinations - ordinary telephones, internet based multicast audio/video communication software, MBONE (discussed in more detail in the next section), and internet relay chat. Once researchers became comfortable with MBONE and chat, the telephone was basically hung up except for instances when an extra communication path was needed. Although the purpose of running Alcator C-Mod from LLNL was to demonstrate remote operation, the run itself was highly successful and important new physics results were produced. The operation also led to important new insight into what capabilities will be needed for routine full remote operations.
IV. The REE Project and DOE 2000
Given the success of the fusion community in advancing capabilities in remote collaborations [1-5,7], DOE through the Distributed Collaboratory Experimental Environments initiative at the Lawrence Berkeley National Laboratory provided funding for the community to develop further capabilities. The project, known as the REE project [8,9], was to develop a testbed Remote Experimental Environment, a "Collaboratory". The testbed was centered around providing a distributed computing environment for the DIII-D tokamak with audio/video communications to support remotely located scientists at Lawrence Livermore National Laboratory, Oak Ridge National Laboratory, and the Princeton Plasma Physics Laboratory. The REE served as a testing environment for advanced control and collaboration concepts applicable to both current and future experiments.
The project not only provided internet based audio and video, but also added web-based camera control so that remote collaborators could select views. Morning planning meetings were again added to the list of broadcast activities. Another audio/video enhancement was the addition of MBONE based tools to the research environment. The MBONE tools  are designed to allow wider desktop access to broadcasts with minimal impact on networks. The network traffic can be allowed to flow worldwide or be restricted to ESNET or even local area networks. The worldwide network of MBONE routers passes the network traffic using point to point protocols, but once the traffic reaches a local area network, it can be multicast on the local networks. This means that as long as one person at a site is "listening" to a particular broadcast, anyone else on the same local area network can "listen" without causing any further network load. A further advantage of this approach is that both broadcasting and receiving sites need only handle one instance of the broadcast leaving/entering their firewall router.
Open System Foundation/Distributed Computing Environment (OSF/DCE)
A central feature of this collaboratory was to test the OSF/DCE software. The OSF/DCE environment includes a secure, open, and distributed operating environment, standard naming services, standard remote procedure calls, an enhanced distributed file service with access from all sites (DFS), and a hierarchy of shared data files with optional automated replication. In this testbed, where applications use remote computational resources, it was necessary to use an asynchronous message/data passing system to provide distributed synchronization of tasks running across the wide area network. The message passing system was implemented by modifying existing codes  to use the OSF/DCE remote procedure calls and formed the basis for other communication needs, such as event synchronization and a distributed task queuing system.
The REE project as a final demonstration culminated in the operation of the DIII-D tokamak from the LLNL Remote Experimental Site. Several simultaneous communication channels were opened between researchers at LLNL and General Atomics. The communication included channels between an LLNL Physics operator and a local DIII-D Chief Operator, between LLNL and local Session Leaders, between local Beam Operations and LLNL Beam Physics staff, plus another set of channels which allowed local paging to be heard at LLNL and LLNL staff to use the DIII-D paging system. With this setup, the remote collaborators were able to direct machine and Neutral Beam operations as well as directly edit and download a number of control waveforms. Collaborators at ORNL remotely analyzed Charge Exchange Recombination data and remotely operated the midplane ASDEX gauge ( a specialized ionization gauge.)
With the recent shutdown of TFTR, the level of interest in collaboration activity at DIII-D has risen dramatically. Much of that collaboration activity become remote in character. The REE project has made a good start at placing necessary infrastructure and sociology in place to enable these activities.
V. International Collaborations
In the international arena, considerable collaborative activity has occured. The International Thermonuclear Experimental Reactor (ITER) project is roughly a decade old, and is currently in its engineering design phase. This activity has required world wide access to rapidly changing information such as CAD drawings, design documents, physics databases, and more. Although some specialized wide area project data management packages have been implemented for this effort, much of the electronic communications have been based on the now standard internet communication tools, such as ftp servers, news servers, web servers, and electronic mail. Similar internet based activities are ongoing with the US-Korean collaboration to design the KSTAR tokamak.
The Japanese experiment JT-60U provides the best example of an ongoing remote experimental collaboration with an international partner. This collaboration has encompassed Los Alamos National Laboratory researchers installing and remotely operating a neutron diagnostic [12,14] and Princeton Plasma Physics Laboratory researchers performing remote analyses on current JT-60U data. While the neutron diagnostic group is allowed to control their detector and access their data, access to other data is provided in a restricted manner. The Japanese have developed a separate computer system, called the Data Link System, which contains only data which is necessary for the collaborative effort. Interpersonal communications with the on-site JT-60U staff is conducted using ISDN based video conferencing systems. There are several such ISDN systems located at JAERI including one in the control room. This communications channel can be implemented with multi-point capability using either the ESNET video conferencing multiport switch or the JT-60U multiport switch. This technology has been used to hold discussions about data analysis, participate in run summary meetings, and in the future is expected to be used to allow researchers to attend the weekly JT-60U Physics meetings. Because of the severe time differences, consideration is being given to recording Physics meetings and then automatically replaying them through the ISDN system during US working hours. As the international network bandwidth increases to be able to support audio/video conferencing, it is expected that these collaborations will shift toward internet based tools.
International collaborations of this nature are likely to become even more wide spread in the future. Developing open but secure access to data systems will require more than implementing the right technology. It will also require a slow, but steady building of trust. This effort will require numerous visits by prospective collaborators before we can approach the stage of actually operating an international tokamak remotely, but this day will arrive.
VI. Web Applications
One of the main enablers of remote collaboration has been the advent of the World Wide Web. This technology has arguably done more than any other to aid in the rapid dissemination of information. Sharing of documents, CAD drawings, machine schedules, machine parameters, and vu-graphs are just a few of the many pieces of information which have been served up by the web. The web has also been used to remotely control hardware and software systems. The DIII-D camera control system is an example of a web based control system. Internet relay chat applications implemented as Java applets can be called up through a web browser to provide monitoring of experiment or computer systems status. The web has also been used by the MIT group to provide periodically updating video snapshots of the machine area and control room. Another interesting use of the web has been to serve up links to recorded audio from meetings. This approach, when coupled with the availability of vu-graphs from the meeting, may help to alleviate some of the time difference problems that remote collaborators suffer.
Remote collaboration is as much about sociology as it is about technology. It is important to provide tools for remote collaborators which they will accept and use. The initial acceptance of audio and video in control rooms and meetings was poor. In fact, cameras were occasionally turned off and audio muted. However, as researchers began to realize that these tools were useful in their own offices, the initial hesitancy turned into a demand for the systems to always be working. Still, meetings are not always planned ahead of time to allow remote participation. Simple choices like reserving a room with video conferencing capability are often overlooked. Handling of vu-graphs is still poor. The low resolution of ISDN based systems mean that it is better if vu-graphs are made available electronically, but this is an administrative headache. Vu-graphs prepared just before a meeting or hand drawn may not make it into the system in a timely fashion.
Another area of weakness is that of impromptu discussions. Hallway, lunchroom, back of meeting room, and after work discussions are a critical part of planning research activities. Remote collaborators tend to miss out on these discussions. A solution might be to provide hallways at both the remote site and the experimental site with connected audio and video so that people passing by could stop and talk. The novelty of such a persistent connection might even stimulate people to hold more frequent discussions of research problems.
A continuing examination of sociological issues needs to remain at the core of our further development of remote collaboration capabilities.
Current collaboration technologies are supporting operations on two major tokamak facilities, DIII-D at GA and Alcator C-Mod at MIT. These technologies will also be incorporated in our new experiments, the National Spherical Torus Experiment (NSTX) at PPPL and the Sustained Spheromak Experiment (SSPX) at LLNL. The fusion community is also considering adoption of the MDSplus system  as a basis for organizing both raw and processed data. This primary benefit of adopting MDSplus as a common system is that it provides a standardized structure for organizing and maintaining data. The ability to find relevant data is made simpler by having a common way of traversing the data structure to see what is available. It also overcomes the tendency for individuals to place their results in private, inaccessible areas which greatly inhibits data sharing. This will also allow an individuals favorite tools to be able to access data within any MDSplus structure, independent of which machines data is being examined or analyzed. This should be a big win for everyone whether they are on site or collaborating remotely. The standardization will allow users to move from one experiment to another without needing to learn a whole new set of tools and data structures.
Further in the future, as technology advances, things can only get brighter for remote collaborations. The long awaited arrival of high definition television should help improve a number of interactions including the transmission of vu-graphs. We need to continue work to improve the audio in our collaborative spaces. Good hands free, full duplex audio with good echo cancellation needs to be the standard. Shared virtual spaces in hallways, near water coolers, or in places where people tend to congregate will encourage informal conversations. As higher bandwidth networks arrive, more desktops will be expected to have audio/video systems available. Video calls using desktop a/v tools will become as common place as telephone calls are today. Systems should become more user-friendly; document interchange should become easier across heterogeneous platforms. Although time zone differences will remain an issue, automated recording and playback of meetings should evolve to the point where both meeting material and meeting a/v should be available on demand. In short, technology is moving to enable us to work together, yet apart, more effectively.
*Work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract number W-7405-Eng-48, by General Atomics under contract number DE-AC03-89ER51114, by the Princeton Plasma Physics Laboratory under contract DE-AC02-76-CHO3073, by Lockheed Martin Energy Systems under contract DE-AC05-84OR21400, by the Los Alamos National Laboratory under contract number W-7405-ENG-36, by the Massachusetts Institute of Technology under contract DE-AC02-78ET51013, and the University of Wisconsin under grant DE-FG02-89ER53296. This work was also funded by the Distributed Collaboratory Experimental Environments Initiative at the Lawrence Berkeley National Laboratory.
 R.J. Fonck, et. al., "Remote operation of the TFTR BES experiment from an off-site location" Rev. Sci. Instrum. 63 (10), October 1992.
 T. A. Casper and W. J. Lennon, "Remote Experimental Site: A Command and Analysis Center for Big Physics Experimentation", Proceedings of the 14th IEEE/NPSS Symposium on Fusion Engineering, Sept. 1991. pp 704-797.
 W.H. Meyer, D.N. Butner, and T. A. Casper, Remote Experimental Site (RES) Release 1.0, Bull. Am. Phys. Soc. 38, 2112(1993).
 S. Horne, M Greenwald, T. Fredian, et. al., "Remote Control of Alcator C-Mod from Lawrence Livermore National Laboratory", Fusion Technology, 32, 152 (1997).
 B. Welch, et al. "Visible and Ultraviolet Light Measurements from Alcator C-Mod", Bull. Am. Phys. Soc. 38,1957 (1993).
 M. Thompson, T. Carroll, T. Gibney, J. Snyder, "Real-Time Remote Access to TFTR Experimental Data", Proceedings of the 15th IEEE/NPSS Symposium on Fusion Engineering, Oct 11-15, 1993, pp. 142-145.
 D. Barnes, S. Davis, P. Roney, P. Henline, T. Casper, M. Greenwald, "Collaborations in Fusion Research: A View to the Future Needs and Tools for Experiments", PPPL Report 3025 (1994).
 T.A. Casper, S.L. Davis, D.E. Greenwood, P.A. Henline, B.C. Howard, et. al., "Distributed Computing Testbed for a Remote Experimental Environment", Proposal to Distrbuted Collaboratory Experimental Environments Initiative, February, 1995.
 G.G. Preckshot and D.N. Butner, "A simple and Efficient Interprocess Communications System for Actually Using a Laboratory Network", 5th Conference on Real-Time Computer Applications in Nuclear Particle and Plasma Physics, San Francisco, CA., 1987
 See: http://www-jt60.naka.jaeri.go.jp