Tele-Immersion (TI) involves the construction of a three dimensional (3D), realistic holographic environment where two or more physically separated users can communicate with each other in real-time. Existing video conferencing applications lack this dimensionality and realism. Tele-Immersion is an engineering solution to this deficiency and is made possible by combining state-of-the-art multimedia technologies with high-speed data transfer. Although the basic premise of TI is essentially the same as video conferencing, its applications span a much broader range of fields. Everything from home theater to war will be affected by the implementation of Tele-Immersive technology. TI might ultimately eliminate the need for unnecessary travel, and in effect be equivalent to science fictive teleportation. This essay explains the revolutionary technology that surrounds Tele-Immersion and discusses several potential applications.
Immersed in Reality
The camera pans and zooms in, focusing on the face of a young man as he leans forward over a candlelit restaurant dinner table. An attractive young woman, his girlfriend, sits across the table from him. The man stares at the reflection of the candle’s burning flame as it flickers in the woman’s eyes. Both of them are completely “immersed” in the moment, oblivious of their surroundings. The man slowly reaches into his pocket and pulls out a diamond ring. Without saying a word, he extends the ring towards the woman’s hand and gently slips it onto her finger. A smile dances across his face. A tear emerges from her eye and rolls down her cheek. It was a stereotypical Hollywood romantic scene, minus the acting and slow motion. This moment was real? Almost. The man is in Los Angeles, the woman in Madrid? Yet they are both in the same room.
Confused? Read on. This is just one of the many potential applications of Tele-Immersion.
Introduction
Tele-Immersion (TI) allows two or more geographically separated groups or individuals to communicate with each other in real-time in a shared virtual environment (NTII). Simply stated, Tele-Immersion is the intersection of virtual reality (VR) and video conferencing. However, Tele-Immersion is more than just a sophisticated mode of video conferencing. Whereas video conferencing is limited to moving images displayed on a computer screen and sound emerging from speakers, TI aims to introduce realism and allow people to communicate with “more than just words.” Ultimately, Tele-Immersion will let people move around and converse in a realistic three-dimensional space, make direct eye contact with each other, and even manipulate virtual objects together.
The importance of non-verbal communication in conveying thoughts and ideas and the human desire for shared physical presence cannot be understated or disregarded. With this understanding, engineers and researchers across American universities are at work to make TI a universal reality. Greg Welch, a computer scientist and part of the Tele-Immersion team at the University of North Carolina Chapel Hill, recognized the significance of “being there,” as he succinctly described it [1]. Similarly, TI engineer Kostas Daniilidis at University of Pennsylvania was motivated “…to make something useful for society and for people.” In order to appreciate the engineering efforts underlying Tele-Immersion, one must understand both the engineering principles involved, as well as the potential applications of TI, especially in the realm of communications.
The Engineering Involved
Tele-Immersion is the synergy from a number of engineering feats. Take hi-definition image/video rendering and processing, add a 360-degree virtual surround sound system, introduce a force-feedback mechanism, combine all this with high-speed data transfer and include networking technology in a three-dimensional, simulated virtual reality environment. The result? Tele-Immersion. Thus, the construction of a realistic Tele-Immersive environment brings together many minds from diverse fields of engineering. Although the sheer number of technologies involved makes TI seemingly complex, its functionality can be traced and understood by almost anyone.
At the core of Tele-Immersion is the successful creation of a real-time holographic environment. Generating a holographic environment requires three basic processes: recognizing human presence, tracking human-environment interaction, and finally projecting this information in a realistic 3D form on a stereo-immersive surface [2].
Recognition
Recognizing the user and his/her surroundings is achieved using a “sea of seven cameras” [3]. The cameras are arranged in a semi-circle around the user; each camera is connected to its adjoining partners. This ensemble constructs five separate views, which are later interspersed by a computer algorithm to produce a single, seamless, 3D image (NTII). Alternatively, the Integrated Media Systems Center (IMSC) at the University of Southern California (USC) utilizes a panoramic camera (comprised of five individual cameras and mirrors) to obtain a similar effect [4]. Additionally, the captured image has to maintain realism. This is accomplished by employing two additional cameras that capture lighting, viewer perspective and depth.
Audio recognition is achieved in a manner similar to visual capture, where the cameras are replaced with an array of microphones surrounding the user. The arrangement of these microphones ensures that dimensionality and directionality are preserved, which is important for sound reproduction in the projection stage.
Tracking
Having examined the recognition aspect of Tele-Immersion, the next step involves understanding how the immersive environment tracks viewer perspective and physical movements. Engineers at the University of North Carolina (UNC) adapted 3rdTech’s HiBall Tracker (a device resembling a saltshaker), which is strapped to the viewer’s head [5]. Relative transmissions to and from the infrared light-emitting diodes implanted in the ceiling enable this gadget to keep track of the viewer?s head angle and location [5]. This non-aesthetic tracking device will most likely be replaced in the future by the adoption of an alternative tracking mechanism, such as lasers.
Tracking the user’s hand movements is necessary to simulate the feeling of touch. In turn, establishing the feeling of touch is essential in maintaining Tele-Immersion?s illusion of reality. IMSC’s “virtual-touch” is the haptics (touch-simulation) technology that makes it possible to “reach out and touch someone” [6]. The associated device in this case is a haptics glove, “a network of exoskeletal tendons” [6], worn by the user and connected to the computer via a Universal Serial Bus (USB).
In the end, head and hand tracking are combined and used to calibrate the recognition and projection processes. The cameras are adjusted to zoom, pan or tilt based on the user?s position; likewise, the projection is adjusted accordingly.
Projection
Finally, the digital information gathered from recognition and tracking needs to be projected to the geographically distant user. Successful realistic projection is obtained using eight computer-controlled front projectors and a stereo-immersive surface/screen, coupled with polarized-lens glasses worn by the viewer [5]. The glasses, which are similar to those worn by the audience of a 3D movie, produce a separate image for the left and right eyes, and the brain perceives the composite image as three-dimensional [1].
The result is a 3D, life-size, realistic hologram that maintains lighting and depth, thus accounting for viewer perception. Since the currently used polarized glasses obstruct eye contact, virtual reality pioneer Jaron Lanier claims that they will eventually be replaced with ?autostereoscopic screens? [5]. These screens will incorporate swiveling pixels that track either eye, and produce an accurate image for each [1].
The current standard audio projection in home theater systems is labeled “5.1 audio.” In comparison, the latest audio projection technology to be employed in Tele-Immersive applications “is so powerful, it could be called ‘10.2’”[4]. Equipped with an arrangement of 16 microphones, it is capable of mapping a location?s acoustics, both electronically and acoustically (Goldman, Ultimate). Thereafter, software processing generates 360-degrees of crisp, crystal-clear surround sound. Projection speakers are arranged in a way that gives the sound direction. The user can therefore differentiate sounds projected from different directions. Chris Kyriakakis, an Assistant Professor at the USC School of Engineering, claims: “We can create the equivalent of virtual images with sound” [4].
The aforementioned processes of recognition, tracking, and projection are synthesized to accurately reconstruct a realistic 3D virtual environment based on viewer orientation. However, in order to maintain the perception of reality all the processes need to function simultaneously. Moreover, the processes are interdependent because they calibrate each other. Therefore, digital information needs to be transferred back and forth between great distances between users at phenomenally high-speeds to avoid delay. Without this, the illusion of reality will collapse. The required bandwidth for Tele-Immersion can be found in Internet2, a superior and much faster Internet being developed by the government and universities all over America. It is hoped that with the eventual nationwide implementation of Internet2, Tele-Immersion will become a public reality and reveal a whole new vista of communication prospects.
Tele-Immersive Applications
The extensive nature of Tele-Immersion makes it applicable for communication in almost any line of work or sphere of life. The primary intent of Tele-Immersion is to promote global communication and interaction, and therefore it can be incorporated in various fields such as medicine, engineering, architecture design, business conferencing, education, international relations, political affairs, travel, home entertainment, gaming, and much more. With TI, the possibilities are truly endless. It is interesting to zoom in on a few specific potential applications:
Medicine
Current medical training programs often consist of videotaped surgical procedures. These tapes are considered to be “marginally effective at best” since they lack interaction and only provide a fixed point of view [5]. In medicine, practical experience is invaluable and requires “learning by doing.” Tele-Immersion offers medical students the facility of “immersive electronic books”, whereby they can witness a surgical procedure performed in 3D, change their viewpoint, and use virtual surgical tools to actually perform the operation themselves [5]. Ultimately, Tele-Immersion will allow doctors to perform real-time operations on patients located thousands of miles away. The problem of medical inaccessibility will hopefully disappear and millions of lives will be saved.
Collaborative Conferencing
The advent of the Tele-Cubicle may have already changed business meetings and conferences forever. Tele-Immersion is a cost-effective solution to unnecessary business travel. A realistic 3D environment ensures that thoughts and ideas are communicated accurately (see Fig. 1). Moreover, the implementation of “virtual tools” will enhance the collaborative design process. Architects, for example, could mutually design a building on a sheet of virtual paper; businessmen could sign a virtual contract and engineers could work on a common research project. Additionally, the application of multiple perspectives permits users to collaboratively explore previously inaccessible domains or regions.
Fuelrefuel/Wikimedia Commons
Figure 1: Tele-Immersion provides a cost-effective solution to business conferencing.
Education
Tele-Immersion will redefine long-distance education. Universities and schools will be able to transmit lectures over great distances, in real-time. In a manner similar to medical training programs, immersive electronic books will enrich students’ learning experiences. Haptics will make it possible for students to “touch and feel” works of art in a virtual museum, conduct large-scale experiments, or participate in an international youth conference. Much like the Internet, Tele-Immersion will be an immensely useful resource for scholars of all ages.
War
Even though war seems too abstract a category for TI, there are some probable applications. Consider a US soldier who could seat himself behind the controls of an immersive tank in Los Angeles. Halfway across the world, an actual tank would correspond to the soldier’s control commands. The tank would then convey information about its surroundings back to the Los Angeles military base in real-time. Such a hyper-real war simulation would alter the very concept of war. People would wage real-time virtual wars; real wars with virtual casualties. Similarly, Tele-Immersion could facilitate international relations by enabling Tele-UN-conferences. Political leaders could confront each other in a virtual setting, without the threat of encountering violence overseas.
Home Entertainment
Tele-Immersion will result in a radical transformation of the entertainment industry. Immersive media technology will initiate unexplored “interaction metaphors and cinematic techniques” [7]. Interactive games and movies will achieve a degree of reality, quite unlike anything we have ever experienced before. “On-site” newscasts will provide an entirely different realm of information. Virtual vacations will offer exciting ways to see the world without having to leave home. Extend this notion to a larger audience, and “transmissions could incorporate ballet performances, or sports events” [5]. Imagine watching a basketball game in 3D from a player’s perspective, or from the viewpoint of the basketball. Tele-Immersion might also change the face of the fast food industry. McDonald’s has already expressed interest in TI and aims to establish immersive-booths so people can eat with their families when away from home [1].
Get Ready
Although many of the TI applications mentioned before may seem fictitious, it is only a matter of time before the power of Tele-Immersion will be available to the public. The year 2005 will witness Tele-Immersive use for medical purposes. In 2007, business meetings will be conducted in an immersive environment. Finally, around the year 2010, Tele-Immersion will be available for household use (NTII).
Tele-Immersion is Star Trek’s “Holodeck”, Vanilla Sky’s, “Lucid Dream”, and Minority Report’s “precog vision” brought together in a Matrix-like realism. It will enhance communication opportunities in every conceivable way and blur the boundaries between what we consider to be real and virtual. Tele-Immersion will revolutionize universal interaction and accelerate globalization at a phenomenal pace. Moreover, Tele-Immersive research will push engineers to develop innovative media technology, which will in turn benefit society as a whole.
References
-
- [1] A. Ananthaswamy. “Tele-Immersion: Virtual Reality New Scientist”. Oct. 20, 2000.
- [2] K. Bonsor. “How Holographic Environments will Work”. Internet: http://www.howstuffworks.com.
- [3] G.F. Welch. The Office of the Future and Related Projects Computer Science Department, University of North Carolina at Chapel Hill, Feb. 2002.
- [4] J. Goldman. Ultimate Home Entertainment TechLive, Mar. 7, 2002.
- [5] S. Ditlea. Tele-Reality Computer Graphics World, Jan. 2001.
- [6] J. Goldman. “Computers Get Touchy-Feely”. TechLive, March 6, 2002.
- [7] Brown University Computer Graphics Group. “Tele-Immersion Applications”. An Electronic Book for Surgical Training.
