Use of Real and Virtual Reality in Training Production Engineers
Session: Virtual Environments
Prof Wilhelm Bruns - Bremen University
P.O. Box 330 440, D-28334 Bremen, Germany
bruns@artec.uni-bremen.de

 

 
 

Summary

Based on the combination of real world phenomena and virtual world simulations through a Graspable User Interface a radical new kind of learning environment for vocational training in production engineering is introduced.

Introduction

The aim of this project is to demonstrate the feasability and the advantages of a new kind of learning environment for vocational training in production engineering. Based on the combination of real world phenomena and virtual world simulations through a Graspable User Interface it will be possible to freely change between operations on real physical objects and their virtual counterparts. These Twin-Objects, real and corresponding virtual components, will compose a new kind of complex construction kit which opens a new era of simulation technology: Real and Virtual Reality. An era in which real physical parts have an adequate functional, structural and behavioural description, to build from these components a composite system which again shows the correspondence between the physical and the virtual system. The new learning environment allows the composition of a real system and the concrete demonstration of its behaviour by human hand operations (sensed by a scanning device), synchronously generating the control software which assures this behaviour.

The pedagogical objectives of our project are to improve (1) the understanding of theoretical and practical interdependencies in mechatronic systems, (2) the competence of action in electro-pneumatics and (3) the communication about them on the level of skilled workers and engineers. The objectives will be achieved by a new intuitive User Interface which is expected to improve the learning process in production automation by supporting action oriented learning, social verbal and non-verbal communication and the grasping (understanding) of physical phenomena by really grasping them. The insight into the correspondence between concrete real and abstract virtual structures is expected to promote adequate mental models for the handling of complex technical systems.
 

Bridging Reality and Virtuality with a Graspable User Interface

Up-to-date multimedia projects include high interactivity, virtuality, visualisation and simulation but the peak of research and development is heading towards a new kind of media that combines both physical reality and virtual reality (see CHI’97 and some Internet resources:  Several efforts are being undertaken to improve the concreteness of modelling. One main stream is Virtual Reality (VR) where the aim is to get a better view and feeling of virtual objects by using interface devices like data gloves, head mounted displays or CAVEs (Barfield & Furness, 1995). Improving the appearance of the virtual by immersing users in a computer simulated environments has some severe disadvantages relating interpersonal communication, is very expensive, and thus is not our aim. Another approach frequently referenced to is Augmented Reality (AR) where the user’s view of his physical environment is merged with computer-generated images (Feiner et al., 1993; Milgram et al., 1995). In contrast to the approach followed with BREVIE, the main emphasis of AR lies on enhancing reality with information stored in the computer but not on creating and modifying models synchronously in two different worlds of representation. Although the idea of combining physical entities and virtual objects is investigated by some researchers, their aims as well as their areas of application differ from ours crucially.

Fitzmaurice et al. (1995) propose a Graspable User Interface. They use tracking sensors as physical handles (bricks) for controlling virtual objects. The bricks are located on a flat screen and are logically linked to their visible virtual counterparts, thus moving a brick yields to a translation of the attached object. They propose a new kind of drawing program with the option of using several bricks simultaneously. At MIT’s Media Lab this approach will be improved in the Tangible Media project (Ishii, 1997). Instead of flat 2D models they additionally use stereo vision and 3D geometrical models, displayed on a desk-like device. The movements of physical handles such as cubes or pyramids are mapped to their graphical counterparts. By doing so, graphical user interfaces are enhanced with physical embodiments of their elements.

An open tool for situative learning is proposed by Suzuki and Kato (1995). They developed a graspable programming language, called AlgoBlock. The language consists of several commands having a physical representation. The physical blocks are equipped with an electronic interface. This way the computer can keep track of which blocks are connected. By connecting block by block a sequence of commands can be created and the result is graphically displayed on a monitor. This arrangement supports learning of abstract issues in groups by manipulating concrete artefacts.

These concepts have a sensorisation of physical objects in common. The implementation must provide a model of how to react on the changes, sensed by these objects. Furthermore, these projects employ visual cues displayed on monitors or special surfaces for giving the users feedback of their actions. This is different from our approach where the user’s hands are sensorised and tracked, and a model of how the hand changes the environment is implemented – as the user acts on physical objects which are self-contained and themselves represent the interface, a technical and perceptual dualism is avoided. Keeping the users in a common real space while their actions are seamlessly recorded by the computer has some tremendous advantages: keeping face-to-face communication, natural orientation in the physical space, and intuitive handling of the interface. From the recording of the users’ actions performed with the interface objects, abstract descriptions such as rules and action patterns are derived and translated into formal representations. These data serve as the basis for the creation of simulation models and can be recalled any time if saved persistently in appropriate file formats.

 
 Synchronous Modelling of a Conveyor System

Due to the tight link between real objects and virtual, computer-based models, the approach proposed here is called "Real Reality". More information is available at: http://www.artec.uni-bremen.de/field1/rugams/

Our approach will not only couple geometrical properties of real and virtual objects, but also their internal structure and their behaviour. Furthermore it will be a two-way coupling: the reality can control the virtuality and vice versa. Pedagogically our approach offers a broad continuity of media allowing the appropriate choice of individual and cultural adequate approaches. Pupils who prefer the exploitation of a new knowledge domain by starting with concrete, graspable object manipulations may do so, and emerge from this entrance to more abstract views, symbolic representations or illusionary virtual objects. Pupils who like the abstract thinking, may start on a symbolic, abstract level and advance in a deduction to the concrete reality. Our complex toolkit will support the teacher and the pupil to choose the adequate access, thus supporting action oriented learning.

The domain of production engineering is increasingly requiring a merge of different qualifications: knowledge and experience about physical and material processes, logical and algorithmic control mechanisms, social and organisational structures. Only a learning and working environment supporting the acquisition of this mixed qualification will meet future needs. Isolated learning with a support of conventional multimedia technology, strictly separated theoretical and practical learning in lectures and exercises, strict separation of vocational school and working place may be overcome with our concept.
 

Conclusion

The successful deployment of our concept will depend on the ease of use of our new real and virtual learning environment and on its acceptance by teachers and learners in vocational educational practice.
 

References

Barfield, W. & Furness, T. (1995): Virtual Environments and Advanced Interface Design. Oxford University Press

Brauer, V. (1996): "Simulation Model Design in Physical Environments." Computer Graphics Vol 30, Nr. 4, p. 55-56, ACM Siggraph

Bruns, F. W., Brauer, V. (1996): "Bridging the Gap between Real and Virtual Modeling - A New Approach to Human-Computer Interaction." 2. IFIP 5.10 Workshop on Virtual Prototyping, Arlington, Texas

Bruns, F. W. (1996): "Grasping, Communicating, Understanding - Connecting Reality and Virtuality." AI & Society, 10: 6-14

Cypher, E. (Ed.) (1994): Watch What I Do - Programming by Demonstration. MIT Press, Cambridge, Massachusetts

Feiner, MacIntyre, Haupt & Solomon (1993): "Windows on the World: 2D Windows for 3D Augmented Reality." Proc. ACM Symp. on User Interface Software and Technology (UIST), Atlanta GA, Oct. 1993, pp. 145-155

Fitzmaurice, G. W., Ishii, H., Buxton, W. (1995): "Bricks: Laying the Foundations for Graspable User Interfaces." CHI’95 Mosaic of Creativity, pp. 442-449

Gorbet, M. G., Orth, M. (1997): Triangles: A Physical/Digital Construction Kit. Tangible Media Group, MIT

Ishii, H, Ullmer, B. (1997): "Tangible Bits: Toward Seamless Interfaces between People, Bits and Atoms." CHI´97, Atlanta, Georgia

Ishii, H. (1996): Tangible Media Group Project List. Internet: media.mit.edu

Milgram, Drascic, Grodski, Restogi, Zhai & Zhou (1995): "Merging Real and Virtual Worlds." Proc. of IMAGINA ’95, Monte Carlo, Feb. 1995

Resnick, M. (1993): "Behavior Construction Kits." Communications of the ACM. 36(7), pp. 64-71

Suzuki, H. & Kato H. (1995): "Interaction-Level Support for Collaborative Learning: AlgoBlock - An Open Programming Language." Proc. of the Computer Supported Collaborative Learning (CSCL) Conf., University of Indiana, 1995

Published in:

European Commission
Important Issues in Today’s Telematics Research
Conference & Exhibition, Barcelona 4-7 Feb. 1998
Conference Abstracts, Jan 1998
Page 30-33