A new Approach to Human-Computer Interaction

– Synchronous Modelling in Real and Virtual Spaces –

Kai Schäfer, Volker Brauer, Willi Bruns Forschungszentrum Arbeit - Umwelt - Technik (artec) University of Bremen Bibliothekstr. (MZH), 28359 Bremen, Germany Tel.: +49/421/218-4833, -4206
image1.gif (6876 Byte)


Concrete models help us to understand, manipulate and communicate complex technical systems. Models of technical systems represent the ideas of the involved users (Fig. 1). In other approaches experts define computational models for documentation, variation, calculation and presentation (Fig. 2). But there is a gap between these worlds. We introduce the Real-Reality concept to bridge this gap. Building and manipulating a scene with concrete objects causes equivalent movements and manipulations of related virtual objects. In the computer a virtual model is defined, which can be used in the known way for computing (Fig. 3). With our Real-Reality approach we try to bring the ease and usability of playing with construction kits into the definition of complex and abstract computer models. 

Fig. 3: Real Reality Modelling of a Conveyor System 

The Concept 

We operate with Complex Objects, which have a real and a virtual part. A data-glove and a tracking-system couple these two parts of the objects. During the modelling process a computer identifies poses in the glove data stream falling within characteristic, previously taught grasp patterns. If this occurs in close vicinity to an object, hand and object are coupled and moved synchronously from a predefined start-position to the place where the grasp is released. The virtual model can be analysed for consequent computer processes. The movements are also recorded for later analysis. This allows the user to define dynamic behaviour for simulation and process programming. While the users act in their natural physical environment with their hands, the computer stands in the background and is ideally not noticed. 

Fig. 1: Physical Model 


Fig. 2: Virtual Model 
(Try a VRML Modeler here)

  Programming by Demonstration for Material Flow

In the running project RUGAMS we apply the concept of modelling and teaching dynamic behaviour to the definition of simulation models for material-flow. We build a model of a manufacturing plant with conveyor and machine models (Fig. 4). The computer analyses the topology of the virtual part of the model and generates  simulation-models that are equipped with a default behaviour for different external simulators (For Robot-Workcells: COSIMIR; For event based material flow: SIMPLE++ and GHOST). The simulated dynamic animates the virtual model in the background projection immediately for evaluation (Fig. 4). Because a predefined behaviour of  elements usually doesn’t meet the users needs, the behaviour can be redefined by demonstration. The user demonstrates the actions at relevant places with variable behaviour with model elements (Fig. 5, 6). A context-knowledge based analysis of the recorded data (written in PROLOG) leads automatically to programs for simulation and process-control.


Fig. 4: Building a Model with Complex-Objects 

Fig. 5: Demonstrating a Branching Decision depending on Object Properties. 

Fig. 6: A Branching Decision depending on the Plant’s State. Red Token mark relevant places. 


Context of Work

This project called RUGAMS* is currently granted by the "Deutsche Forschungsgemeinschaft" (DFG). The idea of modeling with concrete objects has been implemented for the first time in this project. Related projects at the Research Center Work and Technology (artec) currently working on the Real Reality idea are EUGABE and BREVIE.

*Rechnergestützte Uebergänge zwischen gegenständlichen und abstrakten Modellen produktionstechnischer Systeme (Computer supported crossing between concrete and abstract models of production systems)

List of project related papers

Get in contact with me by E-Mail !         mailto: Kai Schäfer 
Last Update: 9. Mai 2000