Spatial Interaction

This chapter describes the basic architecture of the Jack interactive system. The primary tools available to the Jack user involve direct manipulation of the displayed objects and figures on the screen. With articulated figures, movement of one part will naturally affect the position of other parts. Constraints are used to specify these relationships, and an inverse kinematics algorithm is used to achieve constraint satisfaction. As a consequence of user actions, certain global postural manipulations of the entire human figure are performed by the system. This chapter presents the direct spatial manipulations offered in Jack and shows how constraints are defined and maintained. One particular application of the body constraints is included: the generation of the reachable workspace of a chain of joints. 3D direct manipulation is a technique for controlling positions and orientations of geometric objects in a 3D environment in a non-numerical, visual way. It uses the visual structure as a handle on a geometric object. Direct manipulation techniques derive their input from pointing devices and provide a good correspondence between the movement of the physical device and the resulting movement of the object that the device controls. This is kinesthetic correspondence. Much research demonstrates the value of kinesthetically appropriate feedback [Bie87, BLP78, Sch83]. An example of this correspondence in a mouse-based translation operation is that if the user moves the mouse to the left, the object moves in such a way that its image on the screen moves to the left as well. The lack of kinesthetic feedback can make a manipulation system very difficult to use, akin to drawing while looking at your hand through a set of inverting mirrors. Providing this correspondence in two dimensions is fairly straightforward, but in three dimensions it is considerably more complicated. The advantage of the direct manipulation paradigm is that it is intuitive: it should always be clear to the user how to move the input device to cause the object to move in a desired direction. It focuses the user’s attention on the object, and gives the user the impression of manipulating the object itself.

Introduction and Historical Background

People are all around us. They inhabit our home, workplace, entertainment, and environment. Their presence and actions are noted or ignored, enjoyed or disdained, analyzed or prescribed. The very ubiquitousness of other people in our lives poses a tantalizing challenge to the computational modeler: people are at once the most common object of interest and yet the most structurally complex. Their everyday movements are amazingly fluid yet demanding to reproduce, with actions driven not just mechanically by muscles and bones but also cognitively by beliefs and intentions. Our motor systems manage to learn how to make us move without leaving us the burden or pleasure of knowing how we did it. Likewise we learn how to describe the actions and behaviors of others without consciously struggling with the processes of perception, recognition, and language. A famous Computer Scientist, Alan Turing, once proposed a test to determine if a computational agent is intelligent [Tur63]. In the Turing Test, a subject communicates with two agents, one human and one computer, through a keyboard which effectively restricts interaction to language. The subject attempts to determine which agent is which by posing questions to both of them and guessing their identities based on the “intelligence” of their answers. No physical manifestation or image of either agent is allowed as the process seeks to establish abstract “intellectual behavior,” thinking, and reasoning. Although the Turing Test has stood as the basis for computational intelligence since 1963, it clearly omits any potential to evaluate physical actions, behavior, or appearance. Later, Edward Feigenbaum proposed a generalized definition that included action: “Intelligent action is an act or decision that is goal-oriented, arrived at by an understandable chain of symbolic analysis and reasoning steps, and is one in which knowledge of the world informs and guides the reasoning.” [Bod77]. We can imagine an analogous “Turing Test” that would have the subject watching the behaviors of two agents, one human and one synthetic, while trying to determine at a better than chance level which is which.

Simulation with Societies of Behaviors

Recent research in autonomous robot construction and in computer graphics animation has found that a control architecture with networks of functional behaviors is far more successful for accomplishing real-world tasks than traditional methods. The high-level control and often the behaviors themselves are motivated lay the animal sciences, where the individual behaviors have the following properties: . . .• they are grounded in perception. . . . . . . • they normally participate in directing an agent’s effectors. . . . . . . • they may attempt to activate or deactivate one-auother. . . . . . . • each behavior by itself performs some task useful to the agent. . . . In both robotics and animation there is a desire to control agents in environments, though in graphics both are simulated, and in both cases the move to the animal sciences is out of discontent with traditional methods. Computer animation researchers are discontent with direct kinematic control and are increasingly willing to sacrifice complete control for realism. Robotics researchers are reacting against the traditional symbolic reasoning approaches to control such as automatic planning or expert systems. Symbolic reasoning approaches are brittle and incapable of adapting to unexpected situations (both advantageous and disastrous). The approach taken is, more or less, to tightly couple sensors and effectors and to rely on what Brooks [Bro90] calls emergent behavior, where independent behaviors interact to achieve a more complicated behavior. From autonomous robot research this approach has been proposed under a variety of names including: subsumption architecture by [Bro86], reactive planning by [GL90, Kae90], situated activity by [AC87], and others. Of particular interest to us, however, are those motivated explicitly by animal behavior: new AI by Brooks [Bro90], emergent reflexive behavior by Anderson and Donath [AD90], and computational neuro-ethology by Beer, Chiel, and Sterling [BCS90]. The motivating observation behind all of these is that even very simple animals with far less computational power than a calculator can solve real world problems in path planning, motion control, and survivalist goal attainment, whereas a mobile robot equipped with sonar sensors, laser-range finders, and a radio-Ethernet connection to a, Prolog-based hierarchical planner on a supercomputer is helpless when faced with the unexpected.

Body Modeling

In order to manipulate and animate a human figure with computer graphics, a suitable figure must be modeled. This entails constructing a satisfactory surface skin for the overall human body shape, defining a skeletal structure which admits proper joint motions, adding clothing to improve the verisimilitude of analyses (as well as providing an appropriate measure of modesty), sizing body dimensions according to some target individual or population, and providing visualization tools to show physically-relevant body attributes such as torque loads and strength. In computer graphics, the designer gets a wide choice of representations for the surfaces or volumes of objects. We will briefly review current geometric modeling schemes with an emphasis on their relevance to human figures. We classify geometric models into two broad categories: boundary schemes and volumetric schemes. In a boundary representation the surface of the object is approximated by or partitioned into (non-overlapping) 0-, 1-, or 2- dimensional primitives. We will examine in turn those representations relevant to human modeling: points and lines, polygons, and curved surface patches. In a volumetric representation the 3D volume of the object is decomposed into (possibly overlapping) primitive volumes. Under volumetric schemes we discuss voxels, constructive solid geometry, ellipsoids, cylinders, spheres, and potential functions. The simplest surface model is just a collection of 3D points or lines. Surfaces represented by points require a fairly dense distribution of points for accurate modeling. Clouds of points with depth shading were used until the early 1980’s for human models on vector graphics displays. They took advantage of the display’s speed and hierarchical transformations to produce the perceptual depth effect triggered by moving points [Joh76] (for example, [GM86]). A related technique to retain display speed while offering more shape information is to use parallel rings or strips of points. This technique is used in LifeForms™ [Lif91, Cal91]. Artistically positioned “sketch lines” were used in one of the earliest human figure models [Fet82] and subsequently in a Mick Jagger music video, “Hard Woman” from Digital Productions. Polygonal (polyhedral) models are one of the most commonly encountered representations in computer graphics.

Task-Level Specifications

So far we have been talking about real-time interactive display and manipulation of human figures, with the goal of enabling human factors engineers to augment their analyses of designed environments by having human figures carry out tasks intended for those environments. This chapter explores the use of task-level specifications as an alternative to direct manipulation for generating task simulations. By now, the reader should be convinced of the value of being able to simulate, observe and evaluate agents carrying out tasks. The question is what is added by being able to produce such simulations from high-level task specifications. The answer is efficient use of the designer’s expertise and time. A designer views tasks primarily in terms of what needs to be accomplished, not in terms of moving objects or the agent’s articulators in ways that will eventually produce an instance of that behavior – e.g., in terms of slowing down and making a left turn rather than in terms of attaching the agent’s right hand to the clutch, moving the clutch forward, reattaching the agent’s right hand to the steering wheel, then rotating the wheel to the left and then back an equal distance to the right. As was the case in moving programming from machinecode to high-level programming languages, it can be more efficient to leave it to some computer system to convert a designer’s high-level goal-oriented view of a task into the agent behavior needed to accomplish it. Moreover, if that same computer system is flexible enough to produce agent behavior that is appropriate to the agent’s size and strength and to the particulars of any given environment that the designer wants to test out, then the designer is freed from all other concerns than those of imagining and specifying the environments and agent characteristics that should be tested. This chapter then will describe a progression of recent collaborative efforts between the University of Pennsylvania’s Computer Graphics Research Lab and the LING Lab (Language, INformation and Computation) to move towards true high-level task specifications embodying the communicative richness and efficiency of Natural Language instructions.

Behavioral Control

The behaviors constitute a powerful vocabulary for postural control. The manipulation commands provide the stimuli; the behaviors determine the response. The rationale for using behavioral animation is its economy of description: a simple input from the user can generate very complex and realistic motion. By defining a simple set of rules for how objects behave, the user can control the objects through a much more intuitive and efficient language because much of the motion is generated automatically. Several systems have used the notion of behaviors to describe and generate motion [Zel91]. The most prominent of this work is by Craig Reynolds, who used the notion of behavior models to generate animations of flocks of birds and schools of fish [Rey87]. The individual birds and fish operate using a simple set of rules which tell them how to react to the movement of the neighboring animals and the features of the environment. Some global parameters also guide the movement of the entire flock. William Reeves used the same basic idea but applied it very small inanimate objects, and he dubbed the result particle systems [Ree83]. Behaviors have also been applied to articulated figures. McKenna and Zeltzer [MPZ90] describe a computational environment for simulating virtual actors, principally designed to simulate an insect (a cockroach in particular) for animation purposes. Most of the action of the roach is in walking, and a gait controller generates the walking motion. Reflexes can modify the basic gait patterns. The stepping reflex triggers a leg to step if its upper body rotates beyond a certain angle. The load bearing reflex inhibits stepping if the leg is supporting the body. The over-reach reflex triggers a leg to move if it becomes over-extended. The system uses inverse kinematics to position the legs. Jack controls bipedal locomotion in a similar fashion (Section 5), but for now we focus on simpler though dramatically important postural behaviors. The human figure in its natural state has constraints on its toes, heels, knees, pelvis, center of mass, hands, elbows, head and eyes. They correspond loosely to the columns of the staff in Labanotation, which designate the different parts of the body.

Epilogue

To define a future for the work described in this book, it is essential to keep in mind the broad goals which motivated the efforts in the first place. Useful and usable software is desired, to be sure, but the vision of manipulating and especially instructing a realistically behaved animated agent is the greater ambition. Some of our visions for the near future are presented, not just for the sake of prognostication, but for its exciting prospects and possibilities. Any discussion of the future of software must take into account the extraordinary pace of developments in the hardware arena. Even conservative predictions of hardware capabilities such as speed and capacity over the five year term lead one perilously close to science fiction. Accordingly, predictions of “better, faster, cheaper, more reliable, more fault tolerant, more highly parallel computers” are easy to make but do little to inform us of the applications these fantastic machines will facilitate. Rather, as general purpose computers improve in all these ways, specialized hardware solutions will decrease in importance and robust, usable software and symbiotic human-computer interfaces will remain the crucial link between a task and a solution. Transforming research into practice is a lengthy process, consisting of a flow of concepts from ideas through algorithms to implementations, from testing and analysis through iterated design, and finally transfer of demonstratably workable concepts to external users and actual applications. This entire process may span years, from the initial description of the concept to a fielded system. The publication of initial results often breeds over-optimism and has been known to lead researchers to allow false expectations to arise in the minds of potential users, with unfortunate results. (Automatic machine translation of text, speech understanding, and early promises of Artificial Intelligence problem solving are good examples of premature speculations.) At the other end of the spectrum, however, are concepts which take a long time to work their way into mainstream technological consciousness. (3D computer graphics is a good example where concepts and even working systems pre-dated widespread commercial availability by more than a decade.) So we will attempt to strike a balance in making speculations: while looking toward a long term research plan we will generally consider technology transfer to occur when serious but sympathetic users can experiment and accomplish real work with it.