Project > Project Summary

This project is based on the belief that, to be more accessible to the general population, computers must be more proactive in their interactions with people. In human interaction, someone who waits for each command before making any communication attempt would be regarded as uncooperative and unhelpful. In order for a computer to be more proactive, and, thus, to bear its part of the burden of initiation in interactions, it must have (1) much more real-time information about its user, and (2) algorithms that select actions based on this information rather than simply on user commands. The computer needs information about the user's current and past emotional, motivational and cognitive state as well as the state of the task at hand. A general approach, or theory, is needed to guide the development of algorithms that select appropriate actions based on user and task state. The research proposed here constitutes the next steps in an attempt by a multidisciplinary team at Beckman Institute to develop this capability. The testbed is an environment for hands-on education in science and engineering, using the Lego Mindstorms construction and robotics environment, with children of middle school age. An emphasis will be on developing proactive computing methods for encouraging interest and conceptual development of minority children and females, who often show lower achievement in science.

Although the proposed work is within an educational environment, the methods being developed and studied are broadly applicable and, in fact, constitute a major attack on one of the grand challenges for engineering. This project should serve as an exemplar of the type of work that is needed in other computer-aided situations. Unique characteristics of the project are the close cooperation of researchers from electrical engineering, computer science, artificial intelligence, psychology and science education, the base of research and development that has already been laid by prior cooperative work, and the well-equipped, interdisciplinary environment that the Beckman Institute provides for this type of project.
The proposed work is based on four assumptions: 1) The proactive computer assumption: Particularly for users naïve to a computer environment, but for all users to some extent, a computer can be most helpful if it can initiate communications when the user needs assistance or encouragement rather than simply waiting for user commands. Computer behavior often determines whether or not a person will continue engagement in that environment. 2) The empathic computer assumption: Proactive communication, which is communication based on the user's current state rather than simply in response to requests, requires real-time, close user and task state information; hence, empathic computing. 3) The information accuracy assumption: It is not necessary for the computer to have highly accurate state information, nor to be always optimal in its communications, to be helpful. Through dialogue with the user, misunderstandings and inaccuracies can be overcome. The challenge is to be accurate enough, but how accurate that is we do not yet know. 4) The human-centered action decision assumption: A critical step in achieving proactive computing is to develop a conceptual framework for selecting human-centered computer actions based on user and task states. This requires a combined psychology and engineering approach involving current psychological knowledge and computer learning methods.

Beckman Institute researchers have already made much progress on the sensing of user states. In our preliminary research, emotional state is detected by video analysis of facial expression and of non-lexical characteristics of the speech signal, as well as by prosodic and semantic analysis of the speech. Motivational state is detected by video analysis of postural changes. Some aspects of cognitive state, including what users are attending to and whether they are having difficulties, are detected by eye movement recording. Human gestures and other motions are being tracked. Methods are being developed for producing affective communications: talking heads with appropriate emotional expression and emotionally expressive speech. An existing Integration Support Lab makes it possible to bring these diverse modalities together in a common environment.

Proposed research includes: (1) further development of methods to sense user postures, movements, expressions and speech, (2) analysis and fusion of this information to identify and track user states, (3) task state tracking, (4) creating a corpus emotion- and action-labeled videotapes for use with computer learning, (5) further development of affective communication, (6) development of the basis for human-centered state-based action decisions, and (7) evaluation of computer proaction on human behavior and response.