Affective Computing and Interaction: Psychological, Cognitive and Neuroscientific Perspectives
Christine L. Lisetti, Florida International University, School of Computing and Information Sciences, USA
International Journal of Synthetic Emotions, 3(1), 70-74, January-June 2012
Click for Resource
Fifteen years have passed since Rosalind Picard’s seminal first book on Affective Computing appeared (Picard, 1997; Lisetti 1998). Although in the 1990s the few visionaries interested in making connections between affective phenomena and computing were originally laughing matter in the research community (Picard, 2010), they have succeeded to establish what is now recognized as a dynamic interdisciplinary field of research. Affective Computing is now represented by one dedicated international conference (Tao, Picard, & Tan, 2005), two international journals (Gratch, 2010; Vallverdú, 2010), and many top rated conferences and journals regularly dedicate sessions and special issues to a variety of affective computing topics.
In this excellent book, Didem Gökçay and Gülsen Yildirim have compiled an interdisciplinary collection of chapters, from authors of diverse scientific background, which thoroughly covers the topics in affective computing that focus on interaction (between humans, animals, or computational agents). Gökçay and Yildirim point out the many challenges that exist before we can build affective computers that can recognize and express emotions for more natural, effective, entertaining and healthy human-computer interactions. They make the welcomed argument that there is a need to move beyond the current trends in the field, which have focused on recognition of the user’s psychological affective state with machine learning of psychophysiological measures of user’s emotional signals (e.g. skin conductance, heart rate) according to some pre-determined categories of emotion (anger, happiness).
To that end, the book educates its reader on fundamental knowledge necessary in order to understand the current state of the art, as well as in order to advance it by moving toward computational models of internal affective representations that can account for the “time-varying continuum of emotions” (similar to human emotions). The book is well organized in five sections: Section 1 on foundations of affect from neuroscience, bio-psychology, and psychology perspectives; Section 2 on theoretical and computational models of emotions; Section 3 on non-verbal communication of affect; Section 4 on aspects of verbal communication of affect; Section 5 on existing applications of affective computing in human-computer interaction; and an Epilogue on philosophical implications of creating affective machines. Rather than discussing every chapter in each section, I will highlight a few of the discussions and provide my perspective as to which open research questions in affective computing interactions they most contribute to.
In the foundation Section, Erdem and Karaismailoglu provide a detailed discussion of the neuroanatomical and neurochemical substrates linked with the sensation, expression, and subjective experience of emotions. Specific references are also provided about the neural underpinning of processes of empathy found in ‘mirror neurons’. Because empathy has recently emerged as a particularly interesting topic to model supportive virtual agent capable of establishing rapport and helping users in socio-emotional contexts, these references should prove interesting to researchers modeling empathy. In his chapter, Smith highlights the biopsychology of affect and of implicit (unconscious) and explicit (conscious) motivation. He proposes a “dual process account of human behavior that integrates motivation, emotion, and cognition”. An appendix makes further connections between the mammalian ‘incentive motivation network’ and the brain mechanisms that underlie it. Although of real importance, the unconscious aspect of emotion has only started to be studied in affective computing research (Hudlicka, 2009), and Smith’s bio-psychological theory provides very interesting insights into modeling unconscious emotional processes. The foundation section also includes an absolutely necessary chapter by Gökçay on the various theoretical emotional axes discussed in the psychological literature to differentiate emotions along n dimensions. It focuses on the 2-dimensional (2D) approach which classifies emotions along the two valence and arousal dimensions, and describes in details two different 2D models: Circumplex (Russell, 1980) and PANA (Watson, 1999). It addresses some of the controversies regarding these 2D approaches, including their relation to Ekman’s categorical theory on the existence of six (or seven) basic emotions (Ekman, 1994). Gökçay also provides a neural network model of iterative evaluation processes of emotions that supports the Circumplex and PANA. This chapter on the structure of affect and emotions is extremely important to affective computing on interaction because it is at the core of design decisions for both the computational simulation of emotional generation processes, and targeted results of emotion recognition algorithms, i.e. should the algorithm discriminate the user’s expressions of affect on a discrete or continuous space? Although not explicitly mentioned in this section, an interesting account about these controversies can also be found in Russell and Feldman Barrett’s article (1999), in which they ascertain that the term emotion is too broad to be a single category. They introduce two useful distinctions to address some of the dimensional controversies: a) core affect as the non-directed, always present, most elementary affective feelings, structured by two bi-polar dimensions, and b) prototypical emotional episode as (what is usually referred as emotion), a short-lived complex set of interrelated sub-events directed toward some object, and structured by categories.
In the Section on emotional models and frameworks, Scheutz presents an evolutionary model of affective control that support the survival of biologically plausible affective agents with foraging and conflict resolution tasks in a multi-agent competitive environment. Using the method of synthetic ethology that makes claims about the likely evolution of agents by performing simulation experiments, the author discusses results about how transmission of single signals leads to better performance in social coordination among simple agents rather than complex forms of symbolic schemata-based communication. Without excluding the possibility of more complex control systems evolving to meet the needs of agents situated in highly structured environment or with limiting sensory apparatus, Scheutz further proposes that simple affective control states are very likely to evolve and might be the cause for why evolution created so many simple creatures with only simple signaling communication. The discussion about the evolution of biologically-plausible simple agents is an important one as it addresses issues about how to design increasingly efficient affective agents, without imposing a top-down symbolic-based agent architecture that might unnecessarily slow down the agent performance. Korsten and Taylor chose to discuss a model of emotional appraisal using four different value assessments (current, expected, predicted, and standard). Another emotional framework discussed by Castelfranchi studies a different problem: he argues that, in addition to the detection and recognition of expressive multimodal signals associated emotions, a theory of mind is necessary if a computer system is to understand the user’s emotional states during human-computer interaction. His approach aims at addressing some of the limitations of current computational approaches to emotional recognition (i.e. only considering the expressive signals emitted by the user during an emotion experience) by introducing the concept of “cognitive anatomies” of emotions, which refers to the user’s interpretation of the eliciting stimulus (the meaning given to the event). To make his point about the necessity of mental configurations, Castelfranchi focuses on modeling some anticipation-based emotions (hope, fear, disappointment, relief, joy) and some social emotions (shame, envy, guilt, pity). These emotions typically involve the process of expectations, which in turns involves mental concepts such as the belief that an event will occur, the goal to know whether the event will indeed happen as anticipated. Cognitive anatomies for these anticipated emotions are decomposed explicitly in terms of goals and beliefs, and in terms of quantitative independent parameters (goals have a value to the subject in terms of importance, and beliefs have strength in terms of degree of certainty). But Castelfranchi does not stop here, he further argues for the need for connecting these cognitive anatomies with the notion of somatic subjective experience of emotions. He proposes an explicit symbolic model that attempts to address the ongoing chasm between researchers analyzing affective bodily signals-alone, and those modeling the cognitive aspect of emotions - alone.
The Section on nonverbal communication includes three chapters which provide extensive surveys of the state of the art and future directions on automatic interpretation and synthesis of expressions of emotions. Vinciarelli and Mohammadi’s chapter surveys the “technology of nonverbal communication”, with an account for the main trends in the psychology of nonverbal communication (e.g. gestures, postures, facial expressions, gaze and vocal behavior), and of the state of the art in technology of nonverbal communication from an automatic recognition perspective using signal processing - with a special focus on vocal behavior. Ali Salah, Sebe and Gevers’ chapter on automatic interpretation of affect from facial expressions revisits the issues introduced in the foundation Section about choosing an appropriate scheme to classify facial expressions of emotions automatically. They provide a useful starter’s kit for computer analysis of facial expressions, going over the main computational modules necessary for the tasks, while listing a set of resources available for researchers interested to work on the topic (databases, software tools), as well as suggestions for application of facial expression recognition. A discussion on nonverbal communication would not be complete without mentioning latest progress on facial expression synthesis which is provided by Buciu, Nafornita, and Gordan, who present the latest trends in character and avatar animation.
As mentioned by Gökçay and Yildirim, verbal communication has still not been covered extensively in affective computing and interactions. So in the Section on affect in language-based communication, an account of the social-emotional framework of language development given by Hohenberger might offer an incentive to start modeling affective language acquisition in a similar incremental manner that Breazeal started to model infant social intelligence on an affectively expressive robot (2003). Another mention of the ‘mirror neuron’ system is offered in the appendix, which once again points to interesting findings for researchers interested in modeling empathy. Additional chapters on text-based communication address different topics. The role of intimacy and gender on emotions in mobile phone email in Japan is investigated by Kato, Scott, and Kato. On a different topic, Yildirim and Gökçay draw some fascinating parallels between a set of frequently observed behavioral problems with Computer-Mediated Communication (CMC) and problems experienced by patients with brain lesions. This chapter provides a very novel look at text-based communication and its limiting lack of affective features in CMC, as well as some interesting suggestions for requiring changes in text-based CMC applications. This type of approach could also prove inspiring to study implications of the use of a number of other technology applications and their possible improvements with affective interactions.
The last Section on emotions in human-computer interaction (HCI) gathers various chapters describing applications of affective computing. A literature review of the affective aspects of HCI is provided by Akgüm, Kaplan Akilli and Çagiltay. An account of how affective technology can be designed to assist children with autism spectrum disorders is provided by Welch and her co-authors. The growing importance of computers for our entertainment is emphasized by Sykes who gives an overview of how video-games can elicit emotional experiences as rich as those provided by other entertainment media (film and television). El Nasr, Morie and Drachen, on the other hand, discuss in details design techniques developed by artists to create interactive entertainment that engage the user at a very deep emotional level. They also describe a scientific approach to validate the use of these design techniques (e.g. color and lighting techniques) in eliciting emotions (currently arousal) via a series of experiments measuring the user’s physiological signals (e.g. galvanic skin response, temperature) in terms of arousal. One particularly interesting concept introduced in this chapter is the one of “emotional affordance” which parallels Gibson’s perceptual affordances (1979): “if a perceptual affordance is a perceptual cue to the function of an object that causes an action, then an emotional affordance is a sensory cue to the function of a stimulus that causes an emotional reaction” (El Nasr et al., 2011) (p. 282). The study of emotional affordances has many implications in affective computing and interactions, and a similar approach to El Nasr and her co-authors could be used in many applications of affective computing (e.g. health intervention applications, intelligent tutoring systems).
In conclusion, I found that the book as a whole provides an excellent coverage of the main topics on affective computing and interaction, with nice links between the foundation section and the subsequent sections that expand on the issues introduced early on. Readers from different background will be able to understand the material written from an interdisciplinary perspective, acquire a strong background on the foundations and on the state of art in affective computing, and, hopefully join the fun of brainstorming in years to come on the suggested ideas for future research on affective computing and interaction.
References
Breazeal, C. (2003). Emotion and sociable humanoid robots. International Journal of HumanComputer Studies. 59, 119-155.
Ekman, P. (1994). All Emotions are Basic. In Ekman, P. and Davidson, R. J. (Eds.), The Nature of Emotions: Fundamental Questions (7-19). New York: Oxford University Press.
Gibson, J. J. (1979). The Ecological Approach to Visual Perception. New York, New York: Houghton Mifflin.
Gratch, J. (2010). Editorial. IEEE Transactions on Affective Computing, 3(1), 1-10.
Hudlicka, E. (2009). Book Review: Emotion and Consciousness. International Journal of Machine Consciousness, 1(2), 281-297.
Lisetti, C. L. (1998). Invited Book review of Affective Computing by R. Picard from MIT Press. Pattern Analysis and Applications, 1, 71-73.
Picard, R. (1997). Affective Computing. Cambridge, Mass,: MIT Press, 1997.
Picard, R. (2010). Affective Computing: From Laughter to IEEE. IEEE Transactions on Affective Computing, 1(1), 11-17.
Russell, J. (1980). A Circumplex Model of Affect. Journal of Personality and Social Psychology, 39(6), 1161-1178.
Russell, J. and Fedman Barrett, L. (1999). Core Affect, Prototypical Emotional Episodes, and Other Things Called Emotion: Dissecting the Elephant. Journal of Personality and Social Psychology, 76(5), 805-819.
Tao, J., Picard, R., Tan, T. Affective Computing and Intelligent Interaction: First International Conference. Lecture Notes in Computer Science, Springer, 2005.
Vallverdú, J. (2010). Editorial Preface. International Journal of Synthetic Emotions, 1(1), 1.
Watson, D., Wiese, D., Vaidya, J. and Tellegen, A. (1999). The Two General Activation Systems of Affect: Structural Findings, Evolutionary Considerations, and Psychobiological Evidence. Journal of Personality and Social Psychology, 76(5), 820-838.
Cindy Mason, Stanford University, USA
International Journal of Synthetic Emotions, 3(1), 64-69, January-June 2012
Click for Resource
Introduction
“Without having a scientifically deep understanding of cognition, we can’t create the software that could spark the singularity.” Paul Allen (Allen & Greaves, 2011)
In the recently published Affective Computing and Interaction: Psychological, Cognitive and Neuroscientific Perspectives, edited by Gökçay and Yildirim, we find such spark in the world’s (first?) collection of international research linking contemporary issues in neuroscience, psychology and cognitive science to the social discourse between humans and machines. The contributors to this volume are moving away from conventional wisdom on cognition by focusing on the interaction between humans and machines. The book represents the present and future of the 21st century conversation that is human-machine symbiosis. The sections or topics of the book - affective neuroscience, affect in non-verbal communication, affect in language-based communication and human-computer interaction – when placed together in a single volume, point to a pre-paradigm shift. All of these topics are formed in the context of ideas that overturned or shook-up bedrock concepts in cognition. As such, the sections are springboards for further shift.
The significance of the material presented by Gökçay and Yildirim is that the next level of conversation between human and machine is being created using scientific evidence of the importance and contribution of affect, not just in cognition, but in our lives, health and interactions. The research presented in Affective Computing and Interaction taps into affective research discoveries in neurophysiology, psychology and cognitive science. What’s unusual in Gökçay and Yildirim’s volume, and in the context of affective user interaction in general, is that although each of these disciplines has many components and are often treated separately, for the purpose of considering affectively intelligent dialogue, they are integrated, because each affects the other.
The book is organized as a teaching tool and most chapters are written as tutorials or surveys. Each one follows a similar story-telling format that makes the interdisciplinary material easy to read. Affective Computing and Interaction: Psychological, Cognitive and Neuroscientific Perspectives may also serve as a reference in reviewing basic foundations of emotions in the neuro- and cognitive sciences and psychology, because many of the chapters reference decades of research in their respective fields.
The intended audience is computer science, but studies by Clifford Nass’s team and others tell us we (humans) not only anthropomorphize our machines, but have begun to treat one another as machines, so the book also has significance and relevance to fields outside of computer science, including social anthropology and rehabilitation medicine. The Gökçay and Yildirim book is also of interest to futurists and educators, because it is an example of the likely shape of knowledge to come. It contains a collection of interdisciplinary concepts inspired by a singularity of purpose, and built from the vantage point of a world-wide scientific superbridge that is composed of bridges on top of bridges connecting fields of study that each gives us new discoveries.
Paul Allen reflected that scientific progress is affected in part “by the creativity of researchers in dreaming up new theories. It is also governed by the ways that we socially organize research work in these fields, and disseminate the knowledge that results” (Allen & Greaves, 2011). In Gökçay and Yildirim’s book there is an emphasis of psychology, cognitive science and neuroscience, with a heavy slant towards neuroscience. However, within the context of understanding and building enhanced human machine interactions, it also relates to the fields of sociology, anthropology, education, telecommunications, psychoneuroimmunology, psychoneuroendocrinology, behavioral medicine, preventive medicine and more. The integration of discoveries and innovations across these disciplines is giving rise to new fields of study such as social signal processing (Chapter 7, “Towards a Technology of Nonverbal Communication: Vocal Behavior in Social and Affective Phenomena” by Alessandro Vinciarelli at the University of Glasgow) and psychophysiology (Chapter 3, “Emotional Axes: Psychology, Psychophysiology and Neuroanatomical Correlates” by Didem Gökçay, Middle East Technology University).
Gökçay and Yildirim’s book comes to press at the same time as a similar book by Sherer, Banziger and Roesch entitled A Blueprint for Affective Computing: A Sourcebook and Manual (Sherer et al., 2010). Both books are comprehensive interdisciplinary collections, but, blueprint or not, Gökçay and Yildirim’s book is richer and more tightly connected to neuroscience, where many of the most remarkable discoveries on affect are being made. To Molyneux’s 300-year-old question “Is there a connection between touch and sight?” or more broadly, is there a common association of space among our senses, neuroscientists have provided us with “fossil records” – evidence-based accounts that neurons in cortical regions devoted to one sensory modality in fact respond to two or more modalities (Falchier et al., 2002). Psychologists and philosophers no longer need to debate such questions in abstract terms but can include real evidence from high-density electrophysiology, functional magnetic resonance imaging (fMRI), structural MRI including diffusion tensor imaging (DTI), positron emission tomography (PET) and immune and endocrine measures.
For reasons that may never be clear, emotions and affect were minimized or ignored in a variety of fields, most notably AI. It turns out affect is deeply intertwined with cognition and the brain and consequently, many aspects of our health:
- Rate of wound healing is affected by emotional happiness (DeVries et al., 2007).
- Lower cardiovascular reactivity is related to warm partner contact (Kiecolt-Glaser, 2005).
- Brain glucose metabolism is affected by psychosocial stressors (Kern et al., 2008).
- The creation of neural stem cells governing short term memory and the expression of genes regulating the stress response are positively affected by motherly affect (Meaney et al., 2001).
- Positive cognitive state influences positive immune response and vice versa (Azar, 2001; Davidson et al., 2003; Wager et al., 2004).
In many parts of the world, healthcare is either non-existent or impossibly expensive but phones and other gadgets with interfaces are everywhere and relatively affordable. These discoveries mean that positive affect in human machine interfaces promises to globally widen both cognitive and physical health at all social levels. In the US over 50% of the bankruptcies are attributed to health-care expenses (Bortz, 2009). Thus there are also civic and economic implications of creating positive affect in human-machine interaction. Like the first picture of whole earth sphere, human-machine symbiosis may also change our self-conception. So, I ask you, should we be applying what we know about the brain, cognition and affective computing in interfaces? The answer is yes.
Controversy
In the Epilog, “A Philosophical Perspective on Emotions in Human-Machine Interaction,” Zeynep Basgoze and Ahmet Inam, from Middle East Technical University, provide an opening conversation on the philosophical implications, possibility and plausibility of creating affective machines. Affective computing interaction is a Hubble telescope for the human soul but what will we see when we look through the lens? The technical conquest of affect in human machine interaction promises to rattle us, because it poses us with the most fundamental of questions “What are we?” “Are we part software now?” “Where are we going?” If motherly love can change gene expression, as demonstrated by scientists at McGill University (Meaney, 2001), what happens to us when we succeed in building kindness into affective interfaces?
The field of AI is definitely due for a radical revision, but could this whole idea of creating machines with affect be a wrong turn? In John McCarthy’s story, “The Robot and the Baby” a nanny robot saves a baby by staying calm when the mother panics (McCarthy, 2003). The story successfully makes the point that sometimes it is not useful for robots to have emotions. Disruptive emotions also cause wars and break up families. Yet the potential of positive human emotion is enormous. Survivors of Nazi concentrations camps have described how love and kindness helped them overcome the most incredibly horrible situations and beat the smallest odds (Frankel, 1959). In the growing field of psychoneuroimmunology, scientists are beginning to understand the biological mechanisms of positive and negative emotions on our nervous and immune systems (Urry, 2004). The complete syntax for the relation between emotion and health is still unfolding, but studies indicate mood influences pain perception (Weisenberg, Raz, & Hener, 1998; Cogan et al., 1987), and positive emotions such as mirthful laughter enhance immune response (Lambert & Lambert, 1995; Dillon, Minchoff, & Baker, 1985). There are people (and animals) whose affections and ways of interacting are worth imitating. Emotion Oriented Programming was designed with this in mind (Mason, 2010).
As we find ourselves surrounded by gadgets and constantly plugged in, how brilliant to create human computer interactions that can quell road rage or quiet a murderous thought. Steve Jobs was on the right track when his company brought touch in the human machine interaction. We are born of touch and wither away without out it (Montague, 1986). But are we ready to be touched? Is this a Kurzweilian future?
Description and Structure of the Book
Topics Covered in the book include
- Neurophysiology of Emotion
- Neuroanatomy and Psychophysiology of Affect and Cognition
- Biology and Psychology of Affect and Behavior in Humans
- Processing of Affect in Communication, Body Language, Facial Expressions and Speech
- Processing of Context in Affective Human Machine Interactions
- Representation of Affect and Computational Models
- Affective Software Agent Applications
- Applications of Affective Hardware
- Social and Philosophical Issues in Affective Computer Interactions
In the words of Gökçay’s preface “we tried to bring together several aspects of affective interactions within the field of affective computing in a pleasant scholarly reading format.”
The layout of the chapters flows from learning basic foundations of emotions to understanding the contribution of affect in our lives, and finally ends by revealing the current trends and the promising technologies for defeating the emotional gap between humans and machines, all within the context of interactions.
The book is divided into five sections. In the first section, three chapters are devoted to reviewing the foundations of affect in cognition according to the cognitive science, psychology and neuroscience perspectives. Although there are individual books on each of the topics in this section, the coverage of material here focuses on concepts relevant for building human-like systems. The neurophysiology chapter includes current anatomical drawings and pictures that could be used to teach engineers, scientists, medical students and possibly educators about affect in the brain. The second section presents selected examples of emotional models and affective cognitive frameworks, mostly related to the knowledge provided in Section 1. The next two sections are organized according to interaction context. Section 3, “Affect in Non-Verbal Communication,” covers psychology and technology of non-verbal, face-to-face interactions. Section 4, “Affect in Language-Based Communication,” focuses on affect in verbal communication and text-based machine-mediated human communication. Finally, Section 5, “Emotions in Human-Computer Interaction,” covers current trends and promising technologies for designing affective interactive participation for human-computer environments in the entertainment, gaming and assistive technology industries. Below we include chapter summaries from this section.
Summaries of Selected Chapters
In Chapter 13, “A Scientific Look at the Design of Aesthetically and Emotionally Interactive Entertainment Experiences”, authors Magy El Nasr of Simon Fraser University, Jacji Morie from University of Southern California and Anders Drachen, from Dragon Consulting in Copenhagen, give the reader a look at the science of designing aesthetically and emotionally interactive experiences for the entertainment industry, now a multi-billion dollar industry, with revenues overcoming those of the movie industry (ESA, 2009). The subject is of interest because participants (they no longer consider themselves an ‘audience’) in these environments have come to expect certain aesthetic and artistic qualities that engage them at a very deep emotional level.
Chapter 14, “Bringing Affect to Human Computer Interaction,” by Akgun, Akilli and Cagiltay, focuses on affective issues that impact the success of human computer interaction. The stance of the authors emphasizes a need to redesign computers to adapt to people. Affective issues related to the design processes in the chapter include Social Norms, Apologetic Feedback, and Emotionally Supportive interactions. The chapter provides an extensive literature review and synthesizes a number of important studies related with affective aspects of human computer interaction.
Chapter 15 discusses the application of physiology-based affect-sensitive assistivetechnology during human-computer interaction (HCI) and human-robot interaction (HRI) for children with Autism Spectrum Disorder (ASD). ASD is a group of neurodevelopmental disabilities that can cause significant social, communication and behavioral problems and statistically is now more prevalent than childhood cancer. The authors discuss their experiments working with technology to automatically detect and flexibly respond to affective cues such as eye gaze and social distance of children with ASD within a clinical intervention paradigm. The authors measure physiological indices, including electrodermal signals, skin temperature signals, electromyographic signals and other indices extracted from cardiovascular signals. This chapter is authored by a team of 5 researchers from Vanderbilt University – Welch, Lahiri, Sarkar, Warren, Stone, and Liu, and Karla Conn Welch from the University of Louisville.
In Chapter 16, “Affective Games: How iOpiates Elicit an Emotional Fix,” author Jonathan Sykes from Glasgow Caledonian University makes a case for a future in which video games have become the opiate of the masses using techniques based on play to move us emotionally, much like movies and television do now. Sykes gives an accounting of the relationship between play and affect and the neurological significance of play and presents an engaging review of the present in affective game psychology and technology that explains why games are very likely to continue to increase in popularity.
Recommendation
Presently, commercial off the shelf devices do not reflect conventional wisdom in human computer interaction nor much of the research in affective HCI that has addressed social and emotional aspects of cognition, brain function,or genetics. It can be difficult from today’s vantage point (Facebook, Twitter, Skype, cell phones, texting) to see why this is so. Perhaps in our initial machine design, it was not mathematically necessary because at the time people were not so connected to machines. This is no longer the case. Gökçay and Yildirim’s book is an essential volume to create our future and can spark the move to put happier cognition into our devices and our lives!
References
Allen, P., & Greaves, M. (2011). Paul Allen: The singularity isn’t near. MIT Technology Review.
Azar, B. (2001). A new take on psychoneuroimmunology. Monitor on Psychology, 32(11), 34.
Bortz, W. (2009). Next medicine: The science and civics of healthcare. Oxford, UK: Oxford University Press.
Cogan, R., Cogan, D., Waltz, W., & McCue, M. (1987). Effects of laughter and relaxation on discomfort thresholds. Journal of Behavioral Medicine, 10(2), 139–144. doi:10.1007/BF00846422
Davidson, R., Kabat-Zinn, J., Schumacher, J., Rosenkranz, M., Muller, D., & Santorelli, S. (2003). Alterations in brain and immune function produced by mindfulness meditation. American Psychosomatic Society, 65, 564–570. doi:10.1097/01. PSY.0000077505.67574.E3
DeVries, A. C., Craft, T. K. S., Glasper, E. R., Neigh, G. N., & Alexander, J. K. (2007). Curt P Richter Award Winner: Social influences on stress responses and health. Psychoneuroendocrinology, 32, 587–603. doi:10.1016/j.psyneuen.2007.04.007
Dillon, K. M., Minchoff, B., & Baker, K. H. (1985). Positive emotional states and enhancement of the immune system. International Journal of Psychiatry in Medicine, 15, 13–17. doi:10.2190/R7FD-URN9-PQ7F-A6J7
Falchier, A., Clavagnier, S., Barone, P., & Kennedy, H. (2002). Anatomical evidence of multimodal integration in primate striate corte. The Journal of Neuroscience, 22, 5749–5759.
Frankel, V. (1959). Man’s search for meaning. New York, NY: Washington Square Press.
Grewen, K., Anderson, B., Girdler, S., & Light, K. (2003). Warm partner contact is related to lower cardiovascular reactivity. Behavioral Medicine (Washington, D.C.), 29(3), 123–130. doi:10.1080/08964280309596065
Kern, S., Oakes, T. R., Stone, C. K., McAuliff, E. M., Kirschbaum, C., & Davidson, R. J. (2008). Glucose metabolic changes in the prefrontal cortex are associated with HPA axis response to a psychosocial stressor. Psychoneuroendocrinology, 33(4), 517–529. doi:10.1016/j.psyneuen.2008.01.010
Kiecolt-Glaser, J., Loving, T., Stowell, J., Malarkey, W., Lemeshow, S., Dickinson, S., & Glaser, R. (2005). Hostile marital interactions, proinflammatory cytokine production, and wound healing. Archives of General Psychiatry, 62(12), 1377–1384. doi:10.1001/archpsyc.62.12.1377
Lambert, R. B., & Lambert, N. K. (1995). The effects of humor on secretory immunoglobulins A levels in school-aged children. Pediatric Nursing, 21(1), 16–19, 28–29.
Mason, C. (2010). The logical road to AI leads to a dead end. In Proceedings of the Fourth IEEE International Conference on Self-Adaptive and Self-Organizing Systems (pp. 312-316).
McCarthy, J. (2003). The robot and the baby. Unpublished manuscript. Retrieved October 27, 2011, from http://wwwformal.stanford.edu/jmc/robotandbaby/ robotandbaby.html
Meaney, M. J. (2001). Maternal care, gene expression, and the transmission of individual differences in stress reactivity across generations. Annual Review of Neuroscience, 24, 1161–1192. doi:10.1146/annurev. neuro.24.1.1161
Montagu, A. (1986). Touching: The human significance of the skin (3rd ed.). New York, NY: Harper and Row.
Rosenkranz, M., & Davidson, R. (2009). Affective neural circuitry and mind-body influences in asthma. NeuroImage, 47, 972–980. doi:10.1016/j.neuroimage. 2009.05.042
Scherer, K., Banziger, T., & Roesch, E. (Eds.). (2010). A blueprint for affective computing: A sourcebook and manual. Oxford, UK: Oxford University Press.
Urry, H., Nitschke, J., Dolski, I., Jackson, D., Dalton, K., & Mueller, C. J. (2004). Making a life worth living: neural correlates of well-being. Psychological Science, 15, 367–372. doi:10.1111/j.09567976.2004.00686.x
Urry, H., van Reekum, C., Johnstone, T., & Davidson, R. (2009). Individual differences in some (but not all) medial prefrontal regions reflect cognitive demand while regulating unpleasant emotion. NeuroImage, 47, 852–863. doi:10.1016/j.neuroimage.2009.05.069
Wager, T., Rilling, J., Smith, E., Sokolik, A., Casey, K., & Davidson, R. (2004). Placebo-induced changes in FMRI in the anticipation and experience of pain. Science, 303, 1162–1167. doi:10.1126/science. 1093065
Weisenberg, M., Raz, T., & Hener, T. (1998). The influence of film mood on pain perception. Pain, 76, 365–375. doi:10.1016/S0304-3959(98)00069-4
For many years, researchers in computer science have focused on rational thinking but discarded the contribution of emotions in machine learning, artificial intelligence and decision making. Similarly in cognitive science, emotions have been considered as exclusive processes for the survival instincts of animals. After the evolution of mankind from hominoids, the limbic system, which sustained all the emotional processing has been thought to be downplayed by a superior system centered in the prefrontal cortex, which sustained all the high-level executive functions involving monitoring, attention, conflict resolution, working memory. However in the last decade, through astonishing progress in cognitive neuroscience, our understanding of the sub-processes in emotion and cognition have progressed. Nowadays, emotion and cognition are viewed as complementary counterparts, interacting through complex top-down and bottom-up systems.
In the light of recent findings from neuroscience research, the emotional experience of the end-users in human-computer interaction may be thought over. Particularly, creating, triggering, sustaining and detecting emotions at the site of the end-user as well as the capability to imitate emotions will prevail as active research areas of our times. Designing interfaces considering affective tools as well as aesthetics; handling game designs in such a way that the user engagement is managed via emotional experiences; greeting/rehabilitating users with affective agents; creating agents with beliefs, desires and intentions are the most promising examples of this new trendy approach.
In the current literature, affective interactions most often refer to the involvement of affect in the interaction between a user and a system. In this type of interaction, usually psychophysiological measures such as skin conductance, eye-blink, heart rate, or electrophysiological measures such as eeg, or behavioral measures such as facial expressions, speech prosody, gestures are input to an automated system. This system then tries to classify the psychological state of the user through machine learning or pattern recognition techniques. After inferring the psychological state of the human, the machine can adapt or manipulate its ‘behavior’ or more technically its outputs for communicating more efficiently with its user. Although this is an exciting and still unraveled endeavor, there exists a bigger challenge. Rather than inferring a psychological state from the user inputs by choosing from a set of predefined classes, an affective computing system can instead model the actual affective internal world or an affective internal representation as it exists in a human. This allows for a time-varying continuum of emotions, similar to that of the human. If we want the human-computer interaction to occur in a domain similar to the phenomenal world, this new approach seems to be indispensable. Unfortunately, there are two major obstacles that hinder us from implementing such systems in the near-term: 1. The neuroanatomical underpinnings of the affective processes in the human brain are extremely complex and far from being well-understood, hence the field is not quite ready for developing affective models 2. A dynamical platform to model such a system is hard to implement and validate because affective inputs/outputs should be produced and tested in several different temporal scales, while the affective representations across these temporal scales also overlap.
In this book, we humbly aimed at serving two purposes: First, capturing the current state of art in affective computing within the context of affective interactions. And second, providing fundamental knowledge to facilitate development of future affective systems which dynamically interact using affective internal representations. Sections 1 and 2 serve the second purpose, while sections 3, 4, and 5 serve the first purpose.
In section 1, the foundations of affect are reviewed according to the cognitive science, psychology and neuroscience perspectives. Chapter 1 describes the neuroanatomy and neurophysiology of the limbic and prefrontal systems which participate in the sensation, expression, and subjective feeling of emotions in detail. Chapter 2 presents a dual processing approach within the emotional network of the human brain. According to this approach, dual processing occurs based on implicit and explicit motivations; the implicit part is rooted on the low-level limbic areas whereas the explicit system, which is also normative is centered within the pre-frontal areas. Chapter 3 summarizes the widely accepted emotional axes, valence and arousal, and their emergence from cognitive evaluations, as well as psychophysiological measures.
Section 2 contains remarkable examples of theoretical emotional frameworks or models, mostly built by using subsets of knowledge provided in section 1. In chapter 4, design of affective agents is discussed. Based on evolutionary demands, how to change the behavior of these agents is also highlighted. In chapter 5, neural networks are utilized to model six basic emotions and a subset of social emotions, using current, standard, expected, and predicted values of a situational signal. Chapter 6 models complex social emotions using logic, by replacing expectation and predictions with Belief and Goals.
Sections 3 and 4 focus on the expression of emotion in affective interactions. In section 3, up to date reviews of the current work in the technology of nonverbal communication are provided. More specifically, chapter 7 focuses on vocalizations, gestures, postures, while chapter 8 focuses on automatically decoding facial expressions and chapter 9 focuses on facial expression synthesis with computers. Currently, these are the most popular research areas in affective computing. In section 4, a collection of chapters addressing affect in verbal communication is presented. Unfortunately, applications of affective computing in verbal interactions are scarce. This is why we included chapter 10 on the role of affect in human language development. Understanding the primary roles affect plays in language development might aid in the development of affective language learning systems. Chapter 11 highlights the behavioral problems that emerge in text-based computer mediated communication such as email and chat environments. The viewpoint of chapter 11 is centered more towards western populations. In chapter 12, a case study on mobile phone messages is provided in a Japanese population. From here, we learn that affective messages are modulated differently in a non-western population when text-based environments are used.
Finally section 5 brings together fascinating discussions on affective computing in human-computer interaction as well as current trends and promising technologies for designing interactive environments based on affective interactions. In chapter 13, manipulation of affect in interactive environments such as films and games is discussed in detail. This chapter also presents development of adaptive games based on the user’s psychophysiological measures. In chapter 14, a general overview of affective aspects in human-computer-interaction is provided, especially for designing efficient affective interfaces and feedback messages. In chapter 15, a wonderful case study is summarized for utilizing adaptive artificial agents such as avatars, for the rehabilitation of autistic patients who are known to have affective deficits. The behavior of the machine (a robot arm, or an avatar), can be manipulated during the training of the patient by using the patient’s psychophysiological signals in a feedback loop. Finally in chapter 16, an innovative approach to the presence of emotions in game technologies is provided: ‘How can games elicit emotions in the user ?’, and ‘How does this compare to the emotions elicited in films ?’ are the main questions tackled.
The selected chapters end with an Epilog, which contains a brief philosophical discussion on the possibility and plausibility of creating affective machines.
We would like to conclude by saying that in this collection, we tried to assemble several aspects of affective interactions in an interdisciplinary and pleasant reading format so that the literature in neuroscience, psychology and computing fields can be merged lucidly. We hope that this book will reach out to communities associated with computer science, cognitive science or cognitive neuroscience and help in uncovering and representing the elusive interplay between emotion and cognition in affective interactions.
Didem Gökçay & Gülsen Yıldırım
Ankara, Summer of 2010
“Some emotions don’t make a lot of noise. It’s hard to hear pride. Caring is real faint - like a heartbeat. And pure love why, some days it’s so quiet, you don’t even know it’s there.”- E. Bombeck
Real life affective states felt by real people are complex. It would be so much easier for us, scientists, if they were describable simply by the six prototypical categories of Ekman (anger, disgust, fear, happiness, sadness, and surprise), but they are not. For instance, the first and simplest affective state, which we discover in childhood, is curiosity. When does an automatic affect recogniser need to model curiosity, and how? The affective state regret lacks immediacy, and mostly comes upon reflection. Should affect interfaces model regret? More specifically, why do we researchers feel the need to sense affect at all, what models, expressions and affective states should we focus on, in which context, and how can we do all these?
The best way to seek answers to these questions, and provide an insight as to where affective computing field stands today, is to look at its past and present, and carry forward the knowledge and experience acquired into the future. One of the more recent answers to the question of why we need to sense affect came in the early 1990s, when Mayer and Salovey published a series of papers on emotional intelligence suggesting that the capacity to perceive and understand emotions define a new variable in personality. Goleman introduced the notion of emotional intelligence or Emotional Quotient (EQ) in his 1995 best-selling book by discussing why EQ mattered more than Intelligence Quotient (IQ). Goleman drew together research in neurophysiology, psychology and cognitive science. Other scientist also provided evidence that emotions were tightly coupled with all functions we, humans, are engaged with: attention, perception, learning, reasoning, decision making, planning, action selection, memory storage and retrieval. Following these, Rosalind Picard’s award-winning book, Affective Computing, was published in 1997, laying the groundwork for giving machines the skills of EQ. The book triggered an explosion of interest in the emotional side of computers and their users, and a new research area called affective computing emerged.
When it comes to what to model and sense (which affective states, in which context), affective computing has advocated the idea that it might not be essential for machines to posses all the emotional intelligence and skills humans do. Humans need to operate in all possible situations and develop an adaptive behaviour; machines instead can be highly profiled for a specific purpose, scenario, user, etc. For example, the computer inside an automatic teller machine probably does not need to recognize the affective states of a human. However, in other applications (e.g., effective tutoring systems, clinical settings, and monitoring user’s stress level) where computers take on a social role such as an instructor or helper, recognizing users’ affective states may enhance the computers’ functionality.
In order to achieve such level of functionality, the initial focus of affective computing was on the recognition of prototypical emotions from acted (posed) data and a single sensorial source (modality). However, as natural human-human interaction is multimodal, the single sensory observations are often ambiguous, uncertain, and incomplete. Therefore, in the late 1990s computer scientists started using multiple modalities for recognition of affective states. The initial interest was on fusing visual (facial expressions) and audio (acoustic signals) data. The results were promising, using multiple modalities improved the overall recognition accuracy helping the systems function in a more efficient and reliable way. Starting from the work of Picard in the late 1990s, interest in detecting emotions from physiological (bio) signals emerged.
The final stage affective computing has reached today is, combining multiple cues and modalities for sensing and recognition, and moving from acted (posed) data, idealised conditions and users towards real data, real life, and real people. The attempt of making affect technology tangible for the real world and the real people is closely linked to Ray Kurzweil’s prophecy. Kurzweil predicted that by 2030 we can purchase for 1000 USD the equivalent information processing capacity of one human brain, and by 2060 digital computing will equal the processing capacity of all the human brains on the earth. If computing capacity continues to increase; further advances in high resolution digital imaging, compression algorithms and random access mass storage is achieved; broadband/wired/wireless communication is available worldwide; size, cost, and power consumption of computational/communications hardware continue to decrease; portable power generation/storage advancement continue, then computers will become much more connected to people (and vice versa) than they already are today.
Coupling the new horizons reached in technology and cognitive sciences, the focus of affective computing research is gradually moving from just developing more efficient and effective automated techniques to concentrating on more context-/culture-/user-related aspects (who the user is, where she is, what her current task is, and when the observed behaviour has been shown). In this transitional process, affective computing research is constantly attempting to bridge technology and humans not only for more natural human-computer interaction (HCI) but also for improved human-human interaction by becoming curious on how real-life conditions, tasks, and relationships affect humans, and whether and how the field can positively impact these (e.g., affect sensing for autism).
This book contributes to answering such why, what, and how questions by providing an overview of frameworks and models of affect, highlighting the present and future of affect sensing and recognition, and looking at affect in HCI context. Each section explores the give and take of various aspects, from foundations and background of affect in cognition (Section 1) to theoretical models of affect (Section 2), and how these are used in order to create automatic affect recognisers based on verbal and nonverbal signals (Sections 3 & 4), and how created automatic systems have influenced HCI to date (Section 5). The book concludes with an extremely interesting philosophical discussion on whether it is possible and desirable to create machines that work exactly like humans do.
Evolutionary theory hypothesises that specific expressive behaviours were selected over the course of human evolution because they had adaptive value in social relationships. This hypothesis suggest that the evolution and future of affective computing is, indeed, an ongoing wondrous journey where machines (computers) meet humans, and where key issues and themes of affective communication, such as context, emotion colouring, multimodality, back-channelling, intensity, duration, continuity, and sustainability, evolve over the course of time, and over the course of that encounter. The authors and readers of this book thus become a part of this wondrous journey.
Hatice Günes
Imperial College, UK & University of Technology, Australia
Contact Details / You can directly use the details below.
Didem Gökçay: didem@ii.metu.edu.tr, http://brain.ii.metu.edu.tr
Gülsen Yıldırım: gulsen@ii.metu.edu.tr, gulsen.yildirim@gmail.com
IGI: www.igi-global.com