The current state, criticisms and future of affective computing software

Kris Helge
School of Library and Information Sciences
University of North Texas
Denton, Texas, United States

Library Student Journal,
January 2008

Abstract

Affective Computing Software (ACS) attempts to detect manifested human affects, and then change a computer’s research and retrieval processes to better locate needed information. It does so by detecting a user’s physiological symptoms, facial features, or via user self-report options. Such detection is completed by using wearable or non-wearable devices. The types of ACS are self-report and concurrent expression. These two types of software apply an augmented transition network which in turn makes decisions and tries to improve a computer’s research and retrieval methods to please the computer user. The software is receiving some criticism because it is not yet a completely reliable and valid tool. Also, in its current form, it presents some ethical concerns. However, as an emerging technology, ACS has the potential to significantly advance the research and retrieval methods of locating information, if this software is improved.

Introduction

Affective Computing Software (ACS) is an emerging technology that is based on the premise that an individual’s physiological symptoms can indicate his or her current affect. The software can be implemented via direct or indirect reporting methods, and despite its current imperfections and ethical concerns, the software may have future implications in the library science field. ACS attempts to determine whether a person is satisfied with currently retrieved information, and if not, it then tries to offer other types of information. Some forms of ACS implement wearable apparatus, while others implement non-wearable mechanisms.

The two types of ACS discussed consist of self-report and concurrent expression. These two forms implement an augmented transition network that allows a computer to make decisions to alter its information retrieval process to appease its user. A plethora of criticisms about ACS are posited by numerous researchers. Further, many ethical concerns are presented during the utilization of affective software. Yet, as ACS is continuously developed, it could greatly improve how library patrons search for and retrieve needed information.

Basis of affective computing software

“A person’s affect is defined as genuine subjective feelings and moods, such as when someone feels happy, sad or scared” (Russell & Carroll, 1999, p.5). A high correlation exists between an individual’s physiological symptoms and an individual’s affect (Norman, Ortony, & Russell, 2003). It follows that physiological manifestations such as elevated heart rate, elevated blood pressure, or increased perspiration may suggest an individual is experiencing specific affects (Langua, 2000). For example, an individual may experience an elevated heart rate or an increase in blood pressure after witnessing an automobile accident.

Because some research suggests a high correlation exists between one’s affective state and one’s physiological symptoms, some researchers are attempting to apply this knowledge and create an ACS that can determine an individual’s affect from his or her physical symptoms. The purpose of this software is to enable information seekers to more easily obtain desired information (Califf, 2003). An example of how this software could be applied is an individual seeking information on a web site that is not user-friendly. When he or she becomes frustrated, his or her frustration is manifested physically as an elevation in blood pressure and as an increase in perspiration. The ACS, then senses such changes in the user’s physiology. Next, the software attempts to locate information more satisfying to a user.

Augmented transition network: how ACS works

Most ACS programs implement an augmented transition network. Augmented transition networks are a series of internal decision trees that help affective software decide what type of information to retrieve, keep, or delete based on various physiological states of a computer user (Davis, 2005, p. 44). The augmented transition network first evaluates a user’s physiological state, a user’s facial features, or input from a user. The network then accesses an internal decision tree to determine an appropriate response. For example, a user views a specific journal to seek information regarding autism. The user’s facial features indicate the user is experiencing frustration. The network then recognizes the manifested frustration, and from its decision tree of options the network suggests a new journal from which a user can perform research (Davis, 2005).

Wearable and non-wearable devices: the hardware of ACS

ACS attempts to sense human stress via sensory devices. These sensory devices consist of either wearable or non-wearable apparatus. The wearable devices are connected to both a computer user and computer. Currently the wearable devices used in laboratories are large and uncomfortable. These wearables may include helmets, finger-clips and blood pressure monitors (Steele & Steele, 2003, p. 238). Yet, researchers at the Massachusetts Institute of Technology (MIT) are currently working on developing wearables with no wires, such as rings, necklaces, watches, or other types of jewelry. These wearables attempt to measure physiological symptoms, such as heart rate, blood pressure, excessive perspiration, or other physiological symptoms. Then, this physiological data is transmitted via a wire or remotely to the ACS. The ACS then attempts to surmise what type of affect an individual is experiencing (Picard, 2000). In the future, computers may detect a user's emotions while a user wears a comfortable, noninvasive item such as a ring.

Facial action coding system

Some of the non-wearable devices that are being developed consist of cameras and a computer mouse. For example, Brunel University researchers are attempting to code facial expressions using a small camera attached to the top of a computer monitor. "The name of this coding endeavor is the Facial Action Coding System" (Califf, 2003, p. 24). Based on small facial movements, the researchers at Brunel believe specific positions of facial muscles indicate specific affects that an individual is feeling. For example, a person experiences frustration due to some unwanted information he or she has located on a computer. The user therefore frowns at a computer, and his or her frustration is manifested by specific positions of various facial muscles. The Facial Action Coding System recognizes the frustration affect experienced by a user due to the specific pattern of muscles revealed in the user's frown. Then, the technology instructs a computer to revise its search pattern so as to retrieve different information, to provide a user with detailed instructions, or to advise the user to take a break (Califf, 2003).

Some empirical research has indicated the Facial Action Coding System produces moderate results concerning validity and reliability. For example, a study conducted by Greimel, Macht, Krumhuber and Ellgring attempted to evaluate whether the technology could determine when subjects were experiencing a joyous affect or a sad affect after tasting either a bitter liquid or a sweet liquid. The results of the study indicate the Facial Action Coding System fairly consistently deciphered when an individual was experiencing joy after tasting a sweet liquid, and when a subject was experiencing a sad affect after tasting a bitter liquid (Greimel, Macht, Krumhuber, Ellgring, 2006). Brunell University researchers are currently trying to develop a Facial Action Coding System which will provide high levels of reliability and validity instead of just moderate levels.

"A disadvantage of using the Facial Action Coding System to observe facial responses over a prolonged period of time is the difficulty of deciding whether facial expressions are expressed in response to a specific stimulation" (Greimel, Macht, Krumhuber, & Ellgring, 2006, p. 268). For example, when an individual's facial muscles indicate he or she is experiencing a sad affect, is this affect a result of not locating any needed information? Or, is the apparent sad affect a result of an unrelated unconscious memory? These currently unanswerable questions are the barriers for today's ACS developers.

Emotion mouse

Another non-wearable that IBM is currently experimenting with is the prototype Emotion Mouse (Langua, 2000). “This computer mouse uses galvanic skin response sensors to measure a change in one’s heart rate, skin temperature, and hand perspiration to attempt to perceive a users affective state” (Langua, 2000, p. 2). “Galvanic skin response is a measure of the conductivity of the skin. There are specific sweat glands (eccrine glands) that cause skin conductivity to change as a result of the galvanic skin response” (Mandryk & Atkins, 2007, p. 330). These sweat glands become active when one’s affect changes. ACS such as Emotion Mouse attempts to detect an increase or a decrease in sweat gland activity to determine if a computer user is experiencing a positive or negative change in affect. If such a change occurs in a sweat gland, the ACS detects the modification and then makes decisions about how to locate information satisfactory to a computer user. IBM is currently attempting to develop an Emotion Mouse that detects physiological symptoms highly correlated to a computer users’ research process, rather than an individual’s physiological responses to other unrelated stimuli. Yet, the perfection of such a device may not come to fruition for several years (Langua, 2000).

Self-report and concurrent expression affective computing software

Picard is currently studying two forms of ACS. These two forms consist of self-report affective computing and concurrent expression affective computing (Picard, 2000). Self-report software utilizes non-wearables and allows a user to communicate his or her conscious affects to a computer (Picard, 2000). For example, self-report software presents a user with a pull-down menu of various affective states. Then, as a user experiences one of these affects he or she reports a specific affect to the software. After a user reports having a negative affect, the software attempts to change a computer's behavior and retrieve more relevant information.

There are advantages and disadvantages to the self-report system. One convenience is a user does not have to wear an accessory which measures his or her physiological symptoms. A second benefit is the user is in complete control (Picard, 2000). As a result, an annoying help wizard which many computer programs offer does not interrupt a user and perform or suggest some unwanted function (Davis, 2005). Subsequently, the user has the option of changing a computer's research processes only if he or she wants the software to make such a change. For example, as a user is experiencing difficulty with a specific task, he or she has the option of reporting this difficulty to the self-report software, or the user can continue to attempt to solve this problem him or herself.

There are also some disadvantages involved with the self-report software. One such disadvantage is users must take the time to report the problem to the computer (Picard, 2000). This self-reporting can increase the length of a project. Also, the software may misinterpret what a user is attempting to convey and create a more complex problem for a user by making an incorrect adjustment (Picard, 2000). Another disadvantage is that users may have difficulty deciding which affect to choose (Picard, 2000). For example, if a user is having difficulty opening a website, they may experience numerous negative affects, such as anger, frustration, embarrassment, and impatience. It follows that eventually a self-report system needs to be developed that could deal with multiple human affects at one time.

The second type of ACS Picard discusses is called concurrent expression affective computing. Concurrent expression affective computing can be implemented with or without wearable devices. Concurrent expression software attempts to sense affects experienced by a user via a user's physiological state or facial muscles without requiring a user to report such affective experiences. Physiological symptoms may be sensed via wearable devices such as rings or earrings, or by a computer keyboard sensor. However, such comfortable wearables have not yet been perfected. More awkward wearables such as wires attached to a user's body and to a computer have been used in trials with concurrent expression software. Also, various positioning of facial muscles may be sensed by a non-wearable camera (Picard, 2000). The obvious preference would be to implement this system without any wearables.

There are potential benefits and negative ramifications with this system as well. One potential benefit is this system is more natural for a user (Picard, 2000). In other words, a user does not have to discontinue a task and report a negative affect; instead the ACS attempts to sense a negative affect, and then adjusts the computer's research processes to match the affect. Another benefit is a user does not have to attempt to articulate multiple affects (Picard, 2000). Instead of a user implementing a drop-down box to choose a specific affect, the software attempts to sense a user's multifaceted affects.

Some potential negative consequences exist with the concurrent expression software. One such ramification involves users feeling uncomfortable about a computer sensing unconscious affects (Picard, 2000). For example, a user may not be cognizant he or she is experiencing a specific affect (Picard, 2000). As concurrent expression software proposes a change in a computer process, the user oblivious to his or her affect becomes angered by the intrusive suggestion. Also, a user may not wish to admit he or she feels embarrassed about not being able to operate a specific task on a computer. When concurrent expression software suggests that a user is experiencing embarrassment, this may make a user upset and more embarrassed.

Concurrent expressions also pose the possibility of the software misinterpreting an experienced affect and may further complicate the user by making unnecessary corrections (Picard, 2000). Such a misinterpretation may occur as a user has completed exercising and he or she then immediately uses a computer with concurrent expression software to locate information. While using this computer, the user's heart rate is elevated and the concurrent expression software mistakenly assumes a user is dissatisfied with the garnered information. The concurrent expression software then prevents a user from viewing desired information, or annoyingly suggests other unwanted information. Such a misinterpretation is a current barrier to success for affective technology.

Criticisms of affective computing software

Numerous criticisms regarding ACS have been posited. One such criticism from Muller (2004) suggests that researchers attempting to create ACS are inappropriately viewing a computer from a gestalt perspective. Therefore, these researchers are not correctly separating the parts from the whole (Muller, 2004). For example, computer users experience frustration for a variety of reasons. Frustration may be experienced due to a malfunctioning mouse, a faulty keyboard, inefficient software, a lack of computer knowledge, or a multitude of other variables. Sometimes, ACS cannot decipher between these different variables which may be causing a user frustration (Muller, 2004). For example, if a mouse is functioning improperly a user will demonstrate a negative affect. However, the ACS may not be able to perceive a defective mouse. Instead, the software may decide the user is deterred due to a lack of interesting Web pages being presented. Consequently, the ACS presents a user with new information. Such a presentation is an invalid function of the software's ability to determine what is causing a user's negative affect.

It follows that future research and development regarding ACS should include attempting to develop software that can determine which specific variable or variables are creating frustration for a user. The purpose of ACS is to provide desired information to a user. Currently ACS is not able to reliably and consistently do so. As a result, ACS is not serving its purpose.

Muller also offers a critique of the research performed by one of the leading researchers in the ACS field, Picard. Muller suggests Picard is using invalid experimental designs in her research (Muller, 2004). Muller asserts Picard has only focused on one possible variable in user frustration in her ACS research, without controlling the other variables (Muller, 2004). Muller argues Picard did not take into account other research variables such as anger, joy, surprise, and despair. It follows that when a type of ACS is reacting to an elevation in blood pressure, this software automatically assumes the user is frustrated. The software is not designed to consider whether a user is excited, sad, or surprised. Nevertheless, Muller acknowledges Picard is currently implementing other research designs that control for other human affects such as surprise, despair, and other similar affects (Muller, 2004). Muller's critique may have validity and may suggest that future research in the affective computing realm could focus on a multitude of variables so that software can be developed to decipher which human emotion is being manifested. Further, Muller's critique seems to convey ACS is not yet providing users what it promises to offer. ACS is not yet able to accurately sense human affect and then consistently present needed information to users.

Ethical concerns regarding affective software

The Department of Advanced Research Projects Agency (DARPA) has increasingly implemented affective technology after terrorists attacked the United States on September 11, 2001. This implementation of ACS has brought to fruition some ethical concerns. Some of these ethical concerns include invasions of privacy, lack of autonomy and problems associated with informed consent. These concerns are manifested as government officials utilize ACS, and the same concerns could surface if librarians attempt to use this type of software.

Autonomy

DARPA's utilization of ACS has highlighted a concern of undermining one's autonomy. "Autonomy is the need to fully and authentically endorse one's behaviors, and to act as the originator of one's own behavior" (Patrick, Knee, Canevello, & Lonsbary, 2007, p. 434). DARPA has devised a Head Up Device (HUD) which attempts to selectively feed the affective conditions of selected soldiers during combat to a superior officer. Based on the affective situations reported from the HUD, a soldier is instructed by his or her superior to make a specific move. Yet, the question arises, if the affective measurement is inaccurate, will such an imprecision cost a soldier his or her life? Is a soldier better off maintaining his or her autonomy and determining his or her own maneuvers rather than trusting his or her life to an ACS (Reynolds & Picard, 2004)?

Although the military example is extreme, such situations which may compromise autonomy arise with other possible uses of ACS. For example, a library patron who is fairly proficient in legal research is using a computer that implements ACS to attempt to locate information regarding probate law. The library patron locates a specific legal resource and wishes to obtain it. However, the affective technology convinces a patron to select a different legal resource. In this situation, the patron's autonomy has been violated because he or she did not select the legal information he or she desired. Instead, he or she chose the legal information the ACS suggested. As a result, the patron's independent ability to select his or her own legal materials has been forfeited for a non-human software's suggested information. The fact that affective technology has an opportunity to override one's preferences is a frightening concept. What causes further alarm is that some patrons exhibit more passive personalities than others. It follows that these submissive patrons may be more susceptible to affective technologies suggestions, and thus they may never acquire the materials they really need.

Invasion of privacy

ACS may also unduly invade one's right to privacy. Most library patrons are aware their e-mail may be read, and if they exhibit a behavior contrary to library policy they may be asked to leave. However, most library customers do not expect their every move to be monitored. Such excessive scrutiny may be a violation of one's basic right to privacy. For example, affective technology could continuously analyze every web-page, electronic database or other resource a patron views. Most users do not want software, a machine or a human observing their every move (Reynolds, & Picard, 2004). Moreover, users do not want to leave an information trail which can be viewed by any person.

For example, a library patron wants to obtain legal resources regarding probate law. He or she implements a computer containing ACS. This software is observing every site he or she examines. Consequently, the library patron may feel as if Big Brother is supervising his or her search queries excessively and invading his or her right to privacy. This invasion of privacy may lead to a patron experiencing discomfort. Subsequently, such discomfort may compel a library patron to leave. It is not the goal of any library to force patrons to leave. So, for libraries to successfully implement ACS they must do so in a manner in which patrons become desensitized to privacy invasions, or one in which affective technology is not used so invasively.

Informed consent

Another ethical issue manifested when using ACS is informed consent. In a library setting, informed consent suggests disclosing to a patron that ACS is being applied. Further, informed consent proposes a patron comprehends what ACS is and what are its consequences. Also, informed consent implies a library patron voluntarily agrees to subject oneself to affective software (Reynolds, & Picard, 2004). For example, to address the idea of disclosure, a library may post signage at all entrances and at all computer terminals stating ACS is in use. The signage encourages patrons to ask librarians what affective technology involves. Yet, not all patrons read signs posted at a library. Some may be more concerned with quickly getting inside, obtaining one's needed materials, and then leaving.

Dealing with the issues of comprehension and voluntary agreement are different scenarios. Many court cases are filed to determine whether a person comprehended a clause in a contract, or whether an individual signed a document voluntarily. Subsequently, these issues of comprehension and voluntary agreement can be arduous to prove. Nonetheless, a library may attempt to provide a written disclosure to patrons and encourage patrons to discuss any concerns they may have with a librarian. However, concerns are still present as to whether any patron understands what affective technology is, and whether one voluntarily agrees to be subjected to it.

Obviously, the above-mentioned ethical concerns abound with this new technology. However, many new technologies have presented ethical concerns that are eventually ironed out or that individuals learn to live with. For example, when computer monitors were first introduced to the consumer market there was an ethical concern with informed consent to being exposed to radiation. To date, not much proof exists that computer monitor radiation has caused physical or psychological ailments. Also, various forms of current technology invade peoples' privacy. For example, police departments use cameras that forgo individuals' privacy rights to help lessen crime. In these situations, individuals are willing to balance their privacy rights to obtain a safer environment with the implementation of police cameras. Due to the fact some ethical concerns are manifested via the use of ACS, more valid and reliable research should be performed on this type of software. However, ACS should not yet be discarded as a threat or an eternally invalid tool. In the future, researchers may be able to deliver a form of ACS which augments user research in libraries and other information systems.

Future implications of affective computing

Despite the above-mentioned criticisms, if ACS is ever fully functional, its implications could transform the distribution of information. For example, one software application currently under development is termed Suitor. Suitor is an acronym for simple user interest tracker (Langua, 2000). Suitor has the capability of identifying which specific information a user is currently reading (Langua, 2000). After identifying particular information being viewed by a user, Suitor retrieves other Web sites or news information relating to the user's topic of interest (Langua, 2000).

Suitor is a type of ACS being developed which could offer more efficient retrieval of information for library patrons. For example, a library patron searches for electronic journal articles regarding the subject of biology remotely from a computer which uses Suitor. Suitor determines the types of journal articles that contain information regarding biology. Suitor then searches for pertinent articles and recommends several journal articles that contain relevant information. Suitor also has the capability of sending bibliographic information regarding pertinent articles to a patron via RSS feeds, e-mail or by instant updates. It follows that a patron has a vast amount of journal articles from which to select.

Conclusion

In its current state, ACS results in some positive outcomes and in some negative consequences. This software cannot yet determine a user’s exact affective state in a valid manner. Also, researchers are presently attempting to develop ACS which does not require the implementation of uncomfortable wearables. The two forms of ACS discussed, self-report and concurrent expression, both implement an augmented transition network. This network attempts to aid a computer in changing its search and retrieval process to obtain needed information for a user. Many criticisms and ethical concerns abound concerning ACS. Yet, ACS has the potential to augment and improve the search and retrieval processes of researchers. For example, current research has shown ACS implementing cardiovascular measures, electromyography measures, and galvanic skin responses have greatly improved in detecting the difference between positive and negative affects (Mandryk & Atkins, 2007). Next, ACS research must focus on developing software with an ability to decipher between specific affects such as frustration and sadness. Also, current research needs to help develop comfortable wearables users can utilize that can consistently help users locate needed information.

References

Califf, S. (2003). Letting your computer know how you feel. Computer Weekly, 22, 24-25. Retrieved November 1, 2006, from Computer Source electronic database.

Davis, B. (2005). Tell Laura I love her. New Scientist, 188, 42-46. Retrieved November 1, 2006, from Academic Search Premiere electronic database.

Greimel, E., Macht, M., Krumhuber, E., & Ellgring, H. (2006). Facial and affective reactions to tastes and their modulation by sadness and joy. Physiology & Behavior, 89, 261-269. Retrieved May 10, 2007, from Science Direct electronic database.

Langua, F. (2000, March 21). Affective computing. Byte.com, 21, 1-3. Retrieved October 30, 2006, from Computer Source electronic database.

Mandryk, R. L., & Atkins, M. S. (2007). A fuzzy physiology approach for continuously modeling emotion during interaction with play technologies. International Journal of Human Computer Studies, 65, 329-347. Retrieved May 17, 2007, from Science Direct electronic database.

Muller, M. (2004). Multiple paradigms in affective computing. Interacting with Computers, 16, 759-768. Retrieved October 30, 2006, from Computer Source electronic database.

Norman, D. A., Ortony A., Russell, D. M. (2003). Affect and machine design: lessons for the development of autonomous machines. IBM Systems Journal, 42, 38-44. Retrieved October 28, 2006, from Academic Search Premiere electronic database.

Patrick, H., Knee, R. C., Canevello, A., & Lonsbary, C. (2007). The role of need fulfillment in relationship functioning and well-being: a self-determination theory perspective. Journal of Personality and Social Psychology, 92, 434-457. Retrieved May 14, 2007, from PsychARTICLES electronic database.

Picard, R. W. (2000). Toward computers that recognize and respond to user emotion. IBM Systems Journal, 39, 705-719. Retrieved November 1, 2006, from Academic Search Premiere electronic database.

Reynolds, C., & Picard, R. (2004). Ethical evaluation of displays that adapt to affect. Cyber Psychology & Behavior, 7, 662-666. Retrieved April 17, 2007, from Academic Search Premiere electronic database.

Russell, J. A., & Carroll, J. M. (1999). On the bipolarity of positive and negative affect. Psychological Bulletin, 125, 3-30. Retrieved April 11, 2007, from PsycARTICLES electronic database.

Steele, M. M., & Steele J. W. (2003). Applying affective computing techniques to the field of special education. Journal of Research on Technology in Education, 35, 236-240. Retrieved November 2, 2006, from Computer Source electronic database.

Author's Bio

Kris Helge earned a law degree in 2001 from South Texas College of Law. Kris is currently enrolled as a graduate student at the University of North Texas and expects to graduate with an M.L.I.S. in May of 2008. Kris also works at the reference desk at the Baylor Law School Library.

Go to Top