Advancing affective intelligence in virtual agents using affect control theory

Affective Intelligent Virtual Agents (AIVAs) has emerged as a research domain that integrates artificial intelligence, affective computing, and virtual agent technology. This fusion aims to develop interactive systems capable of perceiving, interpreting, and responding to human emotions. Affect Control Theory (ACT), a theoretical framework developed by Heise (1977) [18] and adapted for virtual agent applications by Robillard and Hoey (2018) [34] proposes that individuals unconsciously compare their own affective behavior with that of their interlocutor, forming predictions about the latter. Satisfaction and psychological stress levels are then influenced by the extent to which the interlocutor's behavior aligns with these expectations.In this paper we introduce an AIVA that employs ACT concepts to interpret user text and generate emotionally-aligned responses, facial expressions, and gestures for an animated virtual character, AvataRena, that we are developing to act as a virtual life coach. Using the DeepMoji network, user textual input is mapped to emojis and then to a three-dimensional affect space. We then use prompt engineering to create ChatGPT responses that are moderated by ACT analyses to deliver emotionally-aligned textual and non-verbal responses. This alignment adheres to the principle of deflection within ACT, positing that lower deflection values correspond to heightened positivity in elicited emotions.To validate the model we performed a controlled simulation using 1480 questions derived from counselor-patient interactions [3] to explore differences between prompt-engineered ChatGPT-generated responses with, and without, ACT moderation. Specifically, we found significantly lower deflection measures for the ACT-moderated AIVA responses, indicating that the moderated system adheres more closely to expected affective behavior than unmoderated ChatGPT. This was a large effect (t(1479)=-33.03, p<.001, Cohen's d = 0.862). Future work will investigate whether this promising result transfers to enhanced user satisfaction and response alignment during extended interactions in the life coach setting.