Effectiveness of embodied conversational agents for managing academic stress at an Indian University (ARU) during COVID‐19 – Nelekar – – British Journal of Educational Technology

INTRODUCTION

Study-related stress is a global problem, and students from different countries show different patterns of behaviours to cope with the stress (Inaç et al., 2021). India is a country with a large young population, where the problem of disturbed mental health among youth is a growing concern (Reddy et al., 2018). According to the statistics published by the National Crime Records Bureau, 10,335 students committed suicide in 2019, 1.8% of which were due to study-related stress, and there was an 80% rise in suicide rates from 2018 to 2019 (NCRB-India, 2019). Additionally, a low awareness of mental disorders, social stigma and the scarcity of mental health professionals are contributing major concerns to overcome for Indian youth. The COVID-19 pandemic has only made this situation worse due to a sudden shift to completely virtual spaces with little scope for face-to-face contact.

The rise of the COVID-19 pandemic has seen a proliferation in the use of digital AI tools (Almalki & Azeez, 2020; Chen & See, 2020). Given reduced social contact as a result of COVID-19, technologies that provide humanlike support will provide both informational and social functions. Conversational agents (CA) constitute one of the upcoming technologies for providing personalised healthcare information and services and tracking and management of chronic medical ailments (de Cock et al., 2020). A CA can be defined as a digital system that supports conversational interaction with users by means of speech or other modalities. To specifically address the lack of face-to-face contact due to COVID-19, we are interested in Embodied Conversational Agents (ECAs) that feature computer-generated visual virtual characters capable of both empathic verbal and nonverbal communication (Car et al., 2020). In education, CAs are better known as pedagogical agents (Richards & Dignum, 2019). Since our focus in this paper is on student health and well-being and related behaviour change, we will use the more generic term.

There are several benefits of using ECAs. There is evidence that for mental health, an ECA is preferred over a human therapist and humans are more willing to disclose their feelings since ECAs are perceived to be less judgemental and biased (Lucas et al., 2017). In education, studies have further shown positive outcomes towards the use of pedagogical agents by females and ethnic minorities who felt less included and more judged by the human teacher (Kim & Lim, 2013). Benefits can include: greater access, anonymity, personalization, consistency, patience, empathy and relationship development (Lisetti, 2012). Ranjbartabar and Richards (2019) found that utilizing an ECA can be more efficient in reducing the students’ study stress than providing the same advice in a written form. ECAs may offer a COVID-safe approach for stress management among undergraduate students. Hence, we sought to tackle the problem of reducing student stress in India by promoting health behaviour change intention using a user-specific ECA. However, there remain issues to be addressed such as personalization to a wide range of individuals (Kocaballi et al., 2019) and potential for bias in design and testing of CAs when applied across different countries and cultures as reported in the bulletin of the World Health Organization (de Cock et al., 2020; Luxton, 2020).

In particular, very little research or application of ECAs has been undertaken in the developing world and findings to date are mostly based on studies in developed countries. To address this disparity and to determine whether ECAs are acceptable and useful technology in developing countries, we have undertaken a study to bridge this gap. In this study, we adapt and apply an ECA for study stress reduction that has been used in a Western context to a higher educational institution in India. The main motivation was to assess stress levels and help alleviate the same by providing information about coping behaviours, eg, daily physical exercise, healthy diet and social interaction.

In this paper, we introduce an ECA for mAnaging stRess at University (ARU) following the work by Abdulrahman et al. (2021a) and adapted to the Indian higher education context. ARU, a Hindi language word which literally translates to “Beautiful like the Sun” to drive us towards a better life, is also built with the intention to drive students towards a better and stress-managed academic life. ARU focuses on four healthy coping behaviours: physical activity, healthy diet, joining study groups, and meeting new people. ARU utilises two strategies: verbal relational cues, based on Bickmore et al. (2005), and explanation. It tailors the recommendations according to the student’s mental state (beliefs and goals) mimicking human reason explanation (Abdulrahman & Richards, 2019; Abdulrahman et al., 2021a, 2021b). ARU considers the beliefs and goals of the student and aids students’ reflection on their barriers, desires, practices and preferences concerning these behaviours. Aimed at empowerment, ARU educates the students, through explanations, about their choices and the relative benefits, while leaving decision-making in their hands.

Key ECA design features (ter Stal et al., 2020), including the physical appearance, colour and speech dialect of the ECA, have been modified to match with the cultural context, so that the agent can engage better with the users. Also the dialogue has been adapted for handling the COVID specific advisory. We conducted a study with ARU to evaluate the stress outcomes, user-agent relationship and behaviour change intention amongst Indian students.

The main research questions that we address in this study are as follows:

Research Question 1 (RQ1) How do we adapt an ECA and apply it to a different cultural context?

Research Question 2 (RQ2) Does the modified ECA improve the stress related outcomes, and does its effectiveness depend on the population or cultural context?

Research Question 3 (RQ3) How do explanation-based recommendations affect behavior change intentions?

Following a review of the literature, we present our methodology, then our results, discussion, conclusions and future work.

RELATED WORK

Major stressors and coping strategies among students

The large-scale impact of COVID-19 has taken a toll not only on physical, but also on mental health. It is the result of nationwide lock downs, self-isolation and post COVID stress that has not only led to further deterioration in the condition of individuals with existing mental health disorders, but has also induced new mental health conditions (Torjesen, 2020). Restricted social interaction, sedentary lifestyle, less exposure to natural light, more exposure to social media and overall pandemic stress have resulted in various conditions like hormonal imbalance, disturbed sleep patterns and obesity. These factors have contributed to a rise in the number of students suffering from various mental disorders like depression and anxiety (Majumdar et al., 2020).

There have been several studies on stressors and coping strategies among Indian high school and universities (Deb et al., 2014; Verma & Verma, 2020). It has been observed that academic and emotional aspects contributed the most towards stress in undergraduate students in India such as worry about examination success and intense competition, parental expectations, peer rivalry, lack of self-confidence or low self-esteem, managing transitions and integrating into college system, uncertainties about the future and fear of wrong career choices and financial and relationship difficulties (Kumar & Bhukar, 2013; Kumaraswamy, 2013). Nandi et al. (2012) reported a high incidence of competitive attitude and jealousy amongst students and a need for better social integration with friends and family. Verma and Verma (2020) studied the factors behind study stress among university students and identified several factors including a lack of support from friends and a lack of extracurricular activities.

A COVID-safe approach for stress identification and management among undergraduate students can be the use of the ECA as an advanced tool for human-computer interaction, which involves the combination of ECA dialogues, body movements and facial expressions (Core et al., 2006). It has been shown that with appropriate dialogue management and appearances, ECAs are able to establish good levels of trust and sense of bond with the user (Lucas et al., 2014). It has been observed that users who develop a bond with ECAs tend to share their true emotions and disclose more information to the virtual CAs (DeVault et al., 2014; Harman et al., 2014).

The use of ECAs in mental health is increasingly being promoted in different research studies such as Ellie for assessing anxiety, depression and post-traumatic stress disorder (DeVault et al., 2014) and Sarah for reducing study stress (Abdulrahman et al., 2021a). There are also examples involving CAs, chatbots that do not have an embodiment, as can be seen in (Gaffney et al., 2019; Martínez-Miranda, 2017) and studies specifically related to dealing with high stress situations in academic environments (De Nieva et al., 2020). Dekker et al. (2020) propose an online AI-guided therapy to promote academic, social and health-related goals and provide personalised follow-up coaching, thereby preventing academic and mental health problems.

These CAs have been developed and evaluated in developed countries. It is known from previous studies that in order to evaluate the effectiveness of pedagogical agents, the cultural and ethnic backgrounds of students play an important role in their acceptability and utility (Kim & Lim, 2013). We also know that high-school students (Plant et al., 2009) and college-aged students of colour (Moreno & Flowerday, 2006) prefer pedagogical agents that look like them over human tutors that are not from the same ethnicity.

Therefore, the focus of this paper is to motivate Indian undergraduate students to follow the recommended behaviours to help them manage their stress. ARU is a technology-based intervention to bring about behaviour change by imitating the therapist-patient working alliance, a key predictor of adherence to treatment advice (Bennett et al., 2011). We focus on the four healthy behaviours (eating healthy, doing physical activity, studying in groups, and meeting new people) that had been found difficult to change in Australia, a Western context and developed country (Abdulrahman & Richards, 2019). In a developing country like India with one of the largest populations consisting of young adults, high academic stress due to excessive competition and poor access to mental health services, there is a lack of student support and subsequently an absence of studies using ECAs to assess or manage academic stress. Hence, we pose our first hypothesis as follows:

Hypothesis 1.Indian students will report lower levels of study stress after interacting with the ECA.

ECAs using explanation to encourage behaviour change

Explanation can help to build a trusting relationship between parties (Glass et al., 2008; Rheu et al., 2020). Focusing on behaviour change, a health message can motivate a behaviour change when it is personalized according to one’s mental state (Wheeler et al., 2005). In human-human interaction, people refer to their mental state, particularly their beliefs and goals, to explain their behaviours (Malle, 2005). Explainable agents (XAs), particularly, belief-desire-intention (BDI) agents, (Bratman, 1987), have been introduced to simulate the human natural way of behaviour explanation. An ECA that provides explanations of its behaviours can also be referred to as an XA; we will use both terms interchangeably when referring to ARU.

However, the value of explainable BDI agents is not well investigated yet (Anjomshoae et al., 2019). Broekens et al. (2010) evaluated the perception of different explanation patterns provided to firefighters in training scenarios. The study concluded that the explanation pattern (belief or goal-based explanation) should be tailored according to the agent’s behaviour. In health, with a robot designed for education about diabetes in children, Neerincx et al. (2018) reported the preference of the adults to receive goal-based explanations over belief-based explanations to understand the agent’s behaviours during the interaction.

However, existing XAs explain their behaviours by referring to their mental state (beliefs and goals). Considering the importance of referring to the user’s mental state to motivate health behaviour change (Wheeler et al., 2005), we opt to utilise an XA that is able to refer to the user’s mental state rather than the agent’s. The advice given by an ECA when it plays the role of a virtual advisor is a recommendation to the user to do a particular behaviour. Hence, the users could perceive the recommended behaviours as more relevant and personalized when the explanation associated with the recommendation refers to the user’s mental state. Abdulrahman et al. (2021a) introduced a user-specific XA to explain why a piece of advice is given to the user by referring to the user’s beliefs and goals. The introduced XA was evaluated with university students by measuring their behaviour change intentions towards three recommended behaviours. They concluded the importance of personalizing the explanation pattern according to the user’s profile such as stress level, personality, and age. Hence, based on the above literature, we propose the following two hypotheses.

Hypothesis 2.Indian students will develop different levels of user-agent relationship depending on the type of explanation received: belief-based, goal-based or belief and goal-based explanation.

Hypothesis 3.Indian students will report different degrees of change in behaviour intentions depending on the type of interaction: belief-based, goal-based or belief and goal-based explanation.

METHODOLOGY

We have conducted a study approved by the human ethics committee in the original Western country and also the ethics committee at the university in India where the study was conducted. Our study did not include a comparison with a human advisor, eg, a video call consultation, as our ECA seeks to fills the gap in access to such services. The Indian institution is a reputable university following best/common practices in India. However, it does not provide an advisor to support student stress. Student services that do exist would not have the training or the capacity to provide well-being services to the large student population. Furthermore, due to stigmatization, Indian students are typically unwilling to talk about their mental health.

Undergraduate students in India were recruited through an email advertisement of the study distributed via the university’s communication channels. Participants were randomly assigned by the Qualtrics survey software to one of three experimental groups where they interacted online with one of the versions of ARU: belief-based explainable ARU, goal-based explainable ARU, or belief and goal-based explainable ARU. Interaction with ARU lasted for approximately twelve minutes, while the whole study took approximately twenty minutes. No reward was received for participation. India-specific modifications are described in the following subsections presenting the study materials, procedure and data collected.

Materials

To provide a culturally-sensitive ECA to be used and tested with Indian students in a higher educational institution, we created ARU, an adaptation of a Western XA known as Sarah (Abdulrahman et al., 2021a). Sarah used a BDI (belief, desire, intention)-based cognitive agent architecture, that extends FAtiMA (Fearnot AffecTIve Mind Architecture) (Dias et al., 2014) by adding a repository of user models (one per user), a plan library containing recommendations and an explanation engine that supports tailored explanations based on the user’s beliefs and goals (Abdulrahman et al., 2021a). However, the Western study involved students who received course credit for participation and were required to download a large application to their computer. To increase accessibility and reduce the participation burden for Indian students who received no reward for participation, we created a lightweight version that did not require download of third-party software and that would allow the UNITY3D study to run via WebGL in a browser.

Due to the role of cultural cues in accepting health advice (Levin-Zamir et al., 2016), the visual appearance, name and text-to-speech (TTS) voice accent of Sarah had been adapted to the Indian context. The colour and appearance of the avatar were inspired by the picture of a famous Bollywood actress. The Avatar for ARU is shown in Figure 1. We used Microsoft English (India) TTS voice Heera. The voice was found to be acceptable by the Indian team and by the students who did not comment on the accent. To ensure appropriately tailored responses, dialogue generation did not use natural language processing, but used a dialogue engine that allows interaction through selection of options, as shown in Figure 1. The options seek to help the student reflect on the common barriers (beliefs) or motivations (goals) for performing or not performing an action.

Illustration of conversation with ARU with multiple answer options

ARU starts with a self-introduction and provides tips for study stress management. It provides the user with one of the three explanation types based on his/her beliefs, goals or beliefs and goals. During the interaction, ARU asks the student about his/her beliefs and goals related to every recommended behaviour. The student responds by selecting an option from a list of choices that dynamically change according to the context. After ARU provides a recommendation, it explains how the student’s beliefs/goals can empower or hinder his/her behaviour and how to follow the recommendation accordingly. For example, to a student who believes that “exercise is boring”, ARU responds by stating the student’s belief to convey the feeling of being heard and understood “You feel that doing daily exercise is boring.” followed by a tailored explanation “I know that for most people, exercise itself is rarely the issue. Having to do it for ‘too long’ is the issue … try 5 or 10 min … You may do online Zumba classes from home.”

To provide support during the COVID-19 pandemic, ARU’s dialogue was designed to emphasise the importance of behaviour change to cope with study stress especially during such difficult times. ARU provided tips that took into consideration that participants were either undergoing self-isolation or were quarantined at home. Examples included studying in online groups, abiding by COVID-19 related mandates, doing an online workout or having virtual discussions with friends about doing physical activities. Meeting people in person was not feasible during COVID lockdowns, so ARU reminded them that they can connect virtually and even participate in online volunteering groups. Figures 2-4 show examples of belief, goal and belief and goal conversations, respectively, between ARU and the user. In the example, the agent tries to convince the user of the importance of doing physical activity which could promote better mental health. The dialogues use the ten relational cues (eg, social dialogue, self-disclosure, inclusive pronouns, empathy) identified by Bickmore et al. (2005) to build an empathic relationship and promote a working alliance with the users. Despite the use of these relational cues, there was no attempt to pretend the ECA was a real person or was representing one. ARU was referred to as a virtual character during the course of the study.

image

Belief-based dialogue snippet between ARU and user

image

Goal-based dialogue snippet between ARU and user

image

Belief and goal-based dialogue snippet between ARU and user

Procedure and data collection

The study design and materials were exactly the same as the study design in the Australian studies with two variations in the demographic questionnaire to adapt to the context. The first variation was to change the question about the student’s background from global regions to regions within India. The second variation was related to measuring the student’s academic achievement goal. In the Australian study this was stated in terms of a pass, credit, distinction or high distinction grade. To fit the Indian context, the question referred to the grading system in India using the cumulative grade point average (GPA). Hence, the participants were asked to report their achievement aim: 0 < GPA < 5, 5 < GPA < 7, 7 < GPA < 8.5 or 8.5 < GPA < 10.

To increase the potential benefit of participation, we also provided an optional additional conversation with ARU after collecting all of the study data for students who were interested in receiving more study tips. For the study we had only focused on four behaviours that we had found hard to change in the Western context (Abdulrahman et al., 2021b). However, we hoped that the additional behavioural tips, such as drink more water and study during daylight hours, might also benefit Indian students. We would like to make a note that all of the content was reviewed and modified by the Indian team to ensure cultural appropriateness and sensitivity.

Following common practice in counselling and therapy at the start and the end of each session, we utilised a standard session rating scale to measure the effectiveness of the session. This stress assessment score used a scale of 0 to 10, with 0 being not all stressed and 10 being extremely stressed. Students’ intentions to perform each behaviour were captured using a 5-point Likert scale (never, rarely, sometimes, very often, always). Students also received a set of pre-interaction questionnaires, interacted with the assigned version of ARU, and then received a post-interaction set of questionnaires.

The pre-interaction questionnaires included the demographic information, attitude towards study, propensity to trust and behaviour intention questionnaire. This was followed by interaction with different versions of ARU. The post-interaction questionnaire included stress assessment score and behaviour intention, trust, working alliance and personality test questionnaires. The personality test consisted of ten questions from the Ten Item Personality Measure (TIPI) (Gosling et al., 2003). The questionnaire covered five major personality traits: Extraversion, Agreeableness, Conscientiousness, Openness to experiences and Emotional stability measured using a 7-point likert scale which ranged from “Strongly Disagree” to “Strongly Agree”. The Working-Alliance Inventory Short Revised (Hatcher & Gillaspy, 2006) was used to assess the relationship between ARU and the users. It assesses three key aspects of therapeutic alliance (a) agreement on the task (b) agreement on the goals and (c) establishment of an effective bond between user and agent. It has twelve-items which measure: Task, Goal and Bond dimensions (four items in each). Items are rated on a 5-point Likert scale (seldom, sometimes, fairly often, very often, always). Measuring another aspect of the human-agent relationship, the trust and trustworthiness questionnaire was adapted from Mayer and Davis (1999), which includes measures to capture: perception of the ECA’s ability (6 questions), benevolence (5 questions) and integrity (6 questions); trust in the ECA (4 questions); and the user’s propensity to trust others, in general (8 questions). All the trust measures and the question “I liked ARU” used a 5-point Likert scale ranging from “Strongly disagree” to “Strongly agree”. Participants were able to respond NA (not applicable) to working alliance, trust and liking questions.

RESULTS

Participants

In total 61 university students (age: M = 20.52, SD = 1.30) participated voluntarily in the study. Table 1 presents the distribution of the participants among the three groups, gender distribution and age averages with the standard deviations. The results show that the number of male students (N = 39) was almost double the number of female students (N = 21). Only one participant did not identify as either.

TABLE 1.
Participants in the study
Group N Male Female Others Age
urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0001 SD
Belief 23 14 9 0 20.36 1.177
Goal 21 14 7 0 20.76 1.338
Belief and goal 17 12 4 1 20.35 1.455
Total 61 40 20 1 20.50 1.308

The pie chart in Figure 5 shows the regional distribution of the participants. Around 55.7% of the participants were from south India, 19.7% from western India, and 24.6% were from other parts of India. Table 2 shows the distribution of participants’ personality across groups. After conducting Kruskal–Wallis test on the personality data to compute between group variance, it was found that there were no significant between-group differences in terms of participants’ age or personality.

image

Regional distribution of participants

TABLE 2.
The Indian participants’ personality
Group Extraversion Agreeableness Conscientiousness Openness to experience Emotional stability
M SD M SD M SD M SD M SD
Belief 3.85 1.96 4.70 1.45 4.72 1.45 3.91 1.59 5.07 1.33
Goal 3.06 1.68 4.44 1.06 4.47 1.27 3.29 1.25 4.82 1.42
Belief and goal 3.31 1.52 4.89 0.96 4.89 1.49 3.33 1.14 4.92 1.19
  • Abbreviations: M, mean; SD, standard deviation.

Study stress

Table 3 reports study stress before and after interaction with ARU. Students were fairly distributed between the three groups in terms of their study stress level with no between-group difference (Kruskal–Wallis urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0002 urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0003). The Wilcoxon signed-rank test revealed a significant reduction in their stress scores after interacting with ARU in the three groups as reported in Table 3 with significant p-values indicated in bold.

TABLE 3.
Study stress
Group Before interaction After interaction Wilcoxon signed-rank test
M SD M SD Z p
Belief 7.65 2.10 6.91 2.31 −2.561 0.010
Goal 7.28 2.22 6.43 2.25 −3.218 0.001
Belief and goal 7.59 1.87 5.88 2.76 −3.151 0.002

Human-ECA relationship

This subsection reports trust, working alliance and liking results including mean, standard deviation, and percentage of NA responses, which are removed from the analyses.

Trustworthiness and trust

Table 4 presents the results of the analysis of the trust and WA questionnaires. There was no between-group difference in terms of ability, benevolence, integrity or trust as revealed by the Kruskal–Wallis test.

TABLE 4.
Trustworthiness and trust
Trustworthiness Trust
Ability Benevolence Integrity
Belief group
Not applicable 0.87% 2.17% 2.17% 0%
Mean 3.08 3.20 3.67 3.14
SD 0.97 1.00 0.98 1.05
Goal group
Not applicable 0% 0% 0% 1.59%
Mean 2.99 3.17 3.52 3.09
SD 1.05 0.95 0.91 0.98
Belief and goal group
Not applicable 0% 0% 0% 0%
Mean 3.40 3.53 3.85 3.02
SD 0.98 0.89 0.63 1.02
Between-group difference
Kruskal–Wallis H (2) 2.152 2.113 1.035 0.202
p-value 0.341 0.348 0.596 0.904

Participants reported a high average propensity to trust others in the three groups on a 5-point Likert scale: belief group (urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0004), goal group (urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0005) and belief and goal group (urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0006), with no significant between-group difference (Kruskal–Wallis urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0007). Participants’ propensity to trust was significantly (at urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0008) correlated with their perception of ARU’s ability (Spearman’s urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0009), benevolence (Spearman’s urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0010), integrity (Spearman’s urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0011), trust (Spearman’s urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0012), task (Spearman’s urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0013), goal (Spearman’s urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0014) and bond (Spearman’s urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0015).

Working alliance and liking

Table 5 presents the means and standard deviations for working alliance and liking. Kruskal–Wallis test reported no significant differences between the three groups for task, goal or bond scales.

TABLE 5.
Working alliance
Working alliance Liking the ECA
Task Goal Bond
Belief group
Not applicable 9.78% 6.52% 21.74% 21.74%
Mean 2.74 3.01 3.25 3.17
SD 1.24 1.31 1.15 1.38
Goal group
Not applicable 14.29% 17.86% 15.48% 8.70%
Mean 2.42 2.52 2.95 2.53
SD 0.96 0.79 0.98 1.12
Belief and goal group
Not applicable 0% 0% 4.41% 0%
Mean 3.01 3.19 3.40 2.76
SD 1.11 1.17 0.96 1.44
Between-group difference
Kruskal–Wallis H (2) 2.156 3.399 1.745 2.156
p-value 0.340 0.183 0.418 0.340

Behaviour change intention statistics

Table 6 reports participants’ intentions related to the four behaviours before and after interacting with ARU. There was a significant change in four (goal), three (belief) and two (goal and belief) behaviours. The Kruskal–Wallis test revealed no significant between-group differences to study in a group (urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0016, to do physical activity (urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0017, to meet new people (urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0018 or to eat healthy (urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0019.

TABLE 6.
Behaviour change intentions
Group Behaviour Before interaction After interaction Wilcoxon signed-rank test
Mean SD Mean SD Z p
Belief group Join a study group 1.78 1.09 2.74 0.86 −3.275 <0.01
Do physical activity 2.7 1.26 3.78 1.04 −3.351 <0.01
Meet new people 2.26 1.29 3.00 1.13 −2.846 <0.01
Eat healthy 3.57 0.84 3.74 0.86 −0.718 0.473
Goal group Join a study group 1.67 0.97 2.62 0.92 −2.923 <0.01
Do physical activity 2.71 1.01 3.38 1.12 −2.952 <0.01
Meet new people 2.1 1.04 2.81 0.93 −2.639 <0.01
Eat healthy 3.19 0.93 3.71 0.96 −2.653 <0.01
Belief and goal group Join a study group 2.29 1.26 3.12 0.86 −2.169 <0.05
Do physical activity 2.82 1.24 3.76 1.03 −2.546 0.01
Meet new people 2.41 1.18 2.71 1.16 −1.095 0.273
Eat healthy 3.65 0.86 3.88 0.70 −1.414 0.157

The studies conducted with the Western ECA (Abdulrahman et al., 2021a) found that changes in intentions were connected to the user profile. Thus, we stratified the analysis to study the change under different factors: age, gender, having an exam, achievement aim and personality. The results reported a significant negative correlation between the age and the intention to change to eat healthy in the goal-group (Spearman’s urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0020). Analysis failed to capture significant differences in the intention changes between male and female, or between those who had upcoming exams and those who did not.

Regarding the students’ achievement aim, because all of the students aimed to achieve 7 < GPA < 8.5 or 8.5 < GPA < 10, the Wilcoxon signed-rank test was run on these two groups separately. In the first group aiming to achieve 7 < GPA < 8.5, t the students’ change in the intentions to do the four behaviours in the three explanation groups was not significant at p < 0.05. In the second group, the significant change in intentions for students with the higher achievement aim (8.5 < GPA < 10) was consistent with what is reported in Table 6. Students with higher aims showed a significant change in intention to do the same three, four and two of the behaviours after interacting with belief-based explainable ARU, goal-based ARU and belief and goal-based ARU, respectively, at p < 0.05. Further, the change in the intention to do physical activity was significantly correlated with conscientiousness (Spearman’s urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0021) in the belief group, and to eat healthy was positively correlated with emotional stability (Spearman’s urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0022) and negatively correlated with extrovertedness (Spearman’s urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0023) in the goal group.

DISCUSSION

The main objective of the intervention was to provide support to Indian students in managing their study stress using a COVID-safe method. The intervention aims to change students’ intentions to adopt healthier stress management behaviours. The intervention was motivated by high stress levels and severe adverse consequences in this population. This led to a secondary aim, to explore how support could be provided. The solution chosen was motivated by greater need for low-cost and accessible support in developing economies. Based on previous findings, we chose to adapt and test an ECA designed for study stress management created for a different and less studied cultural context (ie, in a developing rather than a developed country). The adaption was motivated by the need to provide a culturally sensitive solution and to personalise the advice to increase likely adherence. The adaptation steps involved: 1. Understanding the context (large young populations, income disparities, less multicultural/diverse, poor resources, high competitiveness/stress, stigmatised mental health attitudes, COVID-impacts, see Related Work Section); 2. Accordingly modify the ECA’s voice/accent, appearance and dialogue; 3. Modify demographic questions and evaluate the ECA. By developing ARU, we have addressed research question RQ1 for the cultural context of an Indian university student.

It was observed that after interaction with the agent, users reported a significant reduction in their stress levels for all explanation groups, as can be seen from Table 3. This supports hypothesis H1 and also answers RQ2, related to improved levels of study stress, in the affirmative. Quantitative analyses indicate that interaction with ARU created awareness amongst students about their own stress as confirmed in qualitative comments such as:

Pretty well-rounded study. Helped me get in touch with my feelings and thoughts currently. Thank you!!

It was good to engage in this, felt like talking to my best friend and Nice study, ARU was elaborate and helpful.

As further support of the perceived value of the conversations, we note that 26 of the 61 participants completed the optional conversation immediately following completion of this study. This additional conversation took an extra 15–20 min and contained the tips in this study plus another nine suggested behaviours for reducing study stress (Abdulrahman et al., 2021b).

The user-agent relationship was measured by the trust and working alliance questionnaires. The participants in all groups rated ARU as equally trustworthy (fairly to very-often with no significant between-group difference). Task, goal, bond and liking scores were close to “fairly often” for all groups. These results indicate that ARU is able to develop a positive relationship with the Indian students for all explanation patterns, hence, we reject H2. However, this indicates that, in general, Indian young adults are positive towards the use of ECAs to offer personalised advice about their study behaviours to support their mental health and are comfortable sharing their beliefs and goals with an ECA. This is a positive outcome particularly given the reluctance of Indian students to discuss their mental health with others.

Although there were no significant between-group differences in terms of trust and WA scales; participants who received belief and goal-based explanations showed the highest average trust and WA and the lowest percentage of NA responses among the three groups. This could be because those participants may have felt more engaged and found the conversations more relevant and comprehensive when complete explanations based on their beliefs and goals were provided. This interpretation might be supported by the significant decrease in the stress level of the students after interacting with belief and goal-based explainable ARU compared to the two other versions. Student’s decrease in stress was significantly correlated with ARU’s ability (Spearman’s urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0024) and trust (Spearman’s urn:x-wiley:00071013:media:bjet13174:bjet13174-math-0025) only in the belief and goal group.

Participants reported very high averages of propensity to trust strangers in general (above 4.5 on a 5-point Likert scale) compared to participants in the Australian study who showed average propensity of 3.12/5. The Indian students’ propensity to trust strangers was significantly correlated to their trust in ARU which indicates that participants’ trust in ARU could be due to their own individual characteristics and tendency to easily trust others and their advice (trait-like trust) (Zilcha-Mano, 2017) rather than for reasons related to ARU and its dialogues (state-like trust). The impact of this cultural difference could be that interventions involving ECAs with an Indian population are likely to be more effective, relating to RQ2, because the students may be more likely to accept and follow the ECA’s advice and their acceptance would be less influenced by the ECA’s design or dependent on development of a long-term relationship with the ECA than in the case of Western populations. However, one reason why an individual’s propensity to trust others is measured is to check their vulnerability (Mayer et al., 1995). High propensity to trust regardless of what is delivered in the intervention necessitates, for this population, even more careful design and scrutiny of the content by health domain experts and assurance that the advice is appropriately tailored and evaluated to meet different needs and individual differences. Given the large number of students and shortage of student access to support services and mental health expertise in India, generally also true for other developing countries, the benefits versus the risks of this technology will need to be considered in the design and deployment of ECAs in this context.

In answer to RQ3, although there was no significant difference between the three explanation groups (belief, goal and belief and goal) in terms of their changes in intention to do the four behaviours, leading us to reject H3, the results showed preferences for receiving a specific type of explanation for certain student profiles, discussed further below. The influence of individual factors on behaviour change were also found in the Australian studies, however, different factors including stress level, agreeableness, openness to new experiences and having exams were shown to influence intention to change specific behaviours. These differences confirm the need for understanding the cohort and their culture and adaptation of the ECA accordingly.

Results suggest that it is best to provide goal-based explanations, as the students showed a significant change in their intentions to do all four behaviours after receiving this type of explanation. However, extrovert students were significantly less motivated to change after receiving a goal-based explanation. This means more tailoring is needed. The second implication is to equip ARU with more strategies to change the student’s goal first before recommending the behaviours.

Differences in changes in intention to display the behaviours between explanation groups could be due to differences in the recommended behaviours. The participants reported very low intentions to join a study group before interacting with ARU, which aligns with the problem reported in the literature that Indian students show high peer rivalry (Nandi et al., 2012). Resistance to or lack of experience involving participation in study groups could have been related to prior concerns around sharing knowledge with competitors. Despite these initial attitudes, ARU, regardless of its explanation setting, was successful in motivating the Indian students to change their intentions to join a study group. It is interesting to note that participants reported “never” to “rarely” to describe their intention to perform all behaviours except to eat healthy before interaction. The possible reasons could be the prevailing pandemic situation or because participants consider social activities as distracting or a hindrance to academic success. However, after interaction with ARU, for all the behaviours there was an increase in intentions, with the least change in eating healthy probably due to a higher intention to perform this activity before the interaction.

Studying the factors that can explain the change in behaviour intentions (Section 4.3.3) revealed that the Indian students’ achievement aim was the most likely factor to predict behaviour change. Students with the aim to achieve a GPA between 8.5 and 10 are the only participants who reported significant behaviour change intentions.

While age, gender and having exams did not contribute to the change in behaviour intentions, some correlations between the change in the intentions and personality type were identified. According to the correlations between personality and intention change, we can conclude that to motivate conscientious students (ie, students very focused on their academic achievements) to do physical activity, it is best to recommend that the student engage in physical activity due to the benefits it can deliver to create a healthy and clear mind that will facilitate improved academic performance. Further, the ECA should include explanations that reinforce the student’s positive beliefs about exercise and address any negative beliefs towards this behaviour held by the student, such as beliefs that exercise will require too much time away from their study or that exercise will make them too exhausted to study well. Students with higher emotional stability may benefit from talking with goal-based explainable ARU to reflect on and potentially change their attitude towards healthy eating. More interesting is the adverse effect noticed with extrovert students when they received a goal-based explanation on their intentions to eat healthy. Such findings shed light on ECA designs tailored to specific contexts and cultures.

CONCLUSIONS AND FUTURE DIRECTIONS

We have designed and implemented a first of a kind intervention in a developing country context involving the use of an ECA for academic stress reduction. To achieve this, we carried out design changes to a Western ECA. To evaluate whether the intervention can reduce academic stress in the targeted population, we conducted a study at an Indian university. Following student-ECA interaction, students reported lower levels of study stress, higher intentions to perform certain behaviours and development of a positive relationship with the agent. Different explanation patterns resulted in different behaviour change intentions and user agent relationship. To address study limitations, future studies with different populations, larger samples, follow-up and long-term interaction should be conducted to confirm if behaviour intention turns into action (Fishbein & Ajzen, 2011), test generalizability and ensure our findings are not due to technology novelty effects. Ideally access to ARU would be integrated into institutional support and ARU’s conversations would be expanded to cover more topics.

In the future, we would like to review the XA’s explanation patterns and plans based on the beliefs and goals of Indian students (Kumar & Bhukar, 2013; Nandi et al., 2012) and study the effect on their behaviour change. Defining the culture specific motivators and barriers will be a great advance towards building a tailored XA that can understand the user’s characteristics and context. It is hoped that tailoring advice and explanations appropriately will improve the user-agent relationship and motivate health behaviour change for populations based in developing countries; thereby addressing a current gap in access to mental health services and support and potentially raising awareness and reducing associated stigmatised attitudes.

CONFLICTS OF INTEREST

There is no conflict of interest.

ETHICS STATEMENT

The research study reported in this paper was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Human Research Ethics Committee, Macquarie University with reference number: 52020535012841 dated 25th May 2019.

The study has also been approved by the Institutional Human Ethics Committee (Approval no. BITS-Hyd/IHEC/2020/11), BITS Pilani Hyderabad Campus.

The data are not publicly available due to privacy or ethical restrictions. Requests and reasons for access to the data that supports the findings of this study can be sent to the corresponding author.

REFERENCES