I’m a UX designer with product design experience, skilled at balancing analytical and creative thinking. I focus on creating user-friendly experiences, staying flexible in my process, and continually refining details to deliver effective, user-centered design solutions.
The aim of this research was to explore how the dialogue strategies and levels of choice autonomy in a voice-enabled chatbot affects users’ cognitive load, utility and usability satisfaction, using the “food recommender” as an example. To address this, the following questions have been answered:
RQ1. What is the effect of dialogue strategies (proactive versus reactive) and choice autonomy (high versus low) on users’ utility?
RQ2. What is the effect of dialogue strategies (proactive versus reactive) and choice autonomy (high versus low) on users’ cognitive load?
RQ3. What is the effect of dialogue strategies (proactive versus reactive) and choice autonomy (high versus low) on users’ usability satisfaction?
The design thinking process for this project followed a user-centered approach, starting with secondary research to deeply understand user needs and challenges in decision making. Based on insights from secondary research, task scenarios were developed, and multiple prototypes were refined through evaluations. Finally, a controlled user study was conducted to assess the impact of different AI chatbot behaviors on users’ utility, cognitive load and usability satisfaction while deciding.
During the process, I iterated the scripts and interface based on users’ evaluation. By evaluation I turned to use the Wizard of Oz approach. I used the panel to play the corresponding answers according to users’ interactions with the chatbots, simulating the process of the conversations with AI chatbots. I crafted the voice answers based on two dialogue strategies – proactive and reactive, and two levels of choice autonomy- high and low; and provided detailed scripts to guide the chatbots’ responses.
This prototype was designed following the theories with proactive dialogue strategy (like chatbot starts the conversations and will ask users if they need to know any other relevant information), and high level of choice autonomy (like chatbot gives users 2 or 3 options based on users’ want).
This prototype was designed following the theories with reactive dialogue strategy (like users start the coversations, and chatbot will not ask users if they need to know any other relevant information), and high level of choice autonomy.
This prototype was designed following the theories with proactive dialogue strategy and low level of choice autonomy (like chatbot will not give users option, and makes decisions for users).
This prototype was designed following the theories with reactive dialogue strategy and low level of choice autonomy.
For study design, I used a within-subjects controlled design, involving 12 non-native English-speaking students from Loughborough University. Participants interacted with four chatbot prototypes. They would ask chatbot questions about proteins in Prototype A and C and ask chatbot questions about vegetables in Prototype B and D following the task lists, and the order of exposure to the four prototypes counter balanced to avoid the effect of learning transfer effect and make the four prototypes comparable. Then they will be asked to fill the Perceived Usefulness (PU) Scale, NASA Task Load index (NASA-TXL) and System Usability Scale (SUS).
The average PU score reported by participants was highest for Prototype A.
The result of the Friedman’s test showed that there was a statistically significant difference between the four prototypes in terms of perceived usefulness at the 0.05 level (χ²=14.14, p= .003). The effect size, measured by Kendall’s W, was 0.39, indicating a medium effect.
The results of the Wilcoxon signed rank test showed that half pairwise comparisons between prototypes differed significantly from each other at the 0.05 level.[ Prototype A vs Prototype D (Z=−2.05, p= .041, r= .59); Prototype B vs Prototype D (Z=−2.59, p= .010, r= .75); Prototype C vs Prototype D (Z=−2.74, p= .006, r= .79)]. All comparisons indicate large effect sizes, suggesting that prototype D had significant statistical difference with prototypes A, B, and C in terms of utility ratings.
The total NASA-TLX score for the four prototypes were similar, but prototype A still performed better than the other three.
The two-way repeated measures ANOVA showed that the difference between the high and low level of choice autonomy on workload (high level of choice autonomy versus low level of choice autonomy) was statistically significant at the 0.05 level: [F (1–11）= 10.29, p= .008, Partial η²= .48 ], and represented a medium effect size.
Prototype C performed better than the other three in terms of mental demand on NASA-TLX. I found no significant differences between the prototypes.
The SUS score for the four prototypes were similar, but prototype A still performed better than the other three in terms of system usability.
Post-hoc Bonferroni pairwise comparisons tables showed that there were significant differences between high and low autonomy in the case of reactive dialogue strategy (p= .045) in terms of usability satisfaction, while no significant difference was found between high and low autonomy in the case of proactive dialogue strategy (p= .745).
The qualitative data collected through the interviews were systematically analysed to identify insights which could be converted into design opportunities and implications for developing the prototypes.
