Project at a Glance

A Boston-based tech company that creates collaboration tools wanted to understand under what circumstances and conditions would a human want to work with an intelligent agent. My team and I began by conducting a literature review and used affinity mapping to organize the most salient themes around relationship building research. Then we used design thinking to develop scenarios and personas. We wrote the conversational interface script and conducted Wizard of Oz Testing. Finally, we created a prototype and system architecture.

Research Insights

With developments in artificial intelligence, natural language processing and machine learning, it is now becoming increasingly common for people to have ongoing communications with non-human entities such as virtual agents. The most salient elements of relationship building, both between humans and between humans and inanimate objects, like bots, include:

  • Changes Over Time: Relationships are dynamic and evolving. Implicit in relationship building is a shared history that develops over time.
  • Level of Trust: Trust is essential and builds in increments. When a colleague demonstrates dependability, it reduces uncertainty with evidence of behavior and this develops trust.
  • Attunement: Users prefer relational agents that attune to their own personality and communication style. Attunement, at its most basic level, helps to foster mutual understanding.
Kris Carafelli | Experience Designer & Researcher | Intelligent Agent Relationship Building Murally Affinity Map Image

Design Thinking

As an exercise to narrow focus, I used the d.school’s reframing point-of-view (POV) Madlib exercise:

[USER] needs to [USER’S NEED] because [SURPRISING INSIGHT]

Kris Carafelli | Experience Designer & Researcher | Design Thinking: Adulting with Dental Care Bot Image

POV: The Responsible Employee needs to figure out if her dental insurance is worth the cost because she is considering going rogue and cancelling it if costs prove to be more than benefits.

Kris Carafelli | Experience Designer & Researcher | Design Thinking: Travel Bug Bot Image

POV: Intrepid Traveler who loves to travel needs to take all necessary precautions before an adventure because this allows her to feel less anxious about what is actually a very scary thing to her.

Kris Carafelli | Experience Designer & Researcher | Design Thinking: Cable Company Customer Care Bot

POV: Cable Customer needs quick and clear account information because less than this makes her feel like it’s her fault she doesn’t understand all the options and nuances.

I also brainstormed on what the bot might be if it wasn’t an assistant.

Scenario

Our focus ultimately narrowed to the relationship between an HR chatbot who helps a new employee with the onboarding process. Analysis of the onboarding process revealed both user and business goals with substantial overlap. The overlapping mutual goals include areas where a bot could be most effective. Analysis of typical onboarding checklists revealed six categories — workstation, forms, benefits, introductions, tour and position — from which a bot could assist a new employee and establish expertise as they build a working relationship.

Personas

Research shows that people tend to trust and build relationships with others who share similar personality and communication styles. As such, the bot was designed to attune to the user. There are several business tools for understanding behavioral tendencies in the workplace. We used a combination of these profiles to form our personas and bots. We chose to focus on just two, the Analytic and Creative, to better examine contrasting personalities.

In our system, new hires take a personality test during onboarding.  The bot uses this information to determine the content and tone of its responses. In general, an analytic bot style provides data, details, structure, and next steps with less emotion, while a creative bot style shows emotion with lots of emojis, inspires users to reach big goals, and helps the user move through ideas quickly.

Script Development

We created scripts of potential proactive and reactive responses to some assumed questions or requests. Below is a sample script for how the bot might alter its response to different levels of frustration as a relationship develops. Scripts were used to test the first prototype.

Kris Carafelli | Experience Designer & Researcher | Relationship Building with an Intelligent Agent: Breaking the Bot Image

Wizard of Oz Testing

A proxy prototype was shared with the participant. The sophisticated operations of the concept were conducted live by undisclosed human operators. Participants were talked through a scenario by a moderator and invited into a chat with our bot. The operator used the script for the interaction.

One of the first things we learned from the WOZ prototype sessions was that the operator needed to be familiar with the sample lexicons as well as the selected communication style so that improvisational responses could be given to non-standard questions. This highlighted the need for a robust language database to draw from. We saw positive responses to the concept and learned that proactivity — paired with usefulness — created a trust and a willingness to continue engagement in the future.

An element that was not easily prototyped was the over-time interaction that is an intrinsic part of relationship building. This is likely to require a test build, or a semi-longitudinal study through which the participants interact with scripted respondents over a period of time.

Architecture

The architecture for a relationship-building bot needs to include elements of context, language, sentiment and personality analysis.  These are each queries and the results are combined to form new user-specific variables for subsequent queries.  Ultimately these chains determine the style and content of the bot’s response.  The system would likely be a combination of both simple rule-based engines as well as a more Bayesian approach to determine the most likely actions and responses based on the initial variables and learned efficacy over time.

Prototype

For the prototype, we explored means of wireframing a conversational interface. This prototype needed to illustrate the user’s inputs, system functions, and connections to databases and services. We chose to orient the timeline vertically to reflect the flow of a messaging conservation. This flow allows the incorporation of bot processes, databases, and integrated services. Additionally, support systems can be identified for various requests and connections.

Kris Carafelli | Experience Designer & Researcher | Relationship Building with an Intelligent Agent System Wireframe Image

Next Steps

After collaborating with engineering to determine the feasibility of this approach and making necessary adjustments, the next step would be to tightly define the bot’s scope and functionality and create the robust language database.