What do users think about your chatbot? Are they satisfied with the responses it provides? Are they taking full advantage of this tool? Does the conversational agent have a positive impact on recurring contacts? If you don’t know the answers to these questions, it’s because you don’t have the necessary key performance indicators in place. These measurements are indispensable for tracking the results of your chatbot, identifying any stumbling blocks and continuously improving its performance.
Your chatbot is built. You already have the first users. But do you know how successful your chatbot is? The implementation and launch of a chatbot are only the beginning. Once you have mastered this step, identifying metrics that will help you establish relevant chatbot analytics. That way, you can continuously measure the success of your bot and derive further improvement areas.
QUANTITATIVE KPIS: IS YOUR CHATBOT BEING USED SUFFICIENTLY, AND DOES IT RESPOND TO USERS’ NEEDS?
Quantitative key performance indicators allow you to evaluate the effectiveness of your chatbot and the way its target audience uses it.
1. Customer satisfaction
Customer satisfaction is a critical factor when it comes to measuring the success of chatbots in customer service. After all, customer satisfaction is what counts in the end. The measurement of customer satisfaction in a chatbot conversation is similar to measuring a conversation with a human employee.
Customer satisfaction is a critical factor when it comes to measuring the success of chatbots in customer service.
Click to Tweet
It’s usually done via a short questionnaire after the conversation, and it can cover questions, such as:
Was the user able to understand the bot without any problems?
Could the bot answer even very specific questions?
Was it possible to talk to a human employee? And was the user forwarded to him if the bot did not understand a question?
Another possibility is to use chatbot ratings. After the conversation, you can ask the user to rate the conversation. This is what we use for some of our chatbots. Of course, the more detailed the feedback about why a conversation was good or bad, the better, but star ratings are well suited for an initial picture of your bot’s perception.
2. CHATBOT ACTIVITY VOLUME
Measuring a chatbot’s Activity Volume means evaluating the number of interactions from when a user asks a simple question until a constructive dialogue occurs. This indicator helps answer two key questions.
Is your chatbot being used frequently?
Is the number of users increasing?
3. BOUNCE RATE
The Bounce Rate corresponds to the volume of user sessions that fail to result in your chatbot’s intended “specialized” use. An elevated rate indicates that your bot isn’t being consulted on relevant subjects to its area of competence. This should prompt you to update its content, rethink its placement in the customer experience, or both. It’s an indicator that should be observed closely.
4. RETENTION RATE
The Retention Rate refers to the proportion of users who have consulted your chatbot on repeated occasions over a given period. This indicator can be compared with the typical frequency of client contacts in your particular line of business. It will provide a good indication of your chatbot’s relevance and its level of acceptance among your clients.
5. USE RATE BY OPEN SESSIONS
This is the number of sessions that are simultaneously active with your chatbot. This rate must be weighted with the average number of open sessions during a given period to get a meaningful measurement.
6. TARGET AUDIENCE SESSION VOLUME
If you target a specific population, you can measure the penetration rate for this audience to verify that the intended people are making sufficient use of your chatbot. Otherwise, it’s an initiative to rethink your change management or customer experience strategies to get your users on board! This indicator is essential for verifying that you are achieving your goals.
7. CHATBOT RESPONSE VOLUME
This is a concrete indicator that will tell you the number of questions your chatbot has answered.
8. CHATBOT CONVERSATION LENGTH
This metric allows you to evaluate the average length of the interactions between your chatbot and its users. The figure will vary significantly from case to case: a chatbot that resolves computer issues or provides online estimates will require a much longer dialogue than a chatbot that gives the current time in all the world worlds! If your goal is increased efficiency, this KPI will help you quantify the amount of time saved by your clients, as well as your Help Desk.
9. USAGE DISTRIBUTION BY HOUR
At what times of day do users most frequently consult your chatbot? This indicator is particularly helpful, as it often serves to demonstrate how this new 24/7 channel enables you to cover 20, 30 or even 50 percent of the hours during which your user support services were previously unavailable.
10. QUESTIONS PER CONVERSATION
This indicator will help you determine how many questions your chatbot needs to be asked before providing the necessary information to its users. The more questions users have to ask, the more time it will take to obtain adequate responses. Please note that the interpretation of this metric depends heavily on your specific objectives.
11. INTERACTION RATE
If you want to measure user engagement during conversations with your chatbot, you’ll observe this indicator. It will allow you to measure the average number of messages exchanged per conversation.
12. GOAL COMPLETION RATE
This metric enables you to measure the success rate of a given action performed through your chatbot, for example, clicking on a CTA button or link, filling out a form, proceeding to make a purchase. However, it can only apply to identify actions for which customized indicators have been created.
13. NON-RESPONSE RATE
This metric measures the number of times your chatbot fails to respond to a question. Such failure may result from a lack of content, or your bot’s in comprehending user inquiries.
14. MOST FREQUENTLY ASKED QUESTIONS
What inquiries are most often addressed to your chatbot? Thanks to this statistic, you can adapt your chatbot to specialize in the subjects that come up most commonly and thereby improve its performance. Analyzing recurring questions will help guide your corrective work, allowing you to focus on the topics that are of greatest interest to your users and the mechanisms that will enable you to improve the quality of your bot’s bosses, as well as its overall comprehension levels.
To make sense of these quantitative KPIs, you must compare them with other data, particularly the number of calls and the results produced by other channels (e.g., chatbot conversation volume vs. telephone call volume, relative satisfaction rates, etc.). This data will make it possible for you to evaluate the positioning of your chatbot and determine if it’s in it’s the right place with the right knowledge.
QUALITATIVE KPIS: ARE YOUR CHATBOT USERS SATISFIED?
Besides quantity, there’s a matter of quality. The KPIs below will help you measure your chatbot’s “performance,” include” in its levels of comprehension, and the help it provides to its users and its user satisfaction rates.
15. COMPREHENSION LEVEL
Your chatbot will indicate its overall comprehension of user inquiries. This level is constantly evolving, as it depends on: If your chatbot doesn’t have an inquiry, it’s because it has been asked a question with no meaning or because it doesn’t seem related.
For example, a chatbot specializing in computer support won’t support a legal question!
16. SELF-SERVICE RATE
This rate corresponds to the number of users who were able to obtain the help they needed through the responses given by your chatbot without subsequently having to call Customer Service. It is calculated based on the percentage of completed sessions through an interaction with your bot without being redirected to a live operator. In the process, it enables you to evaluate the level of client satisfaction. This is equivalent to a call center’s Call Resolution (FCR) rate, the percentage of problems resolved through a single phone call.
This indicator is very important for analyzing the ROI of your chatbot project.
17. USER FEEDBACK
Finally, it’s sensible to know what users think about your chatbot. Did it provide sufficient help? Are its users satisfied? There are two different ways you can find out:
You can ask users to reply “yes or no to the” question “Were you satisfied?”
You c “n offer users the opportunity to fill out a more in-depth questionnaire to obtain specific information (e.g., “Were t “e responses clear?” “Did “o” understand everything?” or “D” you” have any suggestions for improving our chatbot?”).
This feedback will allow you to calculate two indicators:
The Satisfaction Rate (the average score received by your chatbot in evaluations by its users)
The Evaluation Rate (the percentage of sessions in which the user evaluated your chatbot’s at least once)
Conclusion
Knowing these keys indicators is essential for evaluating your chatbots. However, the best indicators to track aren’t the same for every company or every chatbot: it’s up to you to choose the most relevant ones based on your line of business, your goals, and your users’ needs.
4 comments