The ⬆️⭐ Metric for your Product

Ironically, this picture is from the southern hemisphere :p | Kumeu, Auckland, New Zealand

Phew………..

Yeah!
I am finally putting my final (for all one knows — still arguable) thoughts here, after contemplating for almost half a decade of product management, pondering over what should be an ideal north star metric for a product and will be closing the debate taking Conversational AI Product as the perfect example (as I think it is a genuine HOT POTATO currently).

It has almost been seven years in product management now, till date, working for five different organizations across different domains and spaces like OTA, E-commerce, Gifting, Consulting, etc.; handling products with different scalability, power, usability, compliance, security, and you name it; and at every stage or in every stint at my previous employers, with a different set of product metrics to analyse and worship.

While I shall spend most of my time in the write-up here, building up some momentum around the discovery and settlement of the North Star metric for any Conversational AI product; let me however spend a few minutes on the idea of product metrics / KPIs in general.

I have created the below Venn diagram to better demonstrate my case (if you will) of settling for the right set of metrics for your product depending on the critical factors that directly / indirectly impact the success of your product.

As you’d see below in the chart, I am trying to bring your attention on two areas mainly :-

1. A set of metrics that stay common to mostly all the products, regardless of the domain and am calling it the “As Common as Dirt” set of metrics. — — → Let’s discuss this for now.
2. The other is about finding the perfect North Star metric for your Conversational AI product — — -> We will get to it a little later.

The overlaps you might be interested in

So what am saying is….

Wrapping my head around the silly Venn-diagram myself after making it over a coffee; I reckoned it would be interesting to deep-dive on the intersection of those good looking 5 circles. Aye?

In simple English : My case is that there will always be a set of product metrics that will cry for your attention (and why shouldn’t they) regardless of the domain your product belongs to or the scale at which you want to celebrate your RTM. They have an indispensable value that needs your attention.

If we are looking at five domains, let’s say :-

1 — B2B SaaS (DAP)

2 — Re-com

3 — E-com

4 — Conv. AI

5 — OTA

These “Jack of all Trades” set of metrics am talking about are the intersection of {metrics(1), metrics(2), metrics(3), metrics(4), metrics(5)}, or what I have mentioned as (1∩2∩3∩4∩5).

METRIC 01 — CONVERSION RATE

"The art of conversion lies in making your users feel like they discovered something they didn’t even know they were looking for.” — Robbin Steif

You will always have some macro goal for your product. Largely it will be around some conversion happening on your website or platform.
For instance, if you are an OTA Product Manager, you would be tracking that how many visitors who visit your website in a certain time period, end up booking a property (hotel). Similarly, if you are an E-commerce company’s Product Manager (may be a Gifting company for example), you would be tracking that how many people who visited your cakes listing page, ended up ordering one.

Which brings me to my version of the definition (against what Wikipedia may be serving while am writing) :-

CR = Count of people who achieve your macro goal in that time period / Total people interacting with your website / platform in a given time period.

I feel that it can be argued that why don’t we put the people who visited the website in the particular time period as denominator, instead of people interacting with the platform in that time period. The defense is that, for example, I visited a website on the 1st of January in 2024, and kept visiting (repeat-visitor) every day, until I made a booking on the 7th of January. If the time period, against which the Conversion Rate is to be calculated is let’s say — Jan 01 to Jan 07, then certainly I as a customer will be counted in the denominator under the visitors logic, but if the time period was Jan 02 to Jan 07, I may get excluded by the rule of first-day visit date.

Hence, since I anyway interacted with the website between Jan 02 to Jan 07, so I should be contributing to the denominator which is the total people who interacted with the platform in a certain time period. This may be debatable. Hence, your comments / feedback are invited, generously. Please drop them in the comments.

And further for a Re-commerce platform like for example, a second-hand smartphones marketplace, the Conversion Rate would be once again a very prominent metric, in the sense, that the organization would want to track that out of all the visitors who interacted with the platform, how many ended up selling their old gadget or buying an old one.

In short, Conversion Rate is an inevitable key performance indicator for any platform that sees visitors on a daily basis, who come to either make a purchase or do a booking or even just simply subscribe to a news-letter. It all just depends on what is your goal (macro / micro).

METRIC 02 — CSAT / NPS

“Your most unhappy customers are your greatest source of learning.” — Bill Gates

On that note, isn’t seeing a customer returning happy from your platform, the most satisfying thing ever? Just come out of the money/revenue/profit bubble for once, and look at what I just said from that fresh and a bit of CX-oriented standpoint.

By how is your customer rating the NPS / CSAT scale that you prompt at the end of the user journey on your platform, you get an invaluable insight into how customers perceive and interact with your product / service.

CSAT ratings are a gift from God. They lead you towards new happy customers in future.

Again, taking the examples of OTA / E-commerce / Re-commerce / B2B SaaS / Conversational AI and you name it; I shall still have the same opinion for / in context of each of these domains, that the NPS / CSAT is a clear-cut indication of how satisfied your customer is with your products / services (presence in general), regardless he just booked a hotel on some website or sold his phone on another platform.

If you are a very meticulous and methodical Product Manager, then you may also want to track the satisfaction quotient on every stage/step of the user journey on your platform. For instance, the customer may be happy on the landing page of your platform but may not be equally happy when checking out. The reasons can be diverse :-

• Landing page is beautifully crafted with needful details and too much of cluttered information being kept out
• However on the check-out page, the customer is being bombarded with all the details of the world.
• 2 seconds into the check-out journey and the customer is feeling asphyxiated. Reconfirming the same details again and again with the customer has just left him annoyed.
• He/She simply wants a small grid of the payment details and the options available. And if possible, then some discounts. Period.

• Check-out’s have to be kept minimalistic in terms of UI as well as UX, else you will see many of those customers dropping-off right before the payment.

To not to forget the point I was making, the customers may have a different CSAT quotient at different stages of their user journey on the platform. So it becomes even more imperative for the PMs to capture CSAT at every level for effective consumer engagement.

Also, not to forget, NPS / CSAT like ratings are good indicators of user retention and loyalty. Any customer going satisfied from your website, is likely to return soon. These customers will be very loyal to your company / platform throughout their Customer Lifetime Journey.

Moreover, through the NPS and CSAT feedback received from the customers, the Product Managers may also get some interesting points about the kind of features or bug-fixes that they can prioritize at their end, in view of their existing backlog.

Customers are the best advisors.

METRIC 03 — CUSTOMER LIFETIME VALUE

What is Customer Lifetime Value though?
Total predicted revenue a customer is expected to generate over the entire duration of their relationship with a business.
Business as we know It : The most impactful businesses you would come across these days, do not just constrain their thought process and every decision they take, towards converting the customer that visited today and making the best deal out of that one single transaction.
They do and ought to, look at the bigger picture : that how do we ensure that this customer becomes a loyal one and stays with us for long. The value that the companies can generate over the lifetime of a customer on their platform is undisputedly much bigger than the first-visit conversion.

Why should a company spend their time in generating that value or improving the Customer Lifetime Value and how can they do it?

By creating a positive and engaging experience for the users, the organizations tend to improve their customer lifetime value. What they essentially achieve out of it is that they get repeat business from the loyal customers in future, and long-term relationships are fostered.
So, I just talked about they why part!

Now, coming to how!

There are various ways the companies can make use of their websites (or user-interfaces if you will, devoid of the nature / architecture of their platforms) in order to generate / improve the customer lifetime value :-

The background landscape is from a remote area near Auckland, that hosts a beautiful waterfall — Karekare

1. User-friendly Design : If the Product Managers maintain a clean and intuitive web interface on their websites, it certainly helps the customers to easily look for what they wanted to. This, in turn, helps in providing enough motivation and satisfaction to the users to come back again and convert into loyal repeat-customers.
2. Personalized Content & Recommendations : There are myriad ways that you as a Product Manager, can track and keenly work on the preferences and browsing / purchasing patterns of your users. In turn, those user preferences that you capture shall help you build a platform of recommendations and serve personalized content to your end users.
3. Effective Onboarding Process : Imagine yourself a customer who is being guided at every step, when you navigate through a website for the first time. It just all feels like a smooth onboarding. Doesn’t it? So, now that you are in reality a Product Manager, and are serving products to such end consumers, won’t you want to serve such effective guided onboarding journeys to your end users? If you do, your customer is certainly going to come back and stick with you (your platform) for life.
4. Responsive & Mobile-friendly Design : Initially, as a Product Manager you mostly believed in providing the best user experience on big screens (what I mean here is Desktop, Laptop, Tablets, etc.) Well, that was the Moghul mindset (metaphorically — in terms of how old that psyche is); cut to the today’s era, where Product Managers would want to provide an equally vivid and user-friendly experience on mobile / mobile-like devices too. The reason is simple — your end users can access the information and navigate your websites on the go, seamlessly, across any type of device. Isn’t that going to satisfy your customers big time? Of course, it is. They will be tempted to become repeat-customers and your job is done.
5. Engaging Content & Blogging : As a Product Manager, it should be your bread and butter to regularly update your website’s content with something relevant, interesting, engaging and invigorating for your users. For instance, you should promote blogs and motivate your Content / Marketing or even Engineers (as I find them sharp writers) to write engaging articles / blogs / or any form of , since when your end customers come to your website and find some relevant and engaging blogs to read, they would always want to keep coming back on your website. The blogs are not just going to offer engaging content coming out of your writer’s creative / imaginative scientific best but they are also the best way for you to show your / your platform’s expertise.
6. Loyalty Programs & Incentives : Find me one customer, who won’t say Yes to a reward. Find me one customer who doesn’t like getting incentivized. Name one customer who would say No to discounts. Loyalty programs were introduced for a reason. The idea was simple : make your customers feel accepted and welcomed and reward them for planning to stay for long on the websites / platforms. The older a customer is (repeat-user), you can make
7. Subscription Models : Specially if you are a B2B SaaS Product Manager, then implementing a subscription model on your platform can be the golden ticket to recurring revenue. The users are always happy to return to a platform and even subscribe for upgrades, if they need the services over a long period of time. Technically, subscription models give you a long-term revenue and the customer is retained for good too.
8. Social Proof & Testimonials : While these days, the organizations make it a point to collect reviews and ratings from their customers so as to draw learnings from them and identify the areas of improvement to work on their products and services; but at the same time what the Product Managers need to ensure and work towards is that they need to make the visibility of these social proofs and testimonials good by bringing them at places (screens on your website / app, broadcasting emails, marketing campaigns, blogs, etc.) where the customers happen to visit on a daily basis. The reviews from customers are a moot point if the other customers can’t see them or learn from them.
9. Customer Support & Communication : As a Product Manager, how much do you interact with your Customer Care (Ops Team)? Have you ever tried to understand how easy / difficult the customer interactions are, that these Ops agents have to do on an hourly (even minute TBH) basis. And if in case, the Ops Team doesn’t have enough resources to invest into the improvement of the quality of their Operations, as a Product Manager it can become an opportunity for you to build certain solutions that helps the Ops team in improving the quality of their customer interactions. Here, while you are trying to help your own Ops Team; however if you look at it, indirectly your customers are getting benefited. Remember the last time you had a bad interaction yourself, with any company’s customer care; I am sure you would have decided not to return to that website anytime soon. Precisely, your own customers may be carrying the same mind-set. The customers really tend to get impressed by companies’ customer care and the quality of the interactions that happen on the call. You impress your customers with high-quality interactions and see how they become your most loyal customers for good.
10. Exclusive Access & Early Releases : Once you have identified your set of loyal customers, make them feel important and exclusive. How? Let’s say, as a Product Manager, you are about to roll out an interesting and long-awaited product feature next week. Your market is already aware of such a product feature becoming available soon. What you can probably do is that you roll out an early access of that feature to the cohort of your most loyal customers. These loyal customers will feel very exclusive and the bond between your organization and these customers will only improve and get stronger over time.
11. Community Building : As a Product Manager, your religion should be to get as many followers as possible. I am not trying to trigger communal ideologies here :p
    Or may be I am (however, not on a political note). The society that you stay in, has a community of people who share similar interests or just a mind-set, may be. Now juxtaposing that with your product ecosystem — on a daily basis you have customers visiting your website / platform and they keep coming back time to time. That organically develops a community of such customers who are undoubtedly your loyal customers. Now, if these loyal customers can interact with each other through the mediums of forums, user-generated content, community events, etc. and that too all on your website / platform; it just directly adds to your customer’s satisfaction and the CLV is greatly enhanced. A sense of belonging is developed which leads to a sustained engagement with the customers.
12. Email Marketing Campaigns : When I was working as a Product Manager for an OTA back in 2016, I always maintained a special place and adherence for emails as a marketing tool, in my heart. I shall try to explain why? While, it may take the world for you at times, to bring people to your platform and make then purchase / order anything. Traffic building isn’t easy. However, what can relatively be easier and achieved with sheer art and knack of communication is email marketing. If the customers can’t come to your platform, you can reach them through emails. You always have the options to broadcast emails with relevant content and campaigns that resonate with your customers’ interest. For instance you have a new offering, you can shoot an email to all your existing consumer base. This will keep your customers engaged and in turn your CLV will shoot up.
13. Gamification : Product Managers in the IoT / SaaS and E-commerce space specially have open-heartedly accepted the trends of gamification. They have started introducing reward system, wherein customers are rewarded based on their participation / performance in challenges. All the achievements of the customers are announced at a leaderboard level, which in turn really excites the customer and he/she is propelled to stick for long. Rewarding is the key to happiness and loyalty.
14. Regular Updates & Innovations : The idea here is to keep your website updated and fresh by regularly updating content and introducing new features / innovation that keeps your customers hooked. Staying relevant in the market through regular updates and innovations, encourages your customers to continue choosing your brand.
15. Feedback & Surveys : Collecting feedback from customers through surveys or other means, lets you understand that what is it the customers really want from your platform and if there is anything that your platform is lacking out on. Understanding customer preferences and addressing their concerns through the feedback received, lets you improve continually and tailor your offerings to the needs of those customers.

So, Conversion Rate, NPS / CSAT & CLV are like my top 3 most favorite product metrics that any Product Manager, regardless what is the nature of the product or the type of domain that the product belongs to.

There are though more metrics that would be specific to a particular domain — OTA, E-commerce, Re-commerce, B2B SaaS, etc. I can share my set of metrics with you if need be. Just drop a comment and I shall know.

Coming to the Hot Potato now…..

A hot potato

Why do I call Conversational AI products, a Hot Potato?

Before I answer that, let me quickly explain what does a hot potato really mean!

The term “hot potato” is often used metaphorically to describe something that is controversial, sensitive, or difficult to handle.

Now, several reasons why Conversational A.I. is a hot potato amidst the abrupt uprising in the interests and aspirations related to the adoption of Conversational AI in the day-to-day world of Tech / IoT when the science and math behind it is fairly mis-understood or half-understood, if you will. Furthermore, why does it become increasingly important to look for the most fitting and relevant North Star metric for any Conversational AI product…..

Let’s look at both these points more closely : — — — — — — —

Why is Conversational AI a hot-potato?

1. Ethical Concerns : Conversational AIs involve interactions with users, and there are ethical implications of the conversations generated by AI. There is moreover a very serious concern of data privacy and the potential misuse. The developers and businesses should navigate these ethical considerations very carefully, in order to stay ethically correct and relevant.
2. Bias and Fairness : The Conversational AI systems are a little infamous for inadvertently perpetuating biases that are present in the training data with which they are trained. Ensuring fairness and eliminating such biases becomes a challenging task for Conversational AI developers and Product Managers, but undoubtedly is a really important thing to handle.
3. Miscommunication Risks : The Conversational AI systems are always at a risk of misinterpretation. What this means is that there are always considerable chances that the Conversational AI may wrongly interpret the intent of the user from his/her message and provide inaccurate or irrelevant responses, thus leading to user dissatisfaction / misinformation.
4. Legal and Regulatory Challenges : The Conversational AI systems can face legal challenges due to the data protection, consent and compliance concerns related to the development and consumption of such Conversational AI products.
   There are industry-specific regulations in place that may often get overlooked while training a Conversational AI model around the type of data that can be used for training, the consent that needs to come from the owners of the data, and the compliance terms that need to be adhered to, which define the boundaries for the implementation and consumption of any product (mostly Machine Learning models these days). These boundaries define the limit to which you as a developer can go to leverage publicly available data and produce the desirable results.
5. Security and Privacy Issues : Any Conversational AI system involves processing and storing user data. As a Product Manager, you need to lay out strict guidelines for your developers to ensure that the data security and privacy isn’t breached as it may lead to a lot of legal implications later on.
6. User Trust and Acceptance : Whenever I have interacted with a Conversational AI as a user, I have had this strange (but a valid one, in the retrospect) fear that are my conversations being kept private or are they subject to any third-party surveillance or data collection. In fact, ever since the Cambridge Analytica scandal happened, the netizens are always extremely cautious when sharing / communication any sort of personal information online. In view of all of this — users really need to feel confident that their interactions are secure, private, and reliable. Any breach of trust can be detrimental to the product’s success.
7. Constant Technological Evolution : Conversational AI is a rapidly evolving field. It becomes imperative for the Product Managers to keep up with the latest technological advancements in the field of Natural Language Processing (NLP). All this would help is adapting to user expectations and which will require a lot of ongoing efforts and resources, that shall invite attention and remarks from the people around.
8. Integration Complexity : What often also makes Conversational AI systems difficult to handle is the integration of them into the existing systems and workflows. As a developer or manager, you would want that your Conversational AI product gets seamlessly integrated and remains compatible with diverse platforms. This poses serious challenges for development and implementation.
9. User Experience Expectations : The users of Conversational AIs have very high expectations for natural and human-like interactions with Conversational AIs. To meet these expectations while avoiding misunderstandings and misinterpretation, becomes a force to deal with. Surely, a trait of a hot potato.
10. Competitive Landscape : At every nook and corner, you will see or hear about an organization that is either building a Conversational AI or investing in one. There are numerous players who are vying for market share. Staying ahead in terms of innovation, user experience, and reliability becomes crucial for success, making this space of Conversional AI even more of a hot potato.

May be it was a lot of reading, around the rationale behind calling Conversational AI space a hot potato.

To make it quick and brief from here, let me simply dive into the hunt of the most relevant and single-perfect North Star metric for a Conversational AI.
And just before that — let’s quickly define a North Star Metric :-

Show Time now : Hunting of the most relevant and single-perfect North Star metric for a Conversational AI!

I shall not conclude the hunt today. I will wait for a few days. If you have read it so far, I really want you to comment your North Star Metric for your Conversational AI Product or whatever product you are building.

See you soon, friend :)