Know 5 things to disguise as AI /ML Product Manager

Artificial intelligence is here and is here to stay. As a Product Manager, it is better to develop skill in it. I know we just settled in from the fad of API and then the urgency of knowing about Cloud platforms, Blockchain for a little while, and this new thing again!

I have ~3 years of experience building/working with two ML models. I have developed a perspective- what is our job as an ML/AI Product Manager? and I think it can help you to fake it till you make it.

PS: this is not about the basics of ML. These are straight 5 things you need to know to sound like a Product Manager of an Artificial Intelligence product.

Most of my experience in the AI/ML domain is from building recommendation engines. I believe this is also the most commercially popular application of it. How it works is an ML model would try to classify users and use the interest of the group to generate a suggestion. Getting the classification right is part of the Data scientist. Making data consumable is the job of the Data Engineer. As a Product Manager, you need not know to dive into statistics and programming but only ensure the ML model serves the outcomes it is supposed to, and you do so by giving the right inputs.

#1 Advocate whether you need ML to solve the problem

As a Product manager, your job is to convince your boss your product does not need ML. Building and maintaining an ML model is an expensive and lengthy affair. Most companies building a recommendation engine to promote content efficiently are failing at it, but if your company wants to do the same —

a) First, evaluate the business goal you are trying to achieve and ensure it generates revenue. ML model is expensive, and unless your business can’t survive without it, don’t build it.

My goal was to reduce the churn rate by recommending exclusive content at a discounted rate or with extra benefits.

b) Study the data points, features, or input variables (in Data scientist language) that will help you predict. Make sure they make sense and are not too general.

I used average session time, travel date, search query date, destination, and last login date to deduce if the person is worth giving a discount or if he/she will take a one-time advantage and not return 2nd time.

c) Use alternatives to ML first and see if the need is still there. An alternative to the ML model is the decision tree. If you think the target customer is not evolving rapidly and you will have enough time to readjust your tree, then why not?

I built a decision tree model instead. We gave weight to each criterion that was getting fulfilled, and the system offered users above the threshold a discount when trying to drop in between of sales funnel.

d) Make some predictions manually that you hope the ML model will do automatically and execute them to validate if they meet the business goal. An irrational consumer has sunken many giants.

Our decision tree would assign high total weight to the user who seemed like a potential customer and in urgency but in reality user outcome was less and even those selected by the decision tree didn’t care for a discount, which was surprising because we made really good offers. Hence, the tree was put to sleep after 1 year of ruuning.

#2 Build a Data Strategy

If you must build an ML model, the next step is to make a Data Strategy. As a Product Manager, you are supposed to figure out a reliable and relevant data source. The unavailability of relevant data sources is a good reason to not execute an ML model.

a) You can look internally. Evaluate the data points your product has been capturing (if you are already LIVE). Verify if the data is conclusive enough — at least 2 or 3 years old. This is the least expensive and least litigious option.

b) You can build a relevant product first whose shadow goal would be to source you accurate data. This is a lengthy process, but it helps control the value chain and in the long term, it’s a more flexible and reliable approach. A good example of it is credit card apps like Cred.

c) Open source, Government agencies, Crowdsource. These are inexpensive ways, but generally, the quality is poor and very litigious. An example of open source is Kaggle. Government agency example is Singpass or GovTech in Singapore, Arogyasetu, and HealthCard in India. Facebook asking you to tag photos or Google asking you to verify you are not a robot are examples of Crowdsourcing.

c) You can collaborate with 3rd party vendors to feed and keep feeding you relevant data. The choice of vendor is hard here. Ideally, they should be in the same domain as your business, have a steady source of data themselves, and make sure their data is trustable and authorized to use.
It is expensive. One has to negotiate various rights of usage and levels of privacy. It also runs the risk of legal obligations.

#3 Decide if the ML model needs to be Accurate or Precise or Recall

This is where it gets interesting for a Product Manager. If you have built your ML model now, the question is, what will be the outcome of this?

As a product manager, you decide, depending on the business goals, which of the following will be your ML model’s outcome

a) Accuracy — an accurate ML model takes a high number of correct decisions in minimum attempts. It is the least your model has to do. An ML model has to take the correct decision and gradually learn to do so in fewer attempts.

b) Recall — a good Recall ML model can figure out the number of correct choices from all given the correct choices.

Example, if your model detects Coronavirus in patients, you want it to get 100% of the positive cases. The number of attempts doesn’t matter, but you want the model to get all the right ones.

c) Precise — a precise ML model can figure the number of correct choices in the minimum wrong attempts.

Example, if you are targeting sales leads for up or cross-sell through digital ads and each ad costs you, then you want your model to be right about every suggestion it makes. The number of attempts matters. You can miss out on a few potential leads, but cannot waste an attempt on the wrong one.

d) F1 Score — Recall and Precision are anti. An ML model can have 100% Precision and Recall only when it knows the answers already🙈🤣. If you are in a situation where both precision (can’t waste attempts) and recall (cannot let go of the right ones) are important — the F1 score helps. It is an arithmetic equation to balance out Recall and Precision.

#4 Own the deployment

Once your ML model is ready — it’s been trained, tested, and delivering results then comes deployment. The most popular mode still is API.

ML model can quickly cluster/classify you in your closely resembling group and predict what you will like next or best and send this information via API to the frontend system.

As Product Manager, you will decide, based on the model’s capabilities (processing time of your ML model) and business goals, what will be the frequency of output. It can be daily, weekly, or hourly. But the higher the frequency — the costlier will be infrastructure to keep it running.

#5 ROI of the ML model

When you build the ML model, make sure to timebox it. If not, you are running the risk of it becoming irrelevant. Even if you build it on time, you will have to monitor the response the target audience is giving to the output of the model, as you are running the risk of the audience becoming immune to the recommendations. There is always a risk of your flagship product itself being in nearing the mature or decline stage.

Most of the ML model built as recommendation engine has failed to find relevance. Other better applications for ML models are having groundbreaking success. But As far as we, Product Managers are concerned — if you talk about the above 5 points fluently, it would convince anyone you are building an ML product.