In the 1980s, AT&T commissioned McKinsey to estimate the size of the cellular telephone market. Their report estimated that at the peak there would be 900,000 cellphone subscribers. That is a far cry from reality, today it is estimated that 85-90% of the world owns a phone. People love to bring up this example to poke fun at McKinsey and at humans' general inability to estimate. But if you look a little deeper this miss makes more sense. At the time they did the study, phones were expensive, unreliable, clunky, and just plain ugly. It is hard to feel like that is a product that will take off. Until recently, AI has been in somewhat of a similar boat. It can be expensive, requires technical expertise, and things can get ugly if it is used incorrectly. But things are changing.
Kaggle released their state of AI report for 2023. You probably don’t need a report to tell you that AI’s moment has arrived. It’s a huge buzzword and everyone is rushing to cram AI into their products. But despite the trendiness of AI, I found the report to be valuable, you can find a summary here. The big takeaway of the report is that while LLMs and GPT are the hot topics of today, there is so much more to AI. Many areas are ripe with opportunity. As I read through it, I started thinking about what this means for PMs. There have been AI waves before. Many companies have been using AI/ML/predictive analytics to power their products for a long time. But this time feels different. It feels like the entire tech industry will be caught by this wave. To be seen is if they ride that wave or drown. For product managers, there is an opportunity to carve a new niche and a new specialization. AI represents a chance for PMs to help shape the future of the industry.
It is time for the AI Product Manager. This concept was inspired by swyx. He contends that there is the need and the capacity for a new type of engineer to arise during this AI craze. He's talked about it in depth on his podcast and later in a post. There will always be a need for traditional engineers. They build the frontends and backends of the apps we all use every day. And we also need ML engineers who spend their time refining existing models and building new ones. But between them, there is now the opportunity for engineers who build things based off of AI APIs. They get all the benefits of using models without ever having to train a model. With the speed of AI development, there will be more and more tools to build on top of.
I see product laying out on a similar spectrum. PMs are often divided into customer-focused Software PMs and developer /platform-focused Technical PMs. (This is a generalization for simplicity, there are many other PM types out there.) PMs working on ML models and analytics would be more on the Technical end of the spectrum. They work with their engineering counterparts to build models that can be used by their software counterparts. It is a tough and never-ending job, it doesn’t look much like what people think of as traditional product management. It’s lots of testing hypotheses and finding out that a model can’t be built. Or one can be built but it isn’t accurate. Or that it is accurate for some customers, but not all customers. Or it was working when it went to production, but now it is regressing and needs more fine-tuning. You build a lot of things that never make it to production and the ones that do can be a beast to maintain.
With the prevalence of easy-to-access and easy-to-understand models (think GPT), there is an opportunity for PMs to cut a new path. PMs can dive in and become experts in these ready-built and ready-to-use models. And then use their unique customer perspective to innovate on ways to use these models. These PMs should spend their time researching and understanding the latest models. They should understand how others are using them. Then they can test these models for themselves by building proofs of concept. Finally they work with their engineering counterparts to get these models into products.
Who is An AI PM?
This sounds like a lot, but the good news is that anyone can be an AI PM. Here are a few example profiles of PMs who have gotten into the AI space:
A PM who led the build of a new product at a predictive analytics company. Before taking the job they didn't have technical experience or even exposure to AI.
A PM working to integrate AI into business intelligence. Focused on delivering AI insights to customers at the place they need them. They had no AI (or industry) experience before getting started.
A PM building LLM-based side projects on the nights and weekends. They have to act as both the PM and engineer, understanding what to build and then figuring out how to build it.
None of these people were "qualified" but they all dove in and figured it out. Anyone can be an AI PM if they are willing to learn and passionate about problem-solving. And frankly, we need people from all kinds of backgrounds who are willing to lead in AI. Some people will start with an advantage, technical PMs will grok the tech and how it works faster. And software PMs are more likely to understand how to apply it to customers' needs. That is why we need everyone to bring their strengths to AI. And the good news is that you can bring what you already love to do and apply it to AI work. And eventually, everyone will likely be an AI PM. You can choose to be one now and be ahead of the pack or forced to be one later on.
What Does an AI PM Do?
If Software PMs build great customer-facing products. And Technical/Platform PMs build the tools that power those products. What is the charge of AI PMs? Their first and most important job is to know when a problem or opportunity will benefit from the use of AI. Let’s be frank, AI is sexy right now, everyone wants AI everywhere, or at least they think they do. But the reality is that AI is not always the answer. There are many places where AI will just make things more complicated. Or there may be an existing solution that is more simple and does the job as well. This means AI PMs need to be knowledgeable in two ways. They need to flex to the right on the spectrum so that they can deeply understand the customer and their wants and needs. They need to have all the empathy and user vision of a software PM. And then they need to be able to flex to the left on the spectrum and be an expert in the technology. They need to dive into white papers, listen to podcasts, and go hands-on with the tools. They need to understand what models are out there and the types of problems that each is best suited to solve.
Once you’ve done all that (should be light work) it’s time to put things into action. When looking at how an AI PM solves problems, I look at it in 3 approaches. The first is they use models to answer customer’s questions before they ask them. There are things that customers do every single day that are not their actual work. It is pre-work to doing their actual work. This could be things like generating sales emails or compiling forecasts in spreadsheets. AI can do all that for them. It can do the prep work for the customer so they can start at the decision point.
Once you have got that nailed down it is time for the second approach. This one is like the first but goes deeper into the customer's business and data. You should use models to answer the questions that customers didn’t know they should be asking. Using their data and the unique ability of AI you should reveal trends and opportunities that they may not otherwise see. This approach goes beyond helping your customers do better in their existing work which we did above. Instead, this unlocks new opportunities for their businesses.
The third approach is to make it so that customers don’t worry about whether AI is involved in a product. This may seem counterintuitive for an AI PM but part of the job is knowing when to use AI. Ultimately, customers want answers they can trust, when they need them, and where they need them. As an AI PM, you should care about where the answer comes from, that is why you did all that research and testing of models. You should know which models it does or does not make sense to use for a certain problem. You should determine how to best integrate these models into your product. You should create a delightful and easy-to-understand solution for customers. You should care about how things work under the hood, but customers shouldn’t have to worry about any of that. They should be able to get the answers they need and act on them. They should get all the value without any of the cognitive work.
Where Are the AI Opportunities?
We’ve talked about what AI PMs do and how they should help customers. But where should they be doing this? Where are the areas that make the most sense to be applying AI? There isn’t one right answer. Rather it is on you as a PM to figure out what customer problems are worth solving and if AI is the right way to solve them. But to get you thinking here are some areas that are AI-based or are ripe for AI disruption:
Language-based systems: Large Language Models are having a moment. Depending on who you talk to, you might get the impression that AI = LLM. The current hype has more or less made them synonyms. While AI is much more expansive than LLMs, they have a role to play. They can do a lot including chatbots, search, sentiment analysis, summarization, and language translation. The products that do these best are the ones that tap into a unique dataset. Anyone can build a wrapper around the GPT API. Having proprietary data available to your LLM is what makes these experiences unique.
Synthetic data generation: One of the places many products struggle with is test data. There is usually one data set of mock data that is used for everything. It gets created once and is never updated. It worked well as you were starting out, but now it doesn’t cover all your latest features. And it is never as robust or nuanced as your customer’s data. Creating and maintaining that data is a pain. But now you can turn that job over to AI. Give it the job of creating more practical data for internal use. It can also provide more variety to customers using mock data in trials. Creating test data for products is only one of the ways of using this capability. It could also generate test responses to a survey. This will prevent you from sending it to customers only to realize you are asking the wrong questions. In machine learning applications it can generate training data representing real-world data. This allows for model development while preserving privacy and security.
AutoML: More details below on how PMs should be using AutoML themselves. While most of the big cloud players already offer AutoML, it is not approachable enough for the average user. There is an opportunity here to create simple tools focused on data prep, feature engineering, model selection, and tuning. It is a chance to democratize machine learning capabilities.
So we know what an AI PM is and the spaces they should be exploring. But how do they explore? What are the tools they should have in their belt? The great thing about being an AI PM is the tools and models are so accessible. You can go far and do high-fidelity discovery without needing to distract your engineering peers. There are so many tools out there, with seemingly 100’s more released each week. And if a tool doesn’t exist, it will soon. Places like Product Hunt and Reddit are great for learning about what is available. To get you started here are a few of the things every AI PM should be using:
ChatGPT Plus: At this point, I assume most of us have experienced ChatGPT. Frankly, there may have been too much hype when it came out. It can do some interesting things but there was a camp that thought it was an answer to all our problems. LLMs are good for a lot of things but they aren’t the AI model for anything. That is why ChatGPT Plus is so powerful. It gives you the convenience of using ChatGPT but boosts its abilities with other tools. You can browse the web, generate images, analyze data, generate & run code, and upload PDFs for analysis. In previous iterations, you had to select which capability to use and could only use one. But the latest update allows for all to work together. This gives you a near-limitless toolset to play with. Instead of going straight to the GPT API, I’ve been able to use ChatGPT to mock up user flows and test the abilities of GPT in a fast and iterative way. Once I know that something is possible, I can begin to figure out how to execute it with the API inside of my own product. (I originally listed Claude as another tool to explore based on its large token limit and ability to upload PDFs. But ChatGPT just added both of those same features and now Claude feels redundant).
ChatGPT Plugins: One other element of ChatGPT to call out is plug-ins. If any of the previous ChatGPT tools don’t get you what you need, you can also use plugins. Plugins are 3rd party tools that provide ChatGPT with more abilities. One example that I use often is Definitive Intelligence. Their plug-in allows you to search hundreds of databases for additional information that may not be in ChatGPT’s training data. For PMs the plugins are interesting for a few reasons: they give you more tools to build with, they give you a litany of examples of creative ways to integrate AI into your products, and they give you insight into how AI tools work. In the case of Definitive, they show you step by step how they take your single question and turn it into multiple prompts and queries to construct an answer. The one drawback with plug-ins is that they are all chat-based, and for many use cases, chat is not the right solution. Companies have begun to learn that for themselves. Some have removed their plug-ins in cases where they cannot provide a good user experience. As long as you don’t take the chat interface as gospel, they can teach you a lot about ways to integrate AI into customer workflows.
Perplexity: Need an LLM that can search the web and provide you with citations? You may think that ChatGPT is the tool for this, especially now that its knowledge cutoff has been updated and it can search the internet. Even with those enhancements, it is still far behind Perplexity. Perplexity can get you answers in the format you need, with citations, all with little to no delay (no staring at a spinner each time you ask a question). When put head to head with ChatGPT it outperforms in terms of speed, quality, and completeness. If you need an LLM-based way to search the web, Perplexity is the way to go.
AutoML: That’s enough for LLM’s. What if you want to do some more traditional machine learning work? You’re in luck, at least when it comes to training models. All the big platforms (AWS, GCP, Azure) offer UI-based AutoML. With AutoML you can upload datasets, define your targets, and train models. The platforms will do the hard work to run a bunch of different libraries against your data and determine which one makes the most sense for your goals. You can review the results and detailed procedures. Things get a bit trickier at this point depending on the platform you choose, you can only go so far before you run out of UI capabilities. If you want to go further, some platforms expect you to deploy the model as an API endpoint that you need to hit with code or an API tool. When you get to this point talk to your friendly neighborhood engineer and they should be able to help you. But training a model is a good experience on its own, if you only do that you will be ahead of the pack.
Replit: Replit wants everyone to become an engineer. Or at least have the ability to build and launch things into the world, this includes PMs. It’s a great tool for building proofs of concept and v1 products. Use it to build internal tools for yourself and mock-up new features for customers. This talk by Replit’s CEO does a great job of laying out Replit’s vision for the future of AI and how everyone will use it. They recently added an AI co-pilot named Ghostwriter. You can use it to help you build just about anything. Ghostwriter also provides PMs with an example of how to build a true chat-based, co-pilot. It feels well-designed and integrated into the product. It stands out in comparison to the ChatGPT wrappers that many companies have stuck in the lower-right corner of their product.
GPTs: A speculative and last-minute addition to the list. OpenAI just announced that they will allow anyone to build a custom version of ChatGPT. You will be able to give it custom instructions, seed it with proprietary data, and give it the power to take action (think use the Zapier API). And the most interesting part is they are programmed using natural language and no code. Soon anyone will be able to build AI agents and deploy them into the world. (There is even an app store coming where you can share your creations with others). Plug-ins require some technical chops but GPTs removed that barrier. Every PM should be building with these as soon as they are available. (Update: I just got access to these last night, and it feels like a true revolution. In 30 minutes, using just my phone I was able to spin up a few different, purpose-built tools. For example, one helps product managers craft a product brief. It does things like trend and competitor research on behalf of the PM. It suggests metrics for tracking value and even provides sample SQL for those metrics. It took me under 7 minutes to build and should save me hours each time I research a new product or feature).
These tools play many roles in the life of a PM. We should be using them to make ourselves better and more capable. We should be using them to create PoCs for our products. And they should be inspirations for how to better integrate AI into our products.
Stick to the Basics
There is a lot to explore, it is easy to get excited, to want to dive in and solve everything. But before you get started let’s review the basics. The things AI PMs should be doing to set themselves, their products, and their customers up for success:
Start by deeply understanding user needs and challenges. Ideate on how AI could uniquely address them. Paul Adams the CPO of Intercom was recently on Lenny’s podcast talking about this subject, he said: “...then, ask the question. So go back to basics. ‘Okay, what is my product for? And why do people love it?' And then ask, ‘Can AI do that?’ And for a lot, the answer's going to be, ‘Yes, it can.’ For some, it might be, ‘It can partially do it.’ And then, maybe for others, ‘It can't do that, at least not yet.’”
Focus on incremental value vs complete AI replacement. This reduces risk and avoids disrupting customers' workflows. Hashboard recently shared a case study where an AI-first piece of functionality was too much for customers. It turned out to not be helpful. In the end, they pivoted to using AI to improve their search capabilities which has been well-received.
Use the tools - you should live in them every day. If you don’t understand how they work, you are not going to be in a position to understand if they can solve customers' problems.
Adopt an experimentation mindset - prototype and validate with users. While ChatGPT plugins have flaws they are a great prototyping ground. Lucid just released a large set of collaborative AI features inside their app. Before they developed those, they released a plugin. It allowed them to start gauging customer interest before investing more in the AI space.
Keep learning, AI is evolving so quickly that you need a growth mindset or you will fall behind. Take courses, subscribe to newsletters, listen to podcasts, and look for other content on the latest in AI.
Make ethics, trust, and transparency a priority. Understand how model cards work and their components. Kaggle has a great AI Ethics course that you can take for free.
Looking to the Future
There have been hype cycles before (RIP to all the NFT PMs out there 🪦) but AI feels like it is here to stay. It feels like AI’s time to go mainstream is here. Things that seem novel today, will soon be table stakes in all products. The models and tools are easy to understand, accessible, scalable, and affordable. By getting started now, you are setting your products and customers to be ahead of the curve. And you are also setting yourself up to be a leader in tech’s newest breakthrough. PMs with passion and drive in this area will be able to carve their career path. There’s lots to be excited about with AI and like most things, the best way to start is to get out there and get building!
