A new age of discovery: Balancing short-term value with long-term innovation

This is part 3 in our Field Guide to Machine Learning Product Development: What Product Managers Need to Know series. Read the Introduction Field Guide to Machine Learning Product Development: What Product Managers Need to Know, learn about how to manage data throughout the machine learning product lifecycle, and stay tuned for more deep dives into the six main stages of the machine learning PM process, starting with Discovery below.

Field Guide to Machine Learning Product Development: Discovery

The development of a machine learning product should always start with discovery. As a product manager, I am constantly asking myself “are we building the right thing?”. It’s a question that overrides all other concerns I might have.

I’m sure I’m not alone: we product managers (PMs) are responsible for ensuring the products our teams build deliver value to our users and our business. We are responsible for ensuring that our clients not only use but choose our product. We are responsible for delivering a quality solution in a timely manner.

Ultimately, answering that question often boils down to ensuring we’ve identified a problem that is not only worth the effort of solving, but also one that our team can solve best. Identifying such a problem is at the core of product discovery. In this post, I share my perspective on product discovery, drawing from experiences as a PM at an AI-first organization that emphasizes product innovation.

Focus on the problem space

Just as you wouldn’t want to set off down a long highway in the wrong direction, product discovery is critical to setting the right direction through the product development lifecycle. Head off in the wrong direction and you could find yourself far from your destination as weeks or even months are lost while the team solves for the wrong problem. Nobody wants that. So, it is critical for product discovery to focus on clearly

identifying the right problem to solve to create real value for the users.

But what is the ‘right problem’ and how does a product team find the right problem?

There are already a lot of good resources out there that explore product discovery in a more traditional setting (i.e., without the complexity of incorporating ML, let alone bespoke, novel ML solutions, into products).

For example, I like how Silicon Valley Product Group’s Marty Cagan lays out 5 areas of risk to explore during product discovery: user value risk, usability risk, feasibility risk, business viability risk, and ethics risk. Product discovery coach Teresa Torres also has a great framework to evaluate opportunities and other useful tips around how to organize findings from the discovery process.

There is also a lot of good content available on how to talk with users and customers to uncover pain points, how to test assumptions, and how to build POCs and prototypes quickly. Indeed, much of the traditional product discovery process, which focuses on the problem space (versus solutioning), remains applicable to a machine learning PM in the innovation space.

What to do when identifying the problem space is not enough

For those PMs who are focused on applying ML techniques in novel and innovative ways, there are two additional considerations from the solution space that need to come into the mix when identifying the ‘right problem’ during discovery.

The first is to consider whether ML is actually the best solution to solve the problem. It is far too easy to get caught up in the excitement of ML and want to apply it to every problem we come across (I’ve been in those conversations before). But an ML solution is relatively complex and comes with its own set of risks and challenges. An ML solution is something you apply only when non-ML techniques are insufficient for solving the problem. Simply put,

just because ML can solve the problem does not mean it’s the best solution for the problem.

When evaluating the applicability of ML to a given problem, I often look at three main axes:

The second consideration in identifying the ‘right problem’ is the opportunity for unique ML contributions. You may find it is possible to use ML in your product without any novel ML research. But for PMs at organizations with teams of researchers aiming to develop novel ML techniques (like those at RBC Borealis), we must also evaluate whether there is an opportunity for unique ML contributions in the products we develop.

It is a tricky consideration for PMs to balance. On the one hand, we know that ‘cool research’ should never be the rationale for creating a product; discovery is about understanding the user and business problems. But on the other hand, PMs should also avoid taking too narrow a lens by focusing solely on the immediate problem, and think more broadly about how our users, businesses and problem space will evolve. This can often uncover even bigger problems that can lead to greater innovation and space for novel research to be applied, to deilver better returns for the users and for the business down the road.

Building trust is key to product discovery

At RBC Borealis, we believe that one of the keys to product discovery comes down to the ‘people’ side of the equation. While RBC Borealis has expertise in product, engineering, and ML research, we work closely with partners in the specific lines of businesses at RBC to leverage their business domain expertise. Successful product discovery (recognizing the considerations above) often relies on establishing trust with our business partners and a focus on long-term relationships.

One of ways Borealis has successfully built trust with our business partners is by starting with a focus on delivering shorter-term value for our business partners and their users at the expense of novel technology and high risks, but with the clear intention to collaborate with partners on more innovative high-risk, high-reward solutions over time. We also often create POCs and prototypes that can be helpful in building trust with our partners.

Over time, and as trust with our business partners builds, the ability to discover more innovative product opportunities grows. We understand more about the pain points and trends of the business domain and the users. Our partners understand more about ML, its limitations and its risks. They gain confidence in ML solutions. And that leads to greater mutual trust as we work together to incorporate novel ML research into products that require longer timelines and carry greater risk (though those are also great opportunities for greater reward!).

Always Be Discovering

At RBC Borealis, our culture is centered around novel research and innovation, deeply intertwined with an ambitious product spirit. As product managers, our challenge is to discover the right problem: one that addresses a meaningful pain point for users and the businesses we collaborate with; that requires a complex technology like ML; and where there is opportunity to incorporate novel ML techniques and research over time.

Discovery of the ‘right problem’ to solve relies on the strength of our business partnerships, where trust needs to be established. As the business partnerships develop and evolve, continuous product discovery leads to more opportunities for innovation and novel ML contributions.

So, product managers out there: are we building the right thing? It’s hard to know for sure. But a good product discovery process is an important piece to answering that question.