If you’ve ever had a data scientist make a machine learning model for you, you probably first experienced excitement, followed by bewilderment.

The potential power is awesome. In its enterprise application, predictive AI is the antidote to information overload. Too many prospective customers? Use a predictive model to prioritize them from most likely to buy down to least likely. Too many web pages that might have the info you need? Google Search uses a model that orders the search results, placing those web pages most likely to pertain at the top. Too many buildings to inspect for fire hazards? The New York City Fire Department’s model tells them which should be prioritized as high risk.

Eventually, you will sit down with your quant to hear how her number crunching went. “The predictive model is great!” she exclaims. “This is a nice dataset, so we ended up with an AUC of 0.83!”

“What’s that?” you’re forced to ask.

“Oh, I mean the area under the receiver operating characteristic curve.”

“Right… so, is 0.83 good?” you ask, hopefully.

“Totally! This model is significantly better than not having a model. It’s, like, much better than random guessing.”

This ML professional is providing you concrete numbers, but she’s not helping you decide what business move to make. You ask, “Okay, so, if we use the model to target our next direct mail send, how much money will it make us?”

“Oh, I’m sure a lot. I mean, that way you’ll be contacting customers more likely to buy, so how could it not?”

It’s strange: Her expertise is credible and you don’t doubt anything she’s saying, but you still feel like you have to corner her. “Bottom line: How much will the marketing profit improve if we only contact the top 20% of our prospect list?”

“Oh… yeah, I could totally estimate that. Let’s meet again tomorrow.”

The next day, you get your answer. “The profit would be an estimated $1.6 million, which is much better than if you did an untargeted mass mailing to the entire list, which would put you in the red by almost $800,000.”

This sounds good, but you feel like, after pressing so hard, you’ve only been given one peek through a keyhole. You need broader visibility. What if you marketed to the top 15% or 35%? Would either of those be better?

Let’s jump to the resolution that would make this awkward yet common conversation unnecessary. The full view that you need is called a profit curve, a rare yet fundamental view of predictive AI’s potential value.

A profit curve estimating the profit of a marketing campaign targeted with a machine learning model. The horizontal axis represents the number of customers contacted—ordered from most likely to buy if contacted down to least likely—and the vertical axis represents the marketing campaign’s profit. The gray vertical line represents the decision threshold. If set to 30% (as shown), the campaign’s profit is estimated to be $1.7 million, depending on the settings at the bottom: an average of $30 for each positive customer response and a $10 cost per contact. This example is based on the data set used for The Second International Knowledge Discovery and Data Mining Tools Competition. Thanks to Dean Abbott for the predictive model used in this example.

The profit curve shows you the range of options offered by a predictive model. With the example curve shown above, 30% is the best stopping point if your aim is to maximize profit. Or, if immediate-term profit isn’t your only priority and, instead, you’d like to market to many customers, then 84% would be a break-even point—you could contact that many for free, which would arguably be a better choice than contacting all of the list at a loss of almost $800,000, the negative profit shown at the far right side of the curve. Across this range of cut-off options, the position you select is called the decision threshold.

As simple and vital as it is, this visual is rare. No matter the pedigree of your quant, you probably won’t get it unless you press hard. Why? It’s not in the typical data scientist’s wheelhouse. It’s not part of the culture, nowhere to be found within their technical books or university courses. Perhaps more importantly, the ML software they use probably doesn’t have it built in—at least not in a useful, extensible way.

After all, such a business-focused vantage sits outside a data scientist’s proper scope of responsibilities. They know how to navigate technical pitfalls as they guide their machine to learn from data how to predict. And they know how to report on the success of their modeling work—using metrics that report on a model’s pure predictive performance. But what about the potential value of the model in straightforward business terms like profit? Estimating that is someone else’s job, the domain of business leaders who, ideally, will connect the model to their operations, improving the operations with the model’s predictions. The thing is, those leaders consider this a technical step that belongs to the data scientist, and rightly so.

Because visibility into the potential business performance of ML models is rarely granted, enterprise ML projects routinely fail to deploy. This is part and parcel of the stubborn business/tech divide, a disconnect between stakeholders and quants wherein they’re not speaking the same language. But without bridging that gap, it’s almost impossible to make an informed decision about whether and precisely how to deploy an ML model—be it for targeting a marketing campaign or improving any other kind of large-scale operation.

As a business stakeholder, don’t settle. For every ML model that you consider deploying, make sure that your data scientists provide you with a full view of its potential business value—reported in terms of straightforward business metrics such as profit.