Originally published in Built In, May 22, 2024.
Your biggest operations are made up of many small decisions. Predictive modeling can help you make them.
Predictive AI is the technology businesses turn to for boosting the performance of almost any kind of existing, large-scale operation across functions.
It learns from data to predict outcomes and behaviors, such as who will click, which vehicle will require maintenance or which transaction will turn out to be fraudulent. These predictions drive millions of operational decisions a day, determining whom to call, mail, approve, test, diagnose, warn, investigate, incarcerate, set up on a date or medicate.
Sound complicated? Actually, predictive scores drive such decisions in a straightforward way. For many predictive AI — aka predictive analytics or enterprise machine learning — initiatives, it’s simply a matter of ranking cases and positioning a decision threshold.
How Does Predictive AI Help Organizations Prioritize?
With modeling, predictive AI can rank individuals with positive cases at the top and fewer positive cases toward the bottom. Organizations can then triage and prioritize, such as by reaching out to their most interested customers, manually auditing more suspicious transactions or inspecting buildings with more hazards.
How Does Predictive AI Work?
Let’s begin with a marketing example.
When I lead AI training workshops, I get everyone to stand up and be the data. Each person holds up a piece of paper with the size of the largest television in their home and the group arranges itself into a row ordered by TV size.
People in order of their TV size (a zero means they have no TV). Image provided by author.
Then I ask a question related to TV usage, such as, “Who has a subscription to Netflix?”
People with a raised hand are subscribed to Netflix. Image provided by author.
The positive cases are more concentrated within the top portion, the left side, of this human dataset. As a point of example, let’s draw a decision threshold after the top 10 — in this case, the decision as to which will receive marketing contact.
Among those 10, 70 percent have Netflix, which is 2.2 times as many as the overall population’s 32 percent (data scientists call this a lift of 2.2). This tells us that you might get a lot more bang for your buck marketing a product that’s linked with Netflix to that top segment, those with larger TVs.
Now, this classroom exercise oversimplifies in a couple ways. It’s a comically small data set, so the results are far from reliable. And the predictions are based on one and only one variable: TV size (that’s what we call a univariate model). Plus, those values used to rank individuals aren’t scaled to be probabilities.
Despite these caveats, the visual effect and sample arithmetic nicely illustrate how predictive AI operates for many of its uses: Many positive cases appear early in the ranking, decreasing down to only small numbers at the end of the list.
To Deploy Predictive AI Is to Triage and Prioritize
Predictive AI uses this same valuable effect across a wide range of use cases. Its predictive scores serve to rank individuals so that the top portion is denser with positive cases and the bottom portion sees far fewer positive cases.
This ranking of individual cases empowers the organization to triage and prioritize. Contact customers more likely to buy. Expend retention efforts on customers more likely to leave. Manually audit transactions more likely to be fraudulent. Inspect buildings more likely to catch fire.
Of course, this also applies for the most literal of triage, medical triage. Tend first to patients scored by a predictive model as more likely to decline in health or more likely to have a positive diagnosis. Reexamine patients predicted as more likely to be readmitted within the next year. Reach out to patients more likely to skip a prescribed medication or healthcare appointment.
Drawing a Profit Curve
The ranked list tells us where to draw the line, or where to set the decision threshold, for driving a yes/no operational decision. Consider the decision as to whether to contact each customer with a marketing outreach. You can see the profit accumulated as we send a promotional brochure to customers, from most likely to buy down to least likely.
Image provided by author.
This is a typical profit curve. The horizontal axis corresponds with how far down the ordered list we’ve gone. As you proceed from left to right, you begin with those scored most highly by the model. At each position, the profit is calculated based on how much we’ve spent to contact that many customers and how much we’ve gained from those who in turn responded with a purchase (view a step-through of the arithmetic here).
Following the upper curve, you can see the campaign’s ups and downs. At the beginning, the more customers you contact, the more your profit goes up. Although you spend more to move along to the right — to contact more and more customers — you’re getting enough positive responses to turn a profit. This is where you’re getting the most bang for your marketing buck.
About one quarter of the way down the list, diminishing returns begin to set in. You’ve exhausted the most responsive portion of the list and you actually start to lose money — the cumulative profit begins to diminish as you contact more customers but you no longer elicit as many positive responses.
The overall marketing campaign is a bust if you actually contact the entire list. If you contact 100 percent of the customers, making your way to the far right of the graph, you end up with a loss of about $550,000.
For the example profit curve shown, if you have no model at all, you only lose. To visualize that situation, the straight, lower line shows what would happen without a model and therefore without any means to order the list.
By following an effectively random order, you would lose money at a constant rate as you make your way through the list, so the lower line just keeps making a beeline down to the final end result. That line serves as a baseline for comparison. In contrast, the rise and eventual fall of the upper profit line is a testimony to how much value a model can deliver.
Either way, at the far right, you end up at the same place, losing about $550,000. This is because, if you’re marketing to everyone, the order in which you do so doesn’t matter. You always end up with the same overall loss, a negative profit. If you intend on just contacting everyone, you aren’t targeting so there’s no purpose to having a predictive model.
Where to Stop: the Decision Threshold
Turning back to the upper profit curve, you’re probably feeling the urge to slam on the brakes, perhaps around the 25 percent mark. If you stopped there, your profit would be $350,000. That’s often the best choice, but it’s not an absolute.
Sometimes, the marketing benefit of contacting more people takes a higher strategic priority, even if doing so isn’t reflected in immediate-term profits. In that case, you may argue that stopping around 72 percent where you break even would be a much better choice than spending more than half a million dollars to contact everyone.
That way, you basically get to market to almost three quarters of the list for free. Ultimately, the choice depends on the longer-term marketing strategy and other pragmatic factors at your organization. In any case, a profit curve like this one helps guide that choice.
Image provided by MIT Press.
Before deployment, you can draw this kind of projected curve just the same for most any predictive AI project. To help decide how many individuals to target, you view the spectrum of options corresponding with how the model has ranked individuals.
As you move along the spectrum, deciding how many to contact, approve for a loan or audit for fraud, you often see the same pattern: an upward ride followed by a decline. There’s a sweet spot, a Goldilocks zone, that’s often the best place to stop. By establishing a decision threshold, a cutoff point, at that position, the model will then serve to be selective, targeting for treatment those who scored above the threshold.
In this way, one model provides a whole range of options. When you draw the line, you’re establishing which option to go with in deployment. You’re deciding precisely how to use the model.
In the end, your most important and largest-scale operations consist of many decisions. Predictive scores can directly drive those decisions by applying a simple threshold.
This article is excerpted from the book, The AI Playbook: Mastering the Rare Art of Machine Learning Deployment, with permission from the publisher, MIT Press.
Follow me on Twitter or LinkedIn. Check out my website or some of my other work here.
About the author
Eric Siegel is a leading consultant and former Columbia University professor who helps companies deploy machine learning. He is the founder of the long-running Machine Learning Week conference series and its new sister, Generative AI Applications Summit, the instructor of the acclaimed online course “Machine Learning Leadership and Practice – End-to-End Mastery,” executive editor of The Machine Learning Times and a frequent keynote speaker. He wrote the bestselling Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die, which has been used in courses at hundreds of universities, as well as The AI Playbook: Mastering the Rare Art of Machine Learning Deployment. Eric’s interdisciplinary work bridges the stubborn technology/business gap. At Columbia, he won the Distinguished Faculty award when teaching the graduate computer science courses in ML and AI. Later, he served as a business school professor at UVA Darden. Eric also publishes op-eds on analytics and social justice. You can follow him on LinkedIn.
 |
 |
 |
