Originally published in Forbes
I’ll break it to you gently: Magic crystal balls don’t exist. But there’s good news. We have the next best thing: probability.
A probability isn’t a very technical thing at all. It’s a reading on how likely something is to happen. It’s a number between 0 and 100 where 100 means it’s predicted to definitely happen and 0 means it’s predicted to definitely not happen. Usually, it lands somewhere on the range in between.
The reason that probability as a field has a super-technical reputation is that the math to calculate each individual probability can be quite sophisticated. But if you’ve been given a probability that someone else’s computer program has calculated, you can follow the guidance it provides without delving into the intricacies of how it came to be.
Probabilities truly pay off when there are a lot of them—when a computer gives you one for each situation, such as each customer who may buy, each transaction that may be fraudulent or each factory machine that may break down. Any individual probability might be entirely off the mark—see above where we discussed our lack of crystal balls—but acting across a number of them, each guiding the decision for an individual case, pays off handsomely. Business is a numbers game and you just tipped the odds in your favor.
That, my friends, is the definition of predictive AI. It’s the use of a predictive model generated by machine learning—in that way, “learned” from prior examples—to drive decisions with probabilities: Whether to call, mail, approve, test, diagnose, warn, investigate, incarcerate, set up on a date or medicate.
Are probabilities really predictions? Essentially, yes. Just pick a cut-off point, such as 50%, and say that every probability above that point counts as a “yes” prediction. Many of the predictions may be wrong, but they’re still predictions.
But predicting isn’t the end game: The purpose is to drive decisions. So, choosing that cut-off point is key. That’s how you purpose an ML model and its outputs into systematic decision making. For that reason, this cut-off is called the decision threshold.
It’s up to the humans. Project stakeholders must navigate the choice of decision threshold by weighing business considerations. The number crunching can’t determine it alone. Once ML generates a model that calculates probabilities, it’s up to you to determine how to act on those probabilities.
In this way, each predictive AI project offers a powerful range of options. One ML model can be used in various ways. For example, here’s a chart plotting the decision threshold against the projected dollars saved for fraud detection:
A chart plotting the decision threshold against the projected dollars saved for fraud detection. The transactions are ordered from left to right as most likely to be fraudulent down to least likely, according to the model.
You get to choose the horizontal position, for which the vertical axis represents the expected business win: the dollars saved using a model to decide which payment card transactions to block as potentially fraudulent.
In choosing that cut-off, you strike a balance. Toward the left, you may be setting the decision threshold so that too few transactions are blocked, which means less fraud is being prevented. On the other hand, toward the right, you’re blocking more transactions that will turn out to be not fraudulent, which also incurs a cost: Inconvenienced payment cardholders will stop using the card. But if you draw the line at the 3.8% peak—so that the riskiest 3.8% of transactions are blocked—you maximize the projected savings (this is across 10 million transactions, based on a decision tree model trained over a dataset of payment card transactions).
The monetary bottom line isn’t the only factor. Stakeholders must also consider other tradeoffs, such as the sheer number of wrongly blocked, inconvenienced customers at each threshold position considered.
It’s the same paradigm for most predictive AI projects: To target operations with an ML model, you must set a decision threshold. Project stakeholders must determine how many cases to treat—to contact, investigate, etc.—by considering the expected business impact across the continuum of possible threshold settings.
In the AI world, viewing this spectrum and the projected business value of each position on the spectrum is not yet an established norm. This contributes centrally to the prevalent failure of predictive AI projects. The remedy is simple: Make sure your data scientist delivers a view like the one in the chart above.
About the author
Eric Siegel is a leading consultant and former Columbia University professor who helps companies deploy machine learning. He is the founder of the long-running Machine Learning Week conference series, the instructor of the acclaimed online course “Machine Learning Leadership and Practice – End-to-End Mastery,” executive editor of The Machine Learning Times and a frequent keynote speaker. He wrote the bestselling Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die, which has been used in courses at hundreds of universities, as well as The AI Playbook: Mastering the Rare Art of Machine Learning Deployment. Eric’s interdisciplinary work bridges the stubborn technology/business gap. At Columbia, he won the Distinguished Faculty award when teaching the graduate computer science courses in ML and AI. Later, he served as a business school professor at UVA Darden. Eric also publishes op-eds on analytics and social justice. You can follow him on LinkedIn.
