Chief Data Scientist John Foreman will be presenting at Predictive Analytics World Boston (Oct 5 – 9) on “Problems, then Techniques, then Toys. Keeping Your Predictive Analytics Right-side Up.”
When we think “data science,” most of the headlines that come to mind are tales of the creepy and the overreaching. And unless you’re Tommy Wiseau, creepy is contrary to a good customer experience. Especially at a company like MailChimp, where I work as Chief Data Scientist.
MailChimp is the world’s largest email service provider, and the way it got there was by placing an enjoyable user experience and intuitive workflow over a tedious part of a marketer’s day. So while MailChimp is swimming in data (we send 10 billion emails a month), this data is not the reason for the site’s success.
Rather, this data is used in service of the user experience.
So how does data science serve the business in a UX-focused or customer-focused world (instead of one that’s all about ad placement a la Google and Facebook)?
Nothing wrong with taking a back seat
MailChimp’s data science team started with meat-and-potatoes operational analytic problems. For example, we built a bunch of AI models that predict and shut down abusers in the system so that they don’t pollute the reputation of the IP addresses our email goes out on. Our users share IP addresses, so you don’t want one bad user peeing in the water supply.
How did this improve the user experience? The mail that good users send through MailChimp doesn’t go to spam. Nice and simple.
Did the user necessarily notice this UX effort? No, but that’s OK.
Here’s another illustration of the same “behind the scenes” approach. We use data to determine whether someone signing up for an account is a bot or not, and if we know for sure they’re not a bot, then we don’t make the user fill out the dreaded CAPTCHA.
Wait … what is that word?
The great thing about using data science to hide CAPTCHA is that the product we built is completely invisible; it’s the removal of an obstacle.
Improving the user experience by employing data has nothing to do with word clouds or infographics or D3 animations, per se. That type of data science is often there to impress the user or to impress the press.
But UX-centric data science is like a good jock strap — it’s there to perform its support function inconspicuously.
Answering the nagging questions
One of the biggest impediments to a user’s success in MailChimp is their own crippling self-doubt. Sure, CAPTCHA stinks, but its annoyingness is nothing compared to the nagging doubts users face before pressing the send button on a newsletter to a large list. Like looking down when walking a tightrope, a lot of knee-weakness comes when sending an email to a million people.
All sorts of questions plague our users. How should I segment my list? What subject line should I choose?
In the absence of data, folklore and anecdote abound, so my favorite way of improving the MailChimp user experience is to put these nagging questions to bed.
And the most nagging of nagging questions was this: When should I send my marketing email?
So we built an entire Send Time Optimization (STO) system to mine this data. And when we ran a bunch of email addresses through STO, the distributions that came back were far more nuanced than folklore, but the answers nonetheless validated common sense. Let’s look at some of the data, starting with recipients in three age groups: college, 40s and over retirement age.
The data shows that the best time to send for any age group is only optimal for 6-7% of email addresses. So when we talk about the “best time to send” we should keep in mind that most people on any marketer’s list are going to deviate at least a little in terms of their email preferences from whatever time is picked.
What’s fascinating here is that college aged recipients have an optimal time of 1 p.m. versus 10 a.m. for the older two categories. College students during the summer likely drag themselves out of bed later and check their email later than older folks.
Like I said, billions of data points can simply validate common sense. That’s okay. Because now we can give an answer to a user, and they can be confident it’s not an anecdote pulled from some suit’s derriere.
Nope, it’s pulled from data’s derriere!
Other than age, what’ve we got? Let’s try location. When we look at the country that subscribers live in, we see shifts as well.
During the summer months, Norway has a very long day, so the distribution of the optimal send times for Norwegian addresses plateaus out from 9 a.m. to 8 p.m. while a country like Egypt experiences a more pronounced fall after typical business hours.
But more than where you live or how old you are, the job you work has a huge effect on when you engage with email. Let’s take a look at three very different occupations: Lawyers, neonatal nurses and bartenders.
Lawyers have a well-defined workday in which they optimally engage with email marketing at 9 am. Bartenders roll out of bed later and aren’t fully engaging until noon. Neonatal nurses on the other hand have little of a pronounced optimum. Rather, they work night and day and their distribution of optimal send times blurs into a plateau all the way past 10 p.m.
But chances are your audience isn’t Norwegian bartenders in their 70s, so if you don’t have access to MailChimp’s STO system, what do you do?
Here’s the thing: When I look at those optimal send time graphs, the one thing that stands out across all of them is that I’m not surprised by any of them.
Since data often validates common sense rather than overturning it, A/B testing common-sense hypotheses is effective if you know your audience. After all, A/B tests on send time give on average a 22% lift in engagement, which is very similar to the performance gains MailChimp users get from STO.
Getting Back to UX
By answering the send time question for MailChimp’s users, the data science team didn’t change the look of the application. We didn’t change any button colors or add some midi music or a D3 graph to a page.
Rather, we helped our customers answer questions best “left to the data,” and in my mind, that’s the purest, least creepy application of data science.
Does your company leverage data for user experience? Tell us about your experience in the comments.
John Foreman is the Chief Data Scientist for MailChimp.com. He’s also a recovering management consultant who’s done a lot of analytics work for large businesses (Coke, Royal Caribbean, Intercontinental Hotels) and the government. These days John does all sorts of awesome data science for MailChimp. John’s book Data Smart: Using Data Science to Transform Information into Insight, is now out from Wiley and is possibly the most legal fun you can have with spreadsheets.
Originally published at mashable.com
 |
 |
 |
