When A/B testing doesn't tell you the whole story

Each year, marketers spend billions of dollars on campaigns to attract, retain, and sell more products to customers.

To understand the shortcomings of how A/B testing is commonly used, let's take a look at the following hypothetical example:

Imagine you work for a large arts organization that is concerned about declining membership retention rates.

Now, let's say you don't find any difference in retention between members who received the giveaway and those who didn't.

Optimizing the “Churn” Prevention Campaign

Churn rate – Churn rate (sometimes called attrition rate): in the broadest sense, a measure of the number of individuals or items moving out of the collective pool during a specific period of time.

This isn't just hypothetical – in fact, this example is based on a real organization I worked with during my research. When it comes to increasing retention, companies often identify “high-risk” customers – that is, customers whose recent behavior or other characteristics indicate they are particularly likely to unsubscribe or stop purchasing the company's products – and then run A/B testing to determine whether their retention campaigns are effective with this group. While this is an understandable strategy, my research shows that it can be seriously counterproductive, as it can cause marketers to make poor decisions that actually reduce overall retention and ROI on marketing spend.

Specifically, I conducted field tests with two large companies implementing user retention campaigns.

Across both companies, I found that customers identified as at highest risk of churn were not necessarily the best targets for retention programs – in fact, there was little correlation between a customer's level of churn risk and their susceptibility to interventions.

Of course, the specific factors that make customers more likely to be receptive to a retention campaign will vary from organization to organization and even from campaign to campaign, but running an experiment as described above can help you identify the characteristics that will best predict customer susceptibility to a particular campaign.

For example, one of the organizations in my study was a telecommunications company that had access to detailed data on behavioral metrics such as the number of calls customers made in the previous month, the number of messages they sent, gigabytes of data downloaded, and more.

Moving from Prediction to Prescription

So what does this mean for marketers?

Instead of trying to predict what customers will do (i.e. trying to determine their risk of churn), marketers should focus on how different types of customers will respond to specific campaigns and then design the campaigns that are most likely to be effective in reducing churn among a given group of customers.

———————————————–

The concept of targeted marketing campaigns is nothing new – but it's important to think carefully about how you make those targeting decisions. Instead of simply guessing at what factors might indicate that someone is a strong target or focusing on groups considered high priority (such as customers at high churn risk), companies should target customers who are most susceptible to specific interventions. they are implementing. To maximize ROI, marketers need to stop asking, “Is this intervention effective?” and start asking, “For whom is this intervention most effective?” – and then target their campaigns accordingly.

Eva Ascarza is Associate Professor of Business Administration at Harvard Business School.