How Intelligent Recommendation Systems Work

An intelligent recommendation system analyzes the available information to produce a detailed, individualized picture of each customer and make predictions about their preferences and behavior, specifically their buying propensity.

Many offers are irrelevant to consumers or may even strike the wrong note. For example, you aren’t putting your best foot forward by recommending a romantic getaway to someone in the middle of a messy divorce, or by listing the benefits of retirement community units to someone under 30. For some customers, the sheer number of offers could annoy them to the point of deleting them all, unread.

Popular product recommendation

The simplest method is to recommend popular products. Recommending what’s popular requires very little analysis. For example, a server in a restaurant uses this method when you ask what they recommend, and they say, “Our Spam tartare is very popular.” Popularity-based recommendations do not consult or track the tastes or behavior of the individual customer. If it’s trending, it’s recommended. This approach will likely perform better than recommending a random product from the catalog, but that isn’t an intelligent recommendation.

Market-basket analysis

A slightly more sophisticated approach, market-basket analysis, takes into consideration what other customers have bought. A common example is Amazon’s recommendation: “Customers who bought [what you’re interested in> also bought [this other product>.”

For example, a basket that contains disposable diapers might also contain baby food, logically enough. Fed by millions of data points in buyer behavior, this approach can provide strong statistical support for intuitive associations, like shelving tissues with the cold and cough medicines. However, this approach can also be thrown off by short-term events (maybe the diapers and baby food were for a visiting relative) or coincidences that don’t actually reveal significant trends. For example, the Harry Potter books are so massively popular on Amazon that they provide hardly any clues about what other titles not about a boy wizard would appeal to that purchaser.

Propensity modeling

In this use case, AI creates value by leveraging propensity modeling to help you avoid those pitfalls and instead target the right offer to the right customer. Jerry Strickland, a Senior Data Specialist with USAA, distilled the various types of propensity modeling into six examples:

• Predicted Customer Lifetime Value (CLV): Instead of looking at the size of individual purchases for big spenders, this metric estimates the total net profit that a given customer will bring to the business for the length of their relationship. It is a prediction of future sales, not a summary of past sales. Predictive modeling evaluates everything you know about the customer, including past sales, demographics, social listening, and other channels, to draw inferences from similarities with the behavior of other customers.
• Predicted Share of Wallet (SOW): Instead of attempting to grow the percentage of sales in a category (market share) by attracting new customers, growing wallet share focuses on increasing the amount an existing customer spends on your brand at the expense of the competition. Predictive modeling looks for growth customers instead of growth markets; that is, it identifies current customers who are spending more with a competitor than with you, introducing opportunities for marginal increases of existing business.
• Propensity to engage: Instead of blasting out emails or social media posts to every customer, match the channel to the customer. AI uses social listening and other techniques to identify the customers most likely to click, reply, or comment so you can maximize your digital marketing efforts.
• Propensity to unsubscribe: Conversely, AI can also identify customers who are less responsive to digital marketing push strategies so you can use other strategies for customers with high CLV and a high propensity to unsubscribe.
• Propensity to buy: You use a different engagement strategy for a customer who is comparison shopping or just looking than for the customer who is ready to buy right now. Propensity modeling uses data from various sources to help you tell the window shoppers from the serious shoppers so you can apply the strategy that encourages a sale without unnecessarily cutting into margins. For example, you might tempt comparison shoppers with a discount, but a customer with a high propensity to buy might be more interested in features and upgrades than discounts.
• Propensity to churn: Some businesses — Internet and TV service providers come to mind — seem prone to maximizing their margins until a customer threatens to cancel, and only then attempt to keep them from jumping ship. Propensity modeling alerts you to customers who are at risk so that you can proactively reach out to them, especially for customers identified as high value by your system.

These predictive marketing approaches date back at least to the mid-twentieth century and the rise of catalog and direct-mail outreach, when punch cards and simple spreadsheets were among the few tools that could crunch the numbers and add statistical support to a merchandising manager’s intuition. The ability to effectively predict and act on buyer behavior came with the arrival of technology that can collect and analyze literally billions of records and transactions, combined with the Internet to facilitate instantaneous communications.

Of course, the information used to predict customer behavior resides in enormous volumes and is encoded in a dizzying variety of data types, from structured data in databases to free-form text in documents, emails, scanned images, and social media feeds. It quickly becomes apparent that machines provide the only efficient method of parsing, understanding, and gaining value from the information. Only machines — which is to say, artificial intelligence — can read at the pace required to acquire and process thousands of documents and articles every second, and then merge, aggregate, and persist the information while analyzing it for content and tone.

The main tools that AI uses to tackle these huge volumes of structured and unstructured data are data mining and text mining. Of course, these techniques can address a wide range of AI business cases besides intelligent recommendations. But first, let me provide a brief overview of these important terms in enterprise AI.

Data and text mining

Machine learning uses information acquired through data mining and text mining to make associations and draw inferences.

Data mining processes structured data such as is found in corporate enterprise resource planning (ERP) systems or customer databases or in an online shopping cart, and it applies modeling functions to produce actionable information.

Text mining uses natural-language processing to extract data elements to populate the structured metadata fields such as author, date, and content summary that enable analysis.

Depending on the characteristics of the domain, the quality of available data and the business goals, recommendation engines apply various data mining techniques such as these:

• Collaborative filtering (CF)
• Content-based filtering (CBF)
• Knowledge-based methods
• Case-based methods

The filtering techniques are the most commonly used techniques for intelligent recommendations.

Collaborative filtering (CF)

Because it relies solely on readily available and easily analyzed transaction-level data to find correlations in consumption patterns, collaborative filtering, shown in Figure 20-1, is one of the more popular recommendation techniques. Several types of collaborative filtering algorithms are available:

• User-based CF measures the similarity between target users and other customers by computing a similarity score for every customer pair, and then offers products that similar customers have chosen in the past. It answers the question “What did customers with similar tastes find interesting?”
• Item-based CF measures the similarity between the items that target customers interact with and other items by computing a similarity score between every product pair. It answers the question “What items are similar to the item this customer finds interesting?”
• Context-aware CF adds another dimension to the user-based and item-based CF in the form of contextual information such as time, location, and social information. It answers the question “What else do I know about this customer that might affect the level of interest in an item?”

Because it is based on comparing shopper behavior, user-based CF is very effective. However, because it must compute a similarity score for every customer pair, it is time- and resource-intensive. A database of N customers generates (N * (N - 1)) / 2 unique customer pairs. For example, 100 customers produce 4,950 unique pairs. If you have 5,000 customers, you would have 12,497,500 unique customer pairs. As you can see, for large user bases, this algorithm is hard to implement without a very strong, parallelizable system.

Item-based CF does not need to compute similarity scores between customers, only between products, and the number of products is likely to be much smaller than the number of customers. As a result, it is less resource-intensive and takes much less time. In addition, if you have a fixed set of products, you can perform the computation once and you’re done.

Context-aware CF enhances a recommendation engine with context, expanding the ability to establish relevance.

Content-based filtering (CBF)

This personalization technique builds a profile of the customer’s interests based on the item metadata in the customer’s buying history and looks for matches with the item metadata for other products. For example, if a customer buys a book on woodworking, the metadata of that purchase could be matched with metadata for related products, such as woodworking tools and protective equipment.

The process involves two main tasks:

• Identify metadata attributes to be associated with each item.
• Build user profiles of items a user has interacted with, giving more weight to metadata attributes that appear more often.

User feedback is critical to fine-tune the profile and subsequent recommendations.

The accuracy of CBF relies heavily on the quality of metadata, and the results are usually less accurate than CF methods. Simpler techniques such as popularity-based methods or market basket analysis generally also lack high predictive power or accuracy.

Cross-validation

Intelligent recommendation systems are typically deployed with the goals of identifying and raising the effectiveness of cross channel/cross product sales campaigns, identifying a customer’s affinity for specific products, reducing customer churn and boosting revenue. So how do you assess how well your recommendations perform?

Cross-validation is one technique used to test the accuracy of the recommendations by calculating the receiver operating characteristic (ROC) curve to produce a metric called “area under the curve” (AUC). This exercise compares predictions (recommendations) to actual outcomes (purchases) to quantify the likelihood that items with the highest ranked recommendation will be purchased.

Historical Fact: The seemingly strange-named Receiver Operating Characteristic curve dates to WWII and was used to measure the effectiveness of a sonar signal to differentiate between a whale and a submarine.

You can customize your intelligent recommendation system to track key performance indicators to optimize performance using these values:

• Click-through rate (CTR): How often does the customer explore the recommended offerings?
• Conversion rate (CR): How often does the customer purchase an item based on the recommended offerings?
• Conversion time: How long does it take to convert a casual shopper into a loyal customer?
• Average order value (AOV): Does the average order value go up?

Other business metrics to consider include ROI and customer lifetime value. However, these metrics depend on several unknowns and thus can be difficult to measure at times.

Data visualization

Data visualization is one of the most important elements of any analytics solutions. This is especially true for predictive analytics software where the results provide actionable insight to improve decision-making. Enterprises can’t afford to leave the interpretation of the results to data scientists; they must be easily digestible by the people who will actually work with them: end-users and business managers.

Data visualization can produce a single view of multi-dimensional data sets to help you with these tasks:

• Assimilating a large amount of information in a glance and focusing on what matters most
• Recognizing the correlation between consumer behavior and metrics such as customer life value and share of wallet
• Identifying trends and connections and preparing strategies in advance
• Improving collaboration between teams

Most predictive analytics solutions provide a range of data visualization capabilities including charts, graphs, reports, and dashboards. The best predictive analytics software will give you easy-to-use, self-service features where users can define their own visualization capabilities to display the results in the way they want.