Catch.com.au is Australia’s largest online marketplace selling more than two million products across food, liquor, clothing, footwear, furniture, sporting goods, electronics and more. The fast growing ecommerce giant wanted to display more relevant search results to its 1.5 million loyal customers, an A/B test was arranged to compare the existing search solution with Sajari's next-generation, machine learning-powered search. Discover how Catch enhanced search relevance and significantly improved sales and profitability using Sajari’s machine learning-powered search.
Australia’s leading online marketplace Catch.com.au wanted to explore how they could display more relevant products through onsite search to lift conversion and deliver more revenue. The online retail giant put their existing solution and Sajari’s search to the test, measuring both solutions in an A/B test across multiple ecommerce metrics to see if their onsite search could generate more revenue.
Catch had been using a tie-breaking based search solution from Algolia. This type of search engine is highly transparent and businesses can easily add business rules to modify the search experience. For Catch this initially worked well, but once hundreds of rules had been written, many in conflict with each other, the limitations of this approach became very clear.
“We had pushed the pre-sorted, tie-breaker algorithm to its limits. There was no upside left.” explained Catch.com.au’s Chief Product Officer Dr. Liron Nehmadi.
Catch wanted to use its own rich business data - like product conversion rate and discount percentages - to influence internal site search rankings and lay the foundation for a better shopper experience.
Catch pitted Sajari against their existing solution in a strict A/B test to determine which search solution could deliver a better conversion rate from Catch.com.au shoppers.
Catch found three major benefits to using Sajari, which included:
Onsite search is a key tool to satisfy ecommerce customers and grow sales revenue. Around 43 percent of all transactions on any given ecommerce site will be influenced by onsite search - most online shoppers head straight to a search box to find what they want to buy. But determining the best products to show for a given query is challenging.
“Search is always a challenge for ecommerce businesses - while shoppers know what they want to buy, the language they use to search for products can be ambiguous and the intent behind the same query varies from person to person.” says Sajari Chief Product Officer Jens Schumacher.
“If someone searches for “apple” it would make sense to assume they are searching for Apple branded products, but often they end up buying non-Apple branded accessories. With pre-sorted ranking everything is in hard preference order, like sorting the columns in an excel spreadsheet, so this kind of ambiguity becomes difficult to deal with. There are many reasons a product should rank well for a particular query, there is no silver bullet and forcing priorities means some products and searches will suffer.”
Sajari gets around this problem by allowing nuanced relationships between queries and product data to more intelligently rank results based on behavioural trends. Using machine learning, Sajari develops an understanding of which product data fields best predict a positive outcome, like a click or a sale, for a specific query. The relevance weighting of that field is then boosted when the query is entered by a user.
Dr. Nehmadi commented: “Managing such a large range of products using pre-sorted ranking meant the business had applied all sorts of rules for specific queries. When someone left the business no one knew why a rule was there, if it mattered and if we could remove it. Sajari has allowed us to use our business intelligence to experiment with different search approaches without having to manage hundreds of specific rules.”
Prior to the A/B test, Catch had a particular problem with their incumbent search solution around the search query “TVs”. Catch stocked smart TVs, TV box sets and TV accessories like wall mounts and antennas. The online retail giant knew that most people typing in the query “TVs' ' were looking for a smart TV, but the current system’s pre-sorted ranking approach meant TV accessories and TV box sets were being displayed ahead of smart TVs.
After Catch implemented Sajari’s machine learning capabilities, the ecommerce team could determine which product categories were more likely to lead to a sale for a particular query - the team could then boost products within the category accordingly using a model created by Sajari’s machine learning systems.
At Sajari, we call this conditional boosting, which is giving a relevance boost to certain products based on conditions such as product metrics, or customer data – or any other ‘condition’ – to display the most relevant result for any given query.
Today when customers search on Catch.com.au, smart TVs outrank other TV-related products, resulting in higher conversions and more sales revenue for these big ticket items.
What’s more, conditional boosting also enabled Catch to display special member products or discounts, based on the searching customer’s Catch.com.au membership status.
“Our members pay us a subscription fee to access discounted products and free delivery, with Sajari we can boost results that are relevant to them so they get the most value from their membership,” explains Dr. Nehmadi.
Experimenting with different relevance approaches was key to unlocking the power of onsite search and higher conversion rates. Sajari allows rapid real-time tests to define the best relevance algorithm for a given circumstance or data point.
“A/B testing new search relevance approaches with pre-sorted, tie-breaker algorithms means running duplicate indexes for each test variant - that just wasn’t feasible.” Dr. Nehmadi said.
“Sajari allows us to immediately run many A/B tests without index duplication. Since we started working with Sajari we’ve learnt a lot about what’s useful in our own data.”
Catch is now running multiple relevance approaches concurrently. Catch.com.au’s users are bucketed into different testing groups with each group shown different results based on different relevance approaches. Some experiments boost discounted products more strongly, others boost products with faster delivery times and some rely more heavily on Sajari’s reinforcement learning.
This means Catch can now measure which relevance strategies are most effective for conversion and session profit. Once a new relevance factor is established, Catch then uses it as a baseline to launch further experiments.
During various A/B test runs over six months, Catch and Sajari were able to increase search conversion or session profitability by more than 10 percent. Sajari also enabled the ecommerce giant to simultaneously combine profit and conversion goals by delivering a six percent increase in conversion rate along with a six percent session profit lift.
Leaving out the cost of no longer manually implementing hundreds of business rules to make search work, Sajari’s search performance directly translated to millions of dollars in incremental revenue for Catch with a return on investment (ROI) exceeding 100x.
“When we set out on this project we knew there was an opportunity to use our investment in data to make our search smarter. With Sajari we’ve definitely delivered. Our results are more relevant to our customers and we’re able to sell more of the products that matter to us, which has resulted in huge improvements to our bottom line.”