This article is a step-by-step guide into how you can build an amazing search experience — from spelling correction to NLP.
Adding basic search to an e-commerce store is easy. Building great, measurable search experiences — and iterating high performance product discovery — is very difficult and often requires an entire team of search engineers.
We set out to change that. Our goal was to make a service to not only streamline the construction of great search but also the analytics and data workflows to create an ongoing flywheel of continual improvement. We wanted to put that power into the hands of any developer.
This article is a step-by-step guide into how you can build an amazing search experience — from spelling correction to NLP. We will show you how it’s possible to easily add everything from search basics to the most advanced, Google-like search features for your site.
There are few initial steps involved to get Sajari search off the ground:
We will walk through each of these below.
Schemas are all about performance. If you aren’t convinced we wrote an article on schema vs schemaless just to convince you 😉
You can do this a) manually in our admin console b) via our API or c) via uploading a sample record during your collection setup, or d) using a connector. This example used option b.
Note: If you’re using our website crawler, Shopify, or another connector this will happen automatically.
The configuration of an intelligent search algorithm can be extremely complicated. So, we re-imagined how engineers can build search by creating pipelines. Pipelines break down search configuration into smaller pieces that can be easily mixed, matched, and combined to create an incredibly powerful search experience. Pipelines are highly composable and extendable. You can read a quick pipeline overview here and more specific details related to this example below.
There are two kinds of pipelines to consider: Record pipelines and query pipelines:
Sajari automatically generates initial pipelines for you that you can modify or append later on via our built-in pipeline editor. When you first set up Sajari, the auto-detecting onboarding flow allows you to:
There is much more you can do with pipelines, which is somewhat expanded on below, but it’s worth noting you also don’t need much to get started.
Query pipelines define the query execution and results ranking strategies used when searching the records in your collection.
Steps in a query pipeline can be used for:
Pipelines allow highly complex queries to be automatically constructed by chaining simple step-based logic together.
Query pipelines can also be A/B tested to continuously improve search performance based on business knowledge or machine learning based feedback.
Let’s walk through just a few of the customizations you can add to search using query pipelines. Below are some explanations of step configurations used in this example.
Field weights allow different fields to be worth more when searching. For example the product name is probably a more valuable match than description.
In this case we’ve set the priority order: name, description, brand, ...
Right out of the gate this will start to optimise to what users expect. But it’s really just a starting point.
Spelling is hard! If you’ve tried a fuzzy match in other search engines you would know it’s underwhelming. It also typically slows things down a lot!
The nice thing about Sajari spelling is that it’s not only very very fast, it’s also much smarter than a standard fuzzy matching. You can add spell correction via pipelines using a few lines of YAML.
Below are sample queries showing how missing characters are handled seamlessly.
The nice thing about Sajari spelling is that it also learns over time. As people execute more queries the suggestions and autocomplete begin to better understand the most likely suggestions your customers want.
Next part is looking at business metrics and how they impact ranking. You may want to promote items that sell more frequently, have inventory in stock, have higher user ratings, or anything else you can think of!
Initially boosts like this are defined by the business, but as more people search different options can be A/B tested and also real performance data begins to feedback and improve the ranking (see below).
So far we’ve seen how everything can be globally ranked in an initial sensible order (same logic applies to all queries) based on simple logic.
Now let’s see how more targeted logic can be applied to specific queries.
Want to add Gmail-style filters to your search bar? For power users these are amazing. See below where the in:cheap syntax is used to filter to lower priced items. This example is contrived, but shows how easy this is to do with pipelines:
The logic here is interesting and only requires the simple addition of two pipeline “steps”:
These two steps sequentially perform two functions:
Below illustrates the transform:
We have two ways to do this, a) basic pattern matches and b) more complex models. This example uses the first option to show how some basic capabilities are easily addressed:
The code for this is a little complicated, but not too bad. The first step is looking for a pattern to match, which it then pulls from the query. A subsequent step then uses the extracted information to build a filter.
This is the power of pipelines: you don’t need to actually write a query, the steps sequentially build it for you.
The above case is looking for a price to be under a dollar amount. Which then removes this text from the query and constructs a filter to ensure the price is under the limit. But equally this can be anything else, colours, sizes, dimensions, anything.
Note here that the first step transformed the initial query and the second step conditionally activates the filter when appropriate. This is interesting as the price input could actually be sent in as part of the initial query (i.e. not extracted). This is a common use case for personalisation and recommendations where the gender, size or preferences may be known.
If someone searches for “tv” they probably want a tv and not a tv aerial, or a tv cabinet, or something as “seen on tv.” I say “probably” because search and discovery is all about ambiguity. This alignment of queries to some form of segmentation is very valuable. Below shows an example of a query for “tv” both before and after applying a conditional boost to make this alignment.
The boost is fairly simple. The condition checks to see if the query contains (~) “tv”, if so it then activates and boosts products in the specified category.
That’s interesting and useful, but obviously difficult to do for many variations right? Actually Sajari supports uploading these in bulk in several ways:
The API is useful if this information already exists in the business (ie, if you already have a solid understanding of what queries are generating purchases, we can import this data for you so you can avoid a “cold start” when migrating to Sajari. Contact us for more details).
The analytically generated conditional boosts are far more interesting though, as they can be regenerated automatically using machine learning to dynamically improve your business performance with no effort required.
This example in this demo doesn’t really make use of these as there are no purchases happening! But illustratively you have a couple of options here:
Here’s how it looks in a pipeline.
So the step we used previously to boost the tv category would actually now be basically unnecessary, as the performance would create this relationship for all queries with any detected statistically significant relationships. It’s less specific than a hand written rule, but it’s far more accurate than any one person can determine!
The record pipeline can update and augment information as it is indexed.
Steps can include:
For this example build we’re looking at the last example to do with image analysis for visual search which is explained further below.
You may have noticed the color palette in the facet and filters menu. This was generated using the Google Cloud Vision API, which is powered by advanced AI image analysis. The color information and image descriptions are AI generated and were not in the original data set.
The image analysis was done as each product was loaded. Uniquely this was done by a pipeline step calling out to a cloud function. Why is that important? Because it shows you can augment product processing with anything you like. Note that these features are available even if you’re using a third-party service such as Shopify to host your store.
There are a couple different ways to sync your data with Sajari:
For this demo, we opted for the latter using our Golang API which neatly uses gRPC. We could alternatively use the REST API with another supported client language. The format required is fairly simple as much of the complexity is actually handled in the pipeline definitions.
We have had people in the past comment that the updates were too fast to be consistent updates, but they indeed are! We achieve this via upserts to create and sync records The record wasn’t added to a buffer to merge later; instead, the differential was calculated and executed within the request-response sequence. This makes any changes instantly available as soon as they are synchronized. If you’re using ElasticSearch or one of the Lucene variants you will love this feature.
So far we have ingested and categorized data intelligently, and built smart query pipelines with deep functionality. Now we need to add it to our site and make it shine for a great user experience.
JSON config sets up everything.
Note a few things here:
In the above example the filter is active, but the other facet counts are still displayed, even though the items are not in the result set. This allows the UI to show the user what else is available, even if not selected. This is optional though.
In less time than it took to write this article we’ve shown how to build a blazing fast e-commerce search and discovery interface with autosuggest, sorting, some basic NLP, machine learning powered automatic result improvement, automated categorical segment alignment, AI generated visual image search, spell correction, gmail style filters, conditional business logic and much more!