How to easily add AI to your Auto-Complete
(Suggest) functionality?

In his previous article, my colleague Andreas described the general importance of instant auto-complete suggest functionality in the eCommerce space. If done right – it’s a great tool to lead your visitors to their desired products. This functionality becomes ever more necessary and plays a crucial role in your customers’ overall experience, especially in light of the increase in mobile traffic to e-commerce sites.

What are the Essential requirements for SmartSuggest from a User Perspective?

1. SmartSuggest needs to be fast and data (keywords, products, brands, content, etc.) always up-to-date.
2. SmartSuggest needs to surface relevant results from the first character typed into the search box.
3. SmartSuggest needs to handle typos and misspellings in the same way your search technology does.
4. SmartSuggest needs to understand the intent, location, and context of your search visitors.

Unfortunately, most eCommerce agencies, search vendors, e-com platforms, and in-house search teams struggle to deliver this kind of experience within their Auto-Complete (Suggest) user journeys. If there was ever a doubt: Search is not a trivial task!

Why is Search so Business Relevant? Let’s dig a bit deeper.

A brief look at the most common reasons your current site-search solution needs optimization:

• Most “suggest tools” only handle product data or log files containing search history.
• Suggest Tools fail to consider valuable search KPI data.
• Lack of sophisticated search tracking.
• The most significant amount of time and effort goes into UI developments.
• Most suggest functionalities cannot add short phrases, e.g. “ap,” to familiar brand names, like “apple,” to identify intent better.
• Your current search tool is not performing at its potential.

How can SearchHub help? The solution!

The foundation of our approach lies in collecting the right data within a customer search journey. SearchHub then uses its AI framework to cluster search terms and autonomously choose the most valuable term. Then, once enough data has been gathered to make a proper decision, pick a MasterQuery for relevant clusters. In this way, we ensure your search understands your audience.

SmartSuggest following initial learning phase.

Suggestions with spell correction and trend sorting

SearchHub SmartSuggest runs in combination with whatever site-search technology our customers have in place. To preface, we’re not replacing what our clients have; we make it more clever. For example: whether you already use an open-source search framework (e.g., Elasticsearch, SolR, or OCSS) or still trust in your proprietary search vendor solution. SmartSuggest sits in front of your site-search and makes sure it understands your audience.

Going a step further: by adding our SearchInsights module (specially designed for eCommerce site-search optimization) to the mix, you will be able to influence your SmartSuggest rankings according to keyword trends and/or conversion rates.

👍Without trend influence (out of the box):

👌With trend influence (using SearchInsights):

This data-driven philosophy is so imperative to a good customer experience that we invented a findability score. Findability represents a weighted ratio between positive and negative user signals for a given Search Term. What does this mean? We consider things like exits, bounces, no-clicks, and long search paths to be negative signals. On the other hand, positive signals are things like clicks, rate of clicks on the first page of results, carts, and buys. A bonus: on top of our best-practice KPIs, you have the flexibility to define your own signals.

Query-Flow Graph (SearchInsights Module):

SearchInsights Query Flow Graph

Last but not least: SmartSuggest keeps track of the search redirect and merchandising landing pages your team maintains. Detecting your merchandising & personalization rules means we always direct your visitors to the appropriate curated landing page.

Should I replace my existing Auto-Complete / Suggest and UI?

No, absolutely not. SmartSuggest works in concert with your current search tech stack. One button in the SearchHub UI is all you have to click to fire up your SmartSuggest service. This tech sits on the same data-driven foundation as our MasterQuery picking technology.

Click here to skip our customer case and call us directly to find out more!

Customer who combined technologies

Have a look at STEG Electronics for a masterfully executed example. I’ll walk you through some details here.

STEG already had an elasticsearch stack in place when we introduced Malte Polzin (CEO at STEG Electronics AG ) and his team to SearchHub. However, the prospect of having to implement the complete logic necessary to build a state-of-the-art suggest experience quickly led to the STEG team opting for a hybrid solution. It was clear that combining the data-driven knowledge base of SearchHub with their previously implemented suggest UI would be an innovative and efficient approach. Following a brief search-data-collection-period to gather all relevant search KPIs SmartSuggest was live.

STEG – initial SmartSuggest trend sorting

“SearchHub gives us the flexibility to develop our unique eCommerce search solution based on Elasticsearch with a data-driven approach. The search experience we deliver to our clients is essential for us, and SearchHub supports us with unique expertise in this area. ” Malte Polzin, CEO STEG Electronics AG.

STEG – SmartSuggest Trends even on Typos

Typos and misspellings are handled autonomously by the SearchHub knowledge base. As a result, expensive, time-consuming algorithm operations are not necessary.

Using SmartQuery suggestions allows STEG visitors to browse a list of dynamically-ranked keyword suggestions. The unique way in which the customer interacts with the list of keywords is quite clever. SearchHub’s SmartSuggest displays suggestions based on what’s trending or most valuable at the moment. This data-driven result sorting opens the door facilitates a remarkable user experience! Most notably, mobile device navigation is more usable. Customers select relevant criteria (attributes, tags, variables, etc.) right from SmartSuggest. This is an ingenious answer to the problem of how to handle facetted search on a mobile device. The customer narrows their search to precisely what they’re looking for, all before clicking on the actual “search” button. As a result, customers find what they’re looking for more quickly.

On top of that, STEG saves the cost of unnecessarily querying their site-search engine for terms already known to be of high value.

STEG Filters on Trends

How do I benefit from SearchHub and make auto-complete data-driven?

All you have to do is add your search tracking data (e.g., Google Analytics) to SearchHub or include our SearchCollector in your Tag Manager. Then, within a mere matter of days, depending on the size of your online business, we provide you with SmartSuggest functionality ready to use as a stand-alone or hybrid solution.

 

What else do you get?

• Maintain SmartSuggest within your SearchHub UI. Find out more here.
• Add multiple Suggest labels to keyword clusters.
• Manipulate and test your SmartSuggest ranking strategy with a live preview.
• Merchandise SmartSuggest with inspirational redirect campaigns.

What does your tech team need to know?

SmartSuggest uses enriched keywords from the searchhub clusters database to generate a sophisticated suggest functionality. Designed on Apache Lucene to provide fast, weighted query suggestions. This particular style of deployment is part of the “Open Commerce Search Stack.” If chosen for your implementation, this module automatically connects the searchhub API retrieving the necessary search KPIs and returning detailed analysis data. Combined with performance figures about the module and its usage, you get a turnkey solution to optimize any site search from top to bottom.

We recommend referencing our sample user story to help get you started with SmartSuggest in your system. Additionally, you can reference our SmartSuggest integration material here.

We are happy to provide a demo environment and support you with your e-commerce search strategy, any time!

More than 2 years ago we open-sourced our search-collector, a lightweight javascript SDK that allows you to run search KPI collection from your e-commerce website. This post will illustrate our progress with search event data collection to date. Since launch, our search event collector has gathered close to a billion events, all while maintaining utmost user privacy – the collector SDK does not track any personally identifiable information, uses no fingerprinting or any associated techniques. The sole focus of our collector is rather, to simply record search events and pass them to an endpoint.

Why the search collector and why you need it too

One may argue that Google Analytics provides everything you need. However, once you dive deeper into site search analytics, its deficiencies become apparent.

• Google Analytics runs on sampled data. As a result, it does not depict an accurate picture.
• It’s not possible to implement certain KPIs. For example, product click tracking per keyword.
• Google Analytics often lacks optimum configuration within the web-shop. Fixing it rarely requires available engineering resources.

These types of scenarios led to the birth of the search event collector. As we would rather not impose a particular type of configuration, we structured the collector as an SDK. This strategy gives every team the flexibility to assemble a unique search metric collection solution fit for a particular purpose.

How does search-collector work?

Search-collector has two key concepts

Collectors

Collectors are simple javascript classes that attach to a DOM element, ex. the search box. When an event of interest happens at that DOM element, the collector reacts, packages the relevant data, and passes it on to a Writer (see below).

We’ve provided many out of the box collectors that can help track events like typing, searches, refinements, add to baskets and more.

Writers

Writers receive data from the collectors and deliver it to a storage location. Chaining Writers together will provide separation of concerns (SoC) and prepare them for reuse. For example, we offer BufferingWriter whose only role is to buffer the incoming data for a certain amount of time before sending the package on to the endpoint. This is necessary to prevent an HTTP request from firing upon each keypress within the search box.

Two key writers of interest to the readers of this post are the RestEventWriter and the SQSEventWriter, sending data either to a specified REST endpoint or to Amazon’s SQS. In production, we mostly use the SQS writer, due to its low cost and reliability.

Search-Collector: Progress vs. Room For Improvement

The Progress

• The search-collector has reliably collected close to a billion events on both desktop and mobile.
• We have not encountered any client issues, while the appeal of precise search tracking captures the interest of web-shops and e-commerce owners immediately. The resulting data is easy to digest and manage.
• We package the collector as a single script bundle. This single line adds the search-collector to the web-shop. This streamlined initial setup ensures flexible updates to the event collection setup later.
• The SQS mechanism turned out to be a cheap and reliable option for search event storage.
• The composable Collectors and Writers are flexible enough to capture almost any case we’ve encountered to date.

Room For Improvement

The tight coupling of the collector code to the DOM model within the web-shop sometimes creates issues.

• For example, when DOM structure changes are made without notice. We’re working on a best practice document and a new code version that encourages the use of custom client-side events.
  • For example, soon, we will recommend web-shops send a custom searchEvent when the search is triggered. At the same time, the collector code will register as a page listener for these events.
• Impression tracking on mobile is difficult. Events are fired differently and detection, whether a product was within the visible screen area, does not work consistently across devices. Although impressions are rarely used, we’re working on improving in this area.
• Combining Google Analytics data (web-shops usually have it and use it) with Search-Collector data is not trivial. We’re close to launching our Search Insights product that does just that. This will be a considerable help in the event you need to combine these data sources manually – mind the bot traffic.

Summary – Making Search More Measurable and Profitable

2 years in, we’ve learned much from our Search-Collector SDK project. On the one hand, we are collecting more data with seamless web-shop integration than ever before. This ultimately allows for a broader understanding of things like findability. On the other hand, the more information we gather, the more necessary the maintenance of the collection pipelines. It’s clear, however, that the value we add to our customer’s e-commerce shops far outweighs any limitations we may have encountered.

As a result, we continue on this journey and look forward to the next version of our search-collector. This new version will offer the benefits of streamlined integration, and added transparency into Google Analytics site-search data. All the while, maintaining integration flexibility to ensure continuity of the collected data even after sudden, unforeseen changes to web-shop code.

We’ll be launching soon, so please watch this space.

Footnotes

1. The Document Object Model (DOM) defines the logical structure of documents and the way a document is accessed and manipulated.
2. (SoC) is a design principle for separating a computer program into distinct sections such that each section addresses a separate concern.
3. SQS is a queue from which your services pull data, and it only supports exactly-once delivery of messages.

Largely, it remains, the neglected stepchild of e-commerce optimization. Site-search optimization has the potential to catapult your customer journey strategy to a new level. The success of an E-commerce shop is tightly coupled with the quality of its site search. Customers cannot physically enter the store and look around. Instead, they interact with the shop’s search. Whether via the navigation, if they haven’t a clear idea of what to buy, or via a search query, if they have something specific in mind. Google states that 64% of people in the midst of an, “I want to buy moment”, use search. 71% of these actually visit a retailer’s website. And from all purchases on retailers’ websites, 39% were influenced by a relevant search.

How-To leverage site-search to Optimize your Customer Journey

Of course, search volumes of an online shop do not come close to those of the Google search, but you can learn a lot about visitor search behavior from this analysis. In fact, if you are using Google Analytics, and you haven’t already, you can check it out yourself by navigating to Behavior> Behavior Flow> Site Search> Overview

Site-Search Analytics of a searchHub customer

Site Search Reveals Your Customer Journey

These figures, impressively, show how important a well-functioning search is. What do you think the worst thing is that can happen to a retailer? A customer, who is willing to buy, cannot find what he’s looking for. And this happens, every day, even though the shop has products in the range that match the search. The problem often goes deeper than merely one missed transaction. In fact, not finding what they are looking for can be the very thing that causes him to jump to a competitor and never come back. According to research done by Algolia, 12% of customers will go to a competitor’s site if they are dissatisfied with the search result.

• Do you know how many shop visitors have had bad experiences with your search?
• Do you have a dedicated resource responsible for optimizing your site search?

Chances are… you don’t.

Econsultancy report on Site Search Administration

You may think e-commerce has progressed beyond this statistic, however, around 42% of online shops still neglect site search completely. For another 42%, it’s just a side topic. The bottom line: start focussing on it! It’s easy to get started. Almost every search provider on the market offers built-in analytics, some more, some less. If your solution includes analytics, please use it! Your site-search analytics will help you determine next actions and improve the shopping experience of your customers. They will be thankful and buy from you again the next time. This point is driven home most recently in the book marketing metrics, by Bendle, Ferris, Pfeifer, and Rebstein. In it they speak to the importance of getting the customer journey right:

The probability of selling to an existing customer measures between 60-70%, whereas the probability of selling to a new customer is only 5-20%. Bendle, N. Marketing Metrics – 2016

Humans are creatures of habit. If a shopping experience is positive, meaning: the search quickly found relevant results, the product(s) arrive quickly, and in good quality, there is no reason why you should not purchase again in the future. You see, search, plays an integral role in the complete service a shop offers. Unfortunately, in the majority of cases, the customer journey will start – and sadly sometimes end – with a search!

Optimize Search, Capitalize on Customer Lifetime Value

If you are not completely convinced yet, ask a trusted source to begin optimizing your current conditions. After all, costs for a one-time site-search optimization are considerably lower compared to the expensive customer acquisition marketing campaigns. Not only that, this kind of customer search journey optimization is more sustainable than a marketing campaign.

Bain and Company underpin this with the following figures:

Acquiring new customers is 5-25x more expensive than retaining existing customers.

and

Company profits increase by 75% by increasing customer retention by 5%

Understanding these kinds of business cases has the potential to be quite compelling to a CFO on the fence, about whether to invest in site-search optimization. The main reason companies fail to optimize site-search is due to lack of budget. In 2019 companies claimed, in 42.7% cases, that there was no budget for e-commerce search. Furthermore, in 38.8% of cases, there was no budget for employees to manage the search.

Why Site-Search is Still a Neglected Stepchild

These figures align perfectly with those of the somewhat antiquated Econsultancy report and reveal a fundamental problem: Businesses are yet unaware of the dire significance, and subsequent consequence, of an optimally tuned site-search. As a result, the ultimate impact it has on cost savings and increased profits are blind to them as well!

I’ll end this post with one last quote from Gartner Group

80% of your company’s future revenue will come from just 20% of your existing customers. – Gartner Group

Watch this space to learn more about what we are doing, on a practical level, to make your site-search analytics more profitable and transparent than anything you’ve seen to date. #searchHub #searchCollector

Let’s go out and Make Search great again!

We all know about artificial intelligence (AI). But what many of us don’t know is that AI is now an essential component of e-Commerce, and specifically, site-search. AI impacts the site search performance of your marketplace, and is a key success factor in holding a conversation with your customer — and in converting that conversation to revenue.

How to Tune Your Ecommerce Site-Search – Not Replace it

searchHub.io uses AI to integrate customers into your site’s search, with their unique ways of searching, their needs, and their preferences. Our AI technology predicts and understands what customers want, seamlessly and fast. We do this through intelligent query clustering, and through understanding the complex and unique characteristics of your customers queries to make search more intelligent, and tailored to each individual who visits your marketplace.

Most standard site-search solutions are not optimized for the critical requirements marketplaces must deal with, such as large product catalogs, high site performance and scalability, and fast-changing product availability.

Managing your audiences’ broad search choices

A broader range of products means that visitors have more choice, which attracts more customers, and leads to increased revenue and profitability. This rule applies both to horizontal marketplaces where customers can find a wide range of products (e.g., otto.de), and vertical marketplaces where customers can find more specialized products (e.g., zalando.de).

That’s why many marketplaces develop their own site-search solution based on Solr or Elastic, the most common open-source technologies. But these technologies don’t provide search results that are relevant to all users out-of-the-box. With a wider range of products, your long tail in search increases, and your site-search has to manage a much wider range of problems (long tail is the broad variety of search and keywords customers use when searching for the same thing). This can dramatically affect your search result quality.

Your site search is your business card

Your catalog is the way you introduce your marketplace to your visitors. Search is a key touchpoint in the buying process. It’s the place where the users are telling you what they want. If your search engine speaks the same language as your users, search becomes a conversation, and a conversation is the first step to a relationship.

Therefore, search is the most vital function on your marketplace. Many businesses make high investments in product descriptions and other site content, but then lose their customers due to a poor search experience.

Like any other e-Commerce site, marketplaces need to make sure that the process of product discovery is frictionless: visitors need to find and compare products easily, without wasting their time browsing an infinite catalog.

Most marketplaces offer only limited capabilities of clustering search behavior. This means if a customer types in a more complex search term, there is a good chance of zero results and your customer will leave your marketplace. With searchHub’s AI technology, we automatically cluster millions of user and search queries by meaning and proximity. Deep learning testing and optimization continuously tests search queries for their economic outcome and optimizes the query rewrites accordingly in near-time.

We are hiring

If you’re excited about advancing our search|hub API and strive to enable companies to create meaningful search experiences, join us! We are actively hiring for Data Scientists to work on next-generation API & SEARCH technology.

www.searchhub.io proudly built by www.commerce-experts.com