Blog

One of the biggest causes of website failure is when users simply can’t find stuff on your website. The first law of e-commerce states, “if the user can’t find the product, the user can’t buy the product.”

Why is Measuring and Continuously Improving Site-Search so Tricky?

This sentence seems obvious and sounds straightforward. However, what do “find” and “the product” mean in this context? How can we measure and continuously improve search? It turns out that this task isn’t easy at all.

Current State of Search Relevancy in E-Commerce:

When I talk to customers I, generally, see the following two main methods to measure and define KPIs against the success of search: relevancy, and interaction and conversion.

However, both have flaws in terms of bias and interpretability.

We will begin this series: Three Pillars of Search Relevancy, by developing a better understanding of “Findability”. But first, let’s begin with “Relevancy”.

Search Relevancy:

Determining search result relevance is a massive topic in and of itself. As a result, I’ll only cover this topic with a short, practical summary. In the real world, even in relatively sophisticated teams, I’ve only ever seen mainly three unique approaches to increase search relevancy.

1. Explicit Feedback: Human experts label search results in an ordinal rating. This rating is the basis for some sort of Relevance Metric.
2. Implicit Feedback: Various user activity signals (clicks, carts, …) are the basis for some sort of Relevance Metric.
3. Blended Feedback: The first two types of feedback combine to form the basis for a new sort of Relevance Metric.

 

In theory, these approaches look very promising. And in most cases, they are superior to just looking at Search CR, Search CTR, Search bounce, and Exit rates. However, these methods are heavily biased with suboptimal outcomes.

Explicit Feedback for Search Relevancy Refinement

Let’s begin with Explicit Feedback. There are two main issues with explicit feedback. First: asking people to label search results to determine relevance, oversimplifies the problem at hand. Relevance is, in fact, multidimensional. As a result, it needs to take many factors into account, like user context, user intent, and timing. Moreover, relevance is definitely not a constant. For example, the query “evening dress”, may offer good, valid results for one customer, and yet, the very same list of results can be perceived as irrelevant for another.

Since there is no absolute perception for relevancy, it can’t be used as a reliable or accurate search quality measurement.

Not to mention, it is almost impossible to scale Explicit Feedback. This means only a small proportion of search terms can be measured.

Implicit Feedback for Search Relevancy Refinement

Moving on to Implicit Feedback. Unfortunately, it doesn’t get a lot better. Even if a broad set of user activity signals are used, as a proxy for Search Quality, we still have to deal with many issues. This is because clicks, carts, and buys don’t take the level of user commitment into account.

For example, someone who had an extremely frustrating experience may have made a purchase out of necessity and that conversion would be counted as successful. On the other hand, someone else may have had a wonderful experience and found what he was looking for but didn’t convert because it wasn’t the right time to buy. Perhaps they were on the move, on the bus let’s say. This user’s journey would be counted as an unsuccessful search visit.

But there is more. Since you only receive feedback on what was shown to the user, you will end up at a dead-end. This is not the case, however, if you have some sort of randomization in the search results. This means that all other results for a query, that have yet to be seen, will have a zero probability of contributing to a better result.

Blended Feedback for Search Relevancy Refinement

In the blended scenario, we combine both approaches and try to even out their short-comings. This will definitely lead to more accurate results. It will also help to measure and improve a larger proportion of search terms. Nevertheless, it comes with a lot of complexity and induced bias. This is the logical outcome, as you can only improve results that have been seen by your relevancy judges or customers.

Future State — Introducing Findability as a Metric

I strongly believe that we need to take a different approach to this problem. Then “relevance” alone is not a reliable estimator for User Engagement and even less for GMV contribution.

In my humble opinion, the main problem is that relevance is not a single dimension. What’s more, relevance should instead be embedded in a multidimensional feature space. I came up with the following condensed feature space model, to make interpreting this idea somewhat more intuitive.

Once you have explored the image above, please let it sink in for a while.

Intuitively, findability is a measure of the ease with which information can be found. However, the more accurately you can specify what you are searching for, the easier it might be.

Findability – a Break-Down

I tried to design the Findability feature (measure) to do exactly one thing extremely well. Pointedly, to measure the clarity, effort, and success in the search process. Other important design criteria for the Findability score were that it should:

a) not only provide representative measures for the search quality of the whole website, but also

b) for specific query groups and even single queries to be able to optimize and analyze them.

Findability not only tries to answer, but goes a step further to quantify the question.

Findability as it Relates to Interaction, Clarity, and Effort

• “Did the user find what he was looking for?” — INTERACTION
  

it also tries to answer and quantify the questions

• “Was there a specific context involved when starting the search process?”. “Was the initial search response a perfect starting point for further result exploration?” — CLARITY

and

• “How much effort was involved in the search process?” — EFFORT

 

Appropriately, instead of merely considering whether a result was found and if a product was bought, we also consider whether the searcher had a specific or generic interest. Additionally, things like whether he could easily find what he was looking for, and if the presented ranking of products was optimal, provide valuable information for our findability score.

Intuitively, we would expect that for a specific query, the Findability will be higher if the search process is shorter. In other words, there is less friction to buy. The same applies to generic or informational queries, but with less impact upon Findability.

We do this to ensure seasonal, promotional, and other biasing effects are decoupled from the underlying search system and its respective configuration. Only by trying to decouple these effects, is it possible to continuously optimize your search system in a systematic, continuous, and efficient way respective to our goals to:

• increase the customer experience (to increase conversions and CLTV)
• increase the probability of interaction with the presented items
• increase the success rate of purchase through search

Building the Relevance and Findability Puzzle

To quantify the three different dimensions clarity, effort, and interaction we are going to combine the following signals or features.

Clarity – as it Relates to Findability:

In this context, clarity is used as a proxy for query intent type. In other words, information entropy. For example, in numerous instances, customers issue quite specific queries. For example: “Calvin Klein black women’s evening dress in size 40”. This query describes what they are looking for. For this type of query, the result is pretty clear. However, there is a significant number of examples where customers are either unable or unwilling to formulate such a specific query. On the other hand, the query: “black women’s dress”, leaves many questions open. Which brand, size, price segment? As a result, this type of query is not clear at all. That’s why clarity tries to model the query and deliver specificity.

Features
Query Information Entropy
Result Information Entropy

Effort – as it Relates to Findability:

Effort, on the other hand, attempts to model the exertion, or friction, necessary for the customer to find the information or product for the complete search process. Essentially, every customer interaction throughout the search journey, adds a bit of effort to the overall search process, until he finds what he is looking for. We must try to reduce the effort needed as much as possible, as it relates to clarity, since every additional interaction could potentially lead to a bounce or exit.

Features (Top 5)
Dwell time of the query
Time to first Refinement
Time to first Click
Path Length (Query Reformulation)
Click Positions

Based on these features, it is necessary, in our particular case, to formulate an optimization function that reflects our business goals. Our goal is to maximize the expected search engine result page interaction probability while minimizing the needed path length (effort).

 

The result of our research is the Findability metric (a percentage value between 0-100%), where 0% represents the worst possible search quality and 100% the perfect one. The Findability metric is part of our upcoming search|hub Search Insights Product, which is currently in Beta-Testing.

I’m pretty confident that providing our customers easier to understand and more resilient measures about their site search, will allow them to improve their search experiences in a more effective, efficient, and sustainable way. Therefore, the Findability should provide a solid and objective foundation for your daily and strategic optimization decisions. Simultaneously, it should give you an overview of whether your customers can, efficiently, interact with your product and service offerings.

Framing the Hurdles to Relevant Ecommerce Search

Every e-commerce shop owner wants to achieve ecommerce search relevance with the search results on their website.

But what does that really mean?

As the German singer Herbert Grönemeyer once stated: “It could all be so easy, but it isn’t”.

It could all be so easy, but it isn’t

It may mean that they look for products matching their search intent.

All of them.

New ones on top.

Or the top-sellers.

Or the ones with the best ratings.

Or the ones now available.

Or maybe the cheap ones, where they can save the most money right now?

What is Search Relevance for the Shop Owner

It may even mean something entirely different, like having the products on top with the best margin. Or the old ones, which should free space in the warehouse.

It’s evident that the goals are not the same; sometimes even contradictory.

How to Overcome Hurdles to Ecommerce Search Relevance?

As with most things, the solution is an even blend of several strategies. These will allow both a strong foundation to reach a broad audience, while simultaneously retaining enough focus to meet individual customer intent.

But even with the perfect ranking cocktail, you will still have to do your homework concerning the basic mechanics of finding all relevant products in the first place.

So let’s start with that.

Step #1 — Data Retrieval is Key in Making Search Relevant:

Ask yourself what kind of data you need and if you are making use of all its potential yet.

Don’t forgo the basics!

It’s easiest if you begin this exercise with the following analogy top-of-mind: Imagine you are building a skyscraper!

Data is the Foundation for Ecom Relevance

If the basement is not level, you can try as hard as you want, the construction will fall apart.

Or, to borrow another analogy: painting a wrecked ship in a fancy orange color will still leave you with a ship wreck. So don’t try to use fancy stuff like machine learning to compensate for crappy data.

Achieving ecommerce search relevance is just as much about, wisely, using every available piece of data you have throughout your databases, as it is conceiving a relevant structure to support it.

Keep in mind details like the findability of terms. Having many technical specifications is great. Having them in a normalized matter is even better.

Create Relevant Search Results – Not New Paint on a Rusty Ship!

A simple example of this are colors. The products of brands tend to use fancy names like “space gray” or “midnight green”.

But that is not what your customers will search for. At least not the majority of customers.

As a result, for the purposes of searchability and facetability, it is necessary to map all brand-specific terms to the generally used terms like black and green.

Keep it simple!

Further to normalization: if your customers are searching for sizes in different ways, e.g., 1 TB vs. 1000 GB, you need to make it convenient for customers to find both.

Key to the success of this kind of approach, is structurally separating facet data from search data. All variations must be findable, but only the core values used for faceting.

True, there are several software vendors out there who can help you normalize your product data. However, a few simple processing steps, that you plug into your data processing pipeline, will improve your data enough to considerably increase both findability, and facetability.

Step #2: Data Structuring – for Ecommerce Search Relevance

Assuming you are satisfied with your general data quality, the next important step is, to think about the database structure. This structure will support you and your customers not only to find all related products to a given query, but also to ensure they are returned in the right order. At least more or less the right order, but we’ll get to that later.

Naturally, part of your data structure needs to be weighting the different pieces of information you declare searchable. This means the product name is more important than the technical features. However, features still take precedence over the long description when describing your product.

Actually, an often missed piece of the relevancy puzzle is doing the necessary work to determine which parts of your data structure are essential for relevant intent-based results.

In fact, in many cases, It has proven more lucrative to eliminate long descriptions all together, as they unnaturally bloat your search results. Random hits are most likely not adding value to the overall experience.

As mentioned previously, it’s always a tradeoff between “total-recall” (return everything that could be relevant, and live with additional false results) and precision (return the right stuff, albeit not every item).

What About Stemming to Increase Relevance?

Some search engines allow you to influence the algorithm in detail on a “per field” level.

Stemming is useful on fields with a lot of natural language. But please — don’t use stemming on brand names!

On a similar note, technical features can have units, e.g., “55 inch” or “1.5 kg”. Making this kind of stuff findable can be tricky because people tend to search for it in different ways (1.5 kg vs. 1.5 kg).

For this reason, it’s important to:

1. normalize it in your data and,
2. make sure to do the same steps during query time.

How Best to Structure Multi-Language Product Data Feeds for Optimal Relevance?

If you sell into multiple countries with different languages, set up your indexes to use the correct type of normalization for special characters like Umlauts or different writings for the same character.

Recently, I ran into a case that illustrates this problem quite well, when I noticed people searching for iPhone with characters like í or ì instead of the normal i. Needless to say, it’s imperative these cases are handled correctly. And it’s not as if you have to configure everything on your own. There are ready to configure libraries available for a variety of search engines.

Ecommerce Product ranking

As stated previously, in the introduction, due to the contradictory nature between a user’s intent and the goals of the shop manager, ranking of found items can be tricky.

However, under normal circumstances, you need simply to apply a few basic rules like de-boosting accessory articles, to get the desired results. To achieve this, you must, first, be able to identify what an accessory item is. This means that you, ideally, have a flag you can set in your data. If there is no flag, and you have no way of marking articles in the database, you may get lucky and have a well-maintained category structure. In this case, you can utilize an alternative method and de-boost articles from specific categories instead.

You may also find it helpful to attempt to reconstruct accessory items by identifying “joining words” like “for” (case for smartphone XY, cartridge for printer YZ).

If neither is the case (haha), I strongly suggest you start flagging your items now. Otherwise, it will be much harder to achieve ecommerce search relevance.

The remainder of the ranking rules depend on your audience and your preferences. Be sure you have ample data within your database to pull from! Things like “margin”, or “sold items count”. This will give you flexibility to utilize different approaches and even be able to A/B test them. Please don’t hesitate to add more values to your data, which you deem relevant for scoring your products!

These types of rankings are applied globally, completely query-agnostic.

Tracking and Search Term Boosting

Now, we come to the part, where you let your customers do the work for you.

How, you ask? By making easy use of customer behavior within the shop to enhance the results. To do this, simply take the queries, clicks, add to carts and buying events and combine them at session level.

Why bother with the session? Isn’t it possible to just use the distinct “click path”? Let me take you through an example. Imagine your customer is searching for something, but doesn’t find it because of a typo or different naming in the shop. As a result, he might leave or try to find the right product via the shop’s category navigation. If he finds what he’s looking for, you both get lucky. You now have a link between the former query and the correct product.

This may even result in you learning new synonyms. Nevertheless, be careful. Should your thresholds be too low to filter out random links, you may end up with many false results.

Now that you have a link between queries and products, you can attach the query to the products and use that for boosting at query time.

Keep in mind that boosting is pretty safe, as long as your engine emphasizes precision over recall. You may want to stick to tracking click paths, if you are returning large result sets with blurry matches. For this reason, it’s essential to make sure the query truly belongs to the subsequent actions to not confuse every action within a given session.

These optimizations will already be visible in better results. At least for your most popular products that is. To mitigate a positive feedback loop (popular products get all the attention) ensure new products get a fair chance of being shown. This is simple enough by adding a boost, to new products, for a short time after their release.

But How Do I Achieve Search Relevance for the Rest of my Products?

Let’s expand this one level further and generalize the links we created in the last stage.

If, for example, for the search term “galaxy”, some real phones are being interacted with, we can insinuate, this behavior could also apply for the rest of the products from that category or product type. As mentioned previously, it is imperative that you have clean data as not to mix up stuff like “smartphones & accessories”. Good luck, if you’re using this type of key to generalize your tracking links! Don’t do it — clean your data first!

In the example at hand, we want to achieve a link between the query and all products of the type “smartphone”. Subsequently, we can add a boosting for all the smartphones found and voilà…

You get a result with smartphones on top. The most relevant ones getting an extra punch from the direct query relation.

And finally, the relevancy of the products is a stack of boostings:

1. First by the field weight
2. Then by ranking criteria
3. And in the end by the tracking events.

If you got this far, you might also be interested in the more advanced techniques like “learning to rank”.

This method applies the principles of machine learning to the product ranking mechanism. However, it will require some supervision to, successfully, learn the right things.

Or perhaps you want to integrate personalization for individual visitors. Wait a minute… maybe that topic is so comprehensive, it would be better left for another blog post…

So, now we’re done, right?

Well, not so fast 😉

Query Preprocessing for Ecommerce Search Relevance

The whole data part is only one side of the coin. Your customers may still need some help finding what they look for.

For this reason, you should implement some preprocessing of the incoming queries before forwarding them to your search engine.

Preprocessing can be as simple as creating tasks to remove so-called stop words, i.e., filler-words words like a, the, at, also, etc.

Remove Obstacles to Relevant Search

If your engine does not come with a list of stop words, you can search the internet and adapt a list to meet your needs. In addition, counting the words from your data and checking which word most frequently qualify as a stop word for you can be very effective.

Some search engines even allow reducing the value of those words to a bare minimum. This method can help you to better rank the one product where, actually, the whole phrase matches (e.g., “live at Wembley” instead of “live … Wembley”).

We also mustn’t forget the need to support your customers should their language differ from the one used describing your products. For this reason, you need to establish a set of synonyms for the cases where you would, otherwise, end up with no match results.

Please keep in mind, if your search engine also provides a way to define antonyms for similar words with diverging meaning, e.g., “bad” and “bat”, make sure you fully understand how this cleans/shapes the results. In some cases, products containing both words will be kicked out of the result for triggering antonyms on both sides of the spectrum.

If you’re able to, use deboosting for the antonym instead of completely removing it. It can save your day!

And finally, your customers might misspell words like I just did… did you notice? Well, your search engine will notice.

Or, what about the scenario when your search just won’t find the right things? Or anything at all, for that matter. Or, maybe the result varies negatively because the boosting for one frequent query is better than the ranking for an alternative spelling.

In this case, you could add a preprocessing rule. And for some frequently used queries, it might work out.

But eventually, you will get lost in the long tail of queries, completely. Tools like our searchHub can help you in matching all variations of a query to the perfect master query. This master query is then sent to your search engine —whatever flavor search engine you might have.

searchHub identifies any types of misspellings, or even ambiguous albeit, correct, spellings (playstation 5, ps 5, PS5, play-station 5) or typos (plystation 5, plyastation 5, etc.).

We know which query performs best so that you don’t have to!

If you want to see your shop’s queries clustered around top-performing master queries, and get to know the full potential this has on your conversions, please feel free to contact us!

You may think the word “dumb services” in connection with software orchestration in the title is clickbait. However, I can assure you, the aforementioned “Orchestration with Dumb Services”, is a real and simple software orchestration concept certainly to improve your sleep.

Any engineer, DevOps, or software architect can relate to the stress of running a loaded production system. To do so well, it’s necessary to automate, provision, monitor, provide redundancy and fail-over to hit those SLAs. The following paragraphs cut to the chase. You won’t see any fancy buzzwords. I aim to help avoid pitfalls into which many companies stumble, when untangling monolithic software projects. Or, for that matter, even when building small projects from scratch. While the concept is not applicable for every use case, it does fit perfectly into the world of e-commerce search. It’s even applicable for full-text search. Generally, wherever the search index read and writes are separate pipelines, this is for you! So, what are we waiting for? Let’s start orchestrating with dumb services.

What is the Difference Between Regular vs. Dumb services

To begin, let’s define the term “service”

A service is a piece of software that performs a distinct action.

Nowadays, a container running as part of a kubernetes cluster is a good example of a service. This container can spin-up multiple instances of the service to meet demand. The configuration of a so-called regular service points it to other services it may need. These could be things like connections to databases, and so on.

Regular Services in action are seen illustrated in the diagram to the right. As they grow, companies run many such hierarchically organized services.

Regular Service Hierarchy

Dumb Services

Now, let’s clarify what “dumb service” means. In this context, a dumb service is a service which knows nothing about its environment. Its configuration is reduced to performance related aspects (ex. memory limits). When you start such a service, it does nothing — no connection to other services, no joining of clusters, just waits to be told what to do.

Orchestrator Services

To create a full system composed of dumb services, you deploy another service type called an “orchestrator”. The orchestrator is the “brain”, the dumb services are the “muscle” — the brain tells the muscles what to do.

The orchestrator sends tasks to each service. Additionally, it directs the data exchange between services. Finally, it pushes data and configurations to the client facing services. Furthermore, the orchestrator initiates all service state changes.

Dumb Service Orchestration

Let’s review our “regular vs. dumb” services in light of two key aspects of a software system — fault tolerance and scalability.

Fault Tolerance

Fault Tolerance with Regular Services

In the regular case diagram we illustrate a typical flow during a user request. The client facing services at level 1 (labeled with L1 in the diagram) need to call the internal services at levels 2 and 3 to complete the request. Naturally, in a larger system, this call hierarchy goes much deeper. To meet the SLA, all services must be up all time as any incoming request could call a service further down the hierarchy. This is obviously a hard task, combining N services with uptime of 99.95% does not result in 99.95% for the entire system — in the worst case, for a request that hits 5 services you’d go down to 99.7% (99.95 to the power of N).

Fault Tolerance with dumb services.

Let’s compare this to the system composed of dumb services. The client facing services on level 1, serve the request without any dependency to the services at level 2 and 3. We only need to ensure the SLA of the L1 services to ensure the SLA of the entire client facing part of the system — services at levels 2 and 3 could go down without affecting user requests.

Scaleability

Scaling Regular Services

Scaling the system composed of regular services, necessarily means scaling the entire system. If only one layer is scaled, it could result in overloading the lower layers of the system as user requests increase. The process of scaling also means more automation as you need to correctly wire the services together to scale.

Scaling Dumb Services Architecture

Let’s take a look again at our dumb services architecture. Each service can be scaled independently as it has no direct dependencies on any other services. You can spin up as many client facing services as you like to meet increased user requests without scaling any of the internal systems. And vice versa, you can increase the number of nodes for internal services on demand to meet a heavy indexing task and then easily spin it down. Again, all this without affecting user requests.

What about Testing?

Finally, testing your service is simple: you start it and pass a task to it — no dependencies you need to consider.

Wrapping it up

In conclusion, you can simplify your architecture significantly by deploying this simple concept. However, as mentioned previously, this does not apply to all use cases. Situations, where your client facing nodes are part of both the reading and writing data pipelines, are harder to organize in this way. Even still, any time you’re faced with designing a system composed of multiple services, think about these patterns — it may save you a few sleepless nights.

Many businesses run an Elasticsearch/Kibana stack. Some use a SaaS-Service for Elastic — i.e., the AWS Amazon Elasticsearch Service; the Elastic in Azure Service from Microsoft; or the Elastic Cloud from Elastic itself. More commonly, Elasticsearch is hosted in a proprietary environment. Elastic and the community provide several deployment types and tips for various platforms and frameworks. In this article, I will show how to deploy Elasticsearch and Kibana in a Kubernetes Cluster using the Elastic Kubernetes Operator (cloud-on-k8s) without using Helm (helm / helm-charts). Overview of Elastic Deployment Types and Configuration:

• Ansible (ansible-elasticsearch)
• Docker (deprecated) (elasticsearch-docker)
• Kubernetes (cloud-on-k8s / helm-charts)
• Puppet (puppet-elasticsearch)
• Chef (cookbook-elasticsearch)
• Hadoop (elasticsearch-hadoop)
• Windows (elastic-stack-installers)

The Motivation for Using the Elasticsearch-Operator:

What might be the motivation for using the Elasticsearch-Operator instead of using any other SaaS-Service?

The first argument is, possibly, the cost. The Elastic Cloud is round about 34% pricier than hosting your own Elasticsearch on the same instance in AWS. Furthermore, the AWS Amazon Elasticsearch Service is even 50% more expensive than the self-hosted version.

Another argument could be that you already have a Kubernernetes-Cluster running with the application which you would like to use Elasticsearch with. For this reason, you want to avoid spreading one application over multiple environments. So, you are looking to use Kubernetes as your go-to standard.

Occasionally, you may also have to build a special solution with many customizations that are not readily deployable with a SaaS provider.

An important argument for us was the hands-on experience hosting Elasticsearch, to give the best support to our customers.

Cluster Target Definition:

For the purposes of this post, I will use a sample cluster running on AWS. Remember to always include the following features:

• 6 node clusters (3 es-master, 3 es-data)
• master and data nodes are spread over 3 availability zones
• a plugin installed to snapshot data on S3
• dedicated nodes where only elastic services are running on
• affinities that not two elastic nodes from the same type are running on the same machine

 

Due to this article’s focus on how to use the Kubernetes Operator, we will not provide any details regarding necessary instances, the reason for creating different instance groups, or the reasons behind several pod anti affinities.

In our Kubernetes cluster, we have two additional Instance Groups for Elasticsearch: es-master and es-data where the nodes have special taints.

(In our example case, the instance groups are managed by kops. However, you can simply add the labels and taints to each node manually.)

The Following is an example of how a node of the es-master instance group looks like:

				
					apiVersion: v1
kind: Node
metadata:
  ...
  labels:
    failure-domain.beta.kubernetes.io/zone: eu-north-1a
    kops.k8s.io/instancegroup: es-master
    kubernetes.io/hostname: ip-host.region.compute.internal
    ...
spec:
  ...
  taints:
  - effect: NoSchedule
    key: es-node
    value: master
				
			

As you may have noticed, there are three different labels:

1. The failure-domain.beta.kubernetes.io/zone contains the information pertaining to the availability zone in which the instance is running.
2. The kops.k8s.io/instancegroup contains the information in which instance the group resides. This will be important later to allow both master and data nodes to run on different hardware for performance optimization.
3. The kubernetes.io/hostname acts as a constraint to ensure only one master node is running the specified instance.

 

Following is an example of an es-data instance with the appropriate label keys, and respective values:

				
					apiVersion: v1
kind: Node
metadata:
  ...
  labels:
    failure-domain.beta.kubernetes.io/zone: eu-north-1a
    kops.k8s.io/instancegroup: es-data
    kubernetes.io/hostname: ip-host.region.compute.internal
    ...
spec:
  ...
  taints:
  - effect: NoSchedule
    key: es-node
    value: data
				
			

As you can see, the value of the es-node taint and the kops.k8s.io/instancegroup label differs. We will reference these values later to decide between data and master instances.

Now that we have illustrated our node structure, and you are better able to grasp our understanding of the Kubernetes and Elasticsearch cluster, we can begin installation of the Elasticsearch operator in Kubernetes.

Let’s Get Started:

First: install the Kubernetes Custom Resource Definitions, RBAC rules (if RBAC is activated in the cluster in question), and a StatefulSet for the elastic-operator pod. In our example case, we have RBAC activated and can make use of the all-in-one deployment file from Elastic for installation.

(Notice: If RBAC is not activated in your cluster, then remove line 2555 – 2791 and all service-account references in the file):

				
					kubectl apply -f https://download.elastic.co/downloads/eck/1.2.1/all-in-one.yaml
				
			

This creates four main parts in our Kubernetes cluster to operate Elasticsearch:

• All necessary Custom Resource Definitions
• All RBAC Permissions which are needed
• A Namespace for the Operator (elastic-system)
• A StatefulSet for the Elastic Operator-Pod

Now perform kubectl logs -f on the operator’s pod and wait until the operator has successfully booted to verify the Installation. Respond to any errors, should an error message appear.

				
					kubectl -n elastic-system logs -f statefulset.apps/elastic-operator
				
			

Once confirmed that the operator is up and running we can begin with our Elasticsearch cluster. We begin by creating an Elasticsearch resource with the following main structure (see here for full details):

				
					apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: blogpost # name of the elasticsearch cluster
  namespace: blog
spec:
  version: 7.7.0 # elasticsearch version to deploy
  nodeSets: # nodes of the cluster
  - name: master-zone-a
    count: 1 # count how many nodes should be deployed
    config: # specific configuration for this node type
      node.master: true
    ...
  - name: master-zone-b
  - name: master-zone-c
  - name: data-zone-a
  - name: data-zone-b
  - name: data-zone-c
				
			

In the listing above, you see how easily the name of the Elasticsearch cluster, as well as, the Elasticsearch version and different nodes that make up the cluster can be set. Our Elasticsearch structure is clearly specified in the array nodeSets, which we defined earlier. As a next step, we want to take a more in-depth look into a single nodeSet entry and see how this must look to adhere to our requirements:

				
					- name: master-zone-a
    count: 1
    config:
      node.master: true
      node.data: false
      node.ingest: false
      node.attr.zone: eu-north-1a
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      metadata:
        labels:
          component: elasticsearch
          role: es-master
      spec:
        volumes:
          - name: elasticsearch-data
            emptyDir: {}
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms: 
              - matchExpressions: # Kniff mit Liste
                - key: kops.k8s.io/instancegroup
                  operator: In
                  values:
                  - es-master
                - key: failure-domain.beta.kubernetes.io/zone
                  operator: In
                  values:
                  - eu-north-1a
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                  - key: role
                    operator: In
                    values:
                    - es-master
                topologyKey: kubernetes.io/hostname
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        - name: install-plugins
          command:
          - sh
          - -c
          - |
            bin/elasticsearch-plugin install -b repository-s3
        tolerations:
        - key: "es-node"
          operator: "Equal"
          value: "master"
          effect: "NoSchedule"
        containers:
        - name: elasticsearch
          resources:
            requests:
              memory: 1024Mi
            limits:
              memory: 1024Mi
				
			

The count key specifies, for example, how many pods Elasticsearch nodes should create with this node configuration for the cluster. The config object represents the untyped YAML configuration of Elasticsearch (Elasticsearch settings). The podTemplate contains a normal Kubernetes Pod template definition. Notice that here we are controlling the affinity and tolerations of our es-node to a special instance group and all pod affinities. In the initContainers section, we are handling kernel configurations and also the Elasticsearch repository-s3 plugin installation. One note on the nodeSelectorTerms: if you want to use the logical and condition instead of, or, you must place the conditions in a single matchExpressions array and not as two individual matchExpressions. For me, this was not clearly described in the Kubernetes documentation.

Once we have created our Elasticsearch deployment, we must create a Kibana deployment. This can be done with the Kibana resource. The following is a sample of this definition:

				
					apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: blogpost
  namespace: blog
spec:
  version: 7.7.0
  count: 1
  elasticsearchRef:
    name: blogpost
  podTemplate:
    metadata:
      labels:
        component: kibana
				
			

Notice that the elasticsearchRef object must refer to our Elasticsearch to be connected with it.

After we have created all necessary deployment files, we can begin deploying them. In our case, I put them in one big file called elasticseach-blog-example.yaml, you can find a complete list of the deployment files at the end of this blogpost.

				
					kubectl apply -f elasticsearch-blog-example.yaml
				
			

After deploying the deployment file you should have a new namespace with the following pods, services and secrets (Of course with more resources, however this is not relevant for our initial overview):

				
					(⎈ |blog.k8s.local:blog)➜  ~ kubectl get pods,services,secrets 
NAME                              READY  STATUS   RESTARTS  AGE
pod/blogpost-es-data-zone-a-0     1/1    Running  0         2m
pod/blogpost-es-data-zone-b-0     1/1    Running  0         2m
pod/blogpost-es-data-zone-c-0     1/1    Running  0         2m
pod/blogpost-es-master-zone-a-0   1/1    Running  0         2m
pod/blogpost-es-master-zone-b-0   1/1    Running  0         2m
pod/blogpost-es-master-zone-c-0   1/1    Running  0         2m
pod/blogpost-kb-66d5cb8b65-j4vl4  1/1    Running  0         2m
NAME                               TYPE       CLUSTER-IP     PORT(S)   AGE
service/blogpost-es-data-zone-a    ClusterIP  None           <none>    2m
service/blogpost-es-data-zone-b    ClusterIP  None           <none>    2m
service/blogpost-es-data-zone-c    ClusterIP  None           <none>    2m
service/blogpost-es-http           ClusterIP  100.68.76.86   9200/TCP  2m
service/blogpost-es-master-zone-a  ClusterIP  None           <none>    2m
service/blogpost-es-master-zone-b  ClusterIP  None           <none>    2m
service/blogpost-es-master-zone-c  ClusterIP  None           <none>    2m
service/blogpost-es-transport      ClusterIP  None           9300/TCP  2m
service/blogpost-kb-http           ClusterIP  100.67.39.183  5601/TCP  2m
NAME                                        DATA  AGE
secret/default-token-thnvr                  3     2m
secret/blogpost-es-data-zone-a-es-config    1     2m
secret/blogpost-es-data-zone-b-es-config    1     2m
secret/blogpost-es-elastic-user             1     2m
secret/blogpost-es-http-ca-internal         2     2m
secret/blogpost-es-http-certs-internal      3     2m
secret/blogpost-es-http-certs-public        2     2m
secret/blogpost-es-internal-users           2     2m
secret/blogpost-es-master-zone-a-es-config  1     2m
secret/blogpost-es-master-zone-b-es-config  1     2m
secret/blogpost-es-master-zone-c-es-config  1     2m
secret/blogpost-es-remote-ca                1     2m
secret/blogpost-es-transport-ca-internal    2     2m
secret/blogpost-es-transport-certificates   11    2m
secret/blogpost-es-transport-certs-public   1     2m
secret/blogpost-es-xpack-file-realm         3     2m
secret/blogpost-kb-config                   2     2m
secret/blogpost-kb-es-ca                    2     2m
secret/blogpost-kb-http-ca-internal         2     2m
secret/blogpost-kb-http-certs-internal      3     2m
secret/blogpost-kb-http-certs-public        2     2m
secret/blogpost-kibana-user                 1     2mm
				
			

As you may have noticed, I removed the column EXTERNAL from the services and the column TYPE from the secrets. I did this due to the formatting in the code block.

Once Elasticsearch and Kibana have been deployed we must test the setup by making an HTTP get request with the Kibana-Dev-Tools. First, we have to get the elastic user and password which the elasticsearch-operator generated for us. It’s saved in the Kubernetes Secret -es-elastic-user in our case blogpost-es-elastic-user.

				
					(⎈ |blog.k8s.local:blog)➜  ~ kubectl get secret/blogpost-es-elastic-user -o yaml 
apiVersion: v1
data:
  elastic: aW8zQWhuYWUyaWVXOEVpM2FlWmFoc2hp
kind: Secret
metadata:
  creationTimestamp: "2020-10-21T08:36:35Z"
  labels:
    common.k8s.elastic.co/type: elasticsearch
    eck.k8s.elastic.co/credentials: "true"
    elasticsearch.k8s.elastic.co/cluster-name: blogpost
  name: blogpost-es-elastic-user
  namespace: blog
  ownerReferences:
  - apiVersion: elasticsearch.k8s.elastic.co/v1
    blockOwnerDeletion: true
    controller: true
    kind: Elasticsearch
    name: blogpost
    uid: 7f236c45-a63e-11ea-818d-0e482d3cc584
  resourceVersion: "701864"
  selfLink: /api/v1/namespaces/blog/secrets/blogpost-es-elastic-user
  uid: 802ba8e6-a63e-11ea-818d-0e482d3cc584
type: Opaque
				
			

The user of our cluster is the key, located under data. In our case, elastic. The password is the corresponding value of this key. It’s Base64 encoded, so we have to decode it:

				
					(⎈ |blog.k8s.local:blog)➜  ~ echo -n "aW8zQWhuYWUyaWVXOEVpM2FlWmFoc2hp" | base64 -d
io3Ahnae2ieW8Ei3aeZahshi
				
			

Once we have the password we can port-forward the blogpost-kb-http service on port 5601 (Standard Kibana Port) to our localhost and access it with our web-browser at https://localhost:5601:

				
					(⎈ |blog.k8s.local:blog)➜  ~ kubectl port-forward service/blogpost-kb-http 5601      
Forwarding from 127.0.0.1:5601 -> 5601
Forwarding from [::1]:5601 -> 5601
				
			

Elasticsearch Kibana Login Screen

After logging in, navigate on the left side to the Kibana Dev Tools. Now perform a GET / request, like in the picture below:

Getting started with your Elasticsearch Deployment inside the Kibana Dev Tools

Summary

We now have an overview of all officially supported methods of installing/operating Elasticsearch. Additionally, we successfully set up a cluster which met the following requirements:

• 6 node clusters (3 es-master, 3 es-data)
• we spread master and data nodes over 3 availability zones
• installed a plugin to snapshot data on S3
• has dedicated nodes in which only elastic services are running
• upholds the constraints that no two elastic nodes of the same type are running on the same machine

 

Thanks for reading!

Full List of Deployment Files

				
					apiVersion: v1
kind: Namespace
metadata:
  name: blog
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: blogpost
  namespace: blog
spec:
  version: 7.7.0
  nodeSets:
  - name: master-zone-a
    count: 1
    config:
      node.master: true
      node.data: false
      node.ingest: false
      node.attr.zone: eu-north-1a
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      metadata:
        labels:
          component: elasticsearch
          role: es-master
      spec:
        volumes:
          - name: elasticsearch-data
            emptyDir: {}
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms: 
              - matchExpressions:
                - key: kops.k8s.io/instancegroup
                  operator: In
                  values:
                  - es-master
                - key: failure-domain.beta.kubernetes.io/zone
                  operator: In
                  values:
                  - eu-north-1a
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                  - key: role
                    operator: In
                    values:
                    - es-master
                topologyKey: kubernetes.io/hostname
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        - name: install-plugins
          command:
          - sh
          - -c
          - |
            bin/elasticsearch-plugin install -b repository-s3
        tolerations:
        - key: "es-node"
          operator: "Equal"
          value: "master"
          effect: "NoSchedule"
        containers:
        - name: elasticsearch
  - name: master-zone-b
    count: 1
    config:
      node.master: true
      node.data: false
      node.ingest: false
      node.attr.zone: eu-north-1b
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      metadata:
        labels:
          component: elasticsearch
          role: es-master
      spec:
        volumes:
          - name: elasticsearch-data
            emptyDir: {}
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: kops.k8s.io/instancegroup
                  operator: In
                  values:
                  - es-master
                - key: failure-domain.beta.kubernetes.io/zone
                  operator: In
                  values:
                  - eu-north-1b
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        - name: install-plugins
          command:
          - sh
          - -c
          - |
            bin/elasticsearch-plugin install -b repository-s3
        tolerations:
        - key: "es-node"
          operator: "Equal"
          value: "master"
          effect: "NoSchedule"
        containers:
        - name: elasticsearch
  - name: master-zone-c
    count: 1
    config:
      node.master: true
      node.data: false
      node.ingest: false
      node.attr.zone: eu-north-1c
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      metadata:
        labels:
          component: elasticsearch
          role: es-master
      spec:
        volumes:
          - name: elasticsearch-data
            emptyDir: {}
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: kops.k8s.io/instancegroup
                  operator: In
                  values:
                  - es-master
                - key: failure-domain.beta.kubernetes.io/zone
                  operator: In
                  values:
                  - eu-north-1c
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        - name: install-plugins
          command:
          - sh
          - -c
          - |
            bin/elasticsearch-plugin install -b repository-s3
        tolerations:
        - key: "es-node"
          operator: "Equal"
          value: "master"
          effect: "NoSchedule"
        containers:
        - name: elasticsearch       
  - name: data-zone-a
    count: 1
    config:
      node.master: false
      node.data: true
      node.ingest: true
      node.attr.zone: eu-north-1a
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      metadata:
        labels:
          component: elasticsearch
          role: es-worker 
      spec:
        volumes:
          - name: elasticsearch-data
            emptyDir: {}
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: kops.k8s.io/instancegroup
                  operator: In
                  values:
                  - es-data
                - key: failure-domain.beta.kubernetes.io/zone
                  operator: In
                  values:
                  - eu-north-1a
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        - name: install-plugins
          command:
            - sh
            - -c
            - |
              bin/elasticsearch-plugin install -b repository-s3
        tolerations:
        - key: "es-node"
          operator: "Equal"
          value: "data"
          effect: "NoSchedule"
        containers:
        - name: elasticsearch
  - name: data-zone-b
    count: 1
    config:
      node.master: false
      node.data: true
      node.ingest: true
      node.attr.zone: eu-north-1b
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      metadata:
        labels:
          component: elasticsearch
          role: es-worker 
      spec:
        volumes:
          - name: elasticsearch-data
            emptyDir: {}
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: kops.k8s.io/instancegroup
                  operator: In
                  values:
                  - es-data
                - key: failure-domain.beta.kubernetes.io/zone
                  operator: In
                  values:
                  - eu-north-1b
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        - name: install-plugins
          command:
            - sh
            - -c
            - |
              bin/elasticsearch-plugin install -b repository-s3
        tolerations:
        - key: "es-node"
          operator: "Equal"
          value: "data"
          effect: "NoSchedule"
        containers:
        - name: elasticsearch
  - name: data-zone-c
    count: 1
    config:
      node.master: false
      node.data: true
      node.ingest: true
      node.attr.zone: eu-north-1c
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      metadata:
        labels:
          component: elasticsearch
          role: es-worker 
      spec:
        volumes:
          - name: elasticsearch-data
            emptyDir: {}
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: kops.k8s.io/instancegroup
                  operator: In
                  values:
                  - es-data
                - key: failure-domain.beta.kubernetes.io/zone
                  operator: In
                  values:
                  - eu-north-1c
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        - name: install-plugins
          command:
            - sh
            - -c
            - |
              bin/elasticsearch-plugin install -b repository-s3
        tolerations:
        - key: "es-node"
          operator: "Equal"
          value: "data"
          effect: "NoSchedule"
        containers:
        - name: elasticsearch
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: ui
  namespace: ui
spec:
  version: 7.7.0
  count: 1
  elasticsearchRef:
    name: ui
  podTemplate:
    metadata:
      labels:
        component: kibana