Overview and Basics of Elastic Stack (ELK)

Overview

The Elastic Stack, often referred to as the ELK Stack, is a collection of tools designed for search, analytics, logging, monitoring, and data visualization.

It is widely used to process and analyze large volumes of data in near real time. The stack is composed of Elasticsearch as the core engine, along with Kibana, Logstash, Beats, and additional enterprise features provided through X-Pack.

Elasticsearch

Elasticsearch is the central component of the Elastic Stack, originally developed as a scalable implementation of Apache Lucene. Internally, Elasticsearch is built on top of Lucene, where each shard in Elasticsearch represents a single inverted Lucene index.

Although Elasticsearch began as a full-text search engine, it has evolved far beyond Lucene's original scope. Today, it is extensively used not only for full-text search but also for data aggregation, metrics analysis, and performing complex analytics across large datasets.

At its core, Elasticsearch is a server that communicates using JSON-based request and response messages, making it accessible from virtually any programming language. A typical interaction involves sending a JSON request to Elasticsearch and receiving a JSON response containing matching documents, aggregation results, and relevant metadata.

Example JSON Request

The following request searches for documents containing the term "error" in the message field and also performs an aggregation to count logs by level.

POST /logs-index/_search

{
  "query": {
    "match": {
      "message": "error"
    }
  },
  "aggs": {
    "logs_by_level": {
      "terms": {
        "field": "level.keyword"
      }
    }
  }
}

Example JSON Response

Elasticsearch responds with a JSON object that includes metadata, matching documents, and aggregation results.

{
  "took": 12,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "max_score": 1.245,
    "hits": [
      {
        "_index": "logs-index",
        "_id": "abc123",
        "_score": 1.245,
        "_source": {
          "timestamp": "2025-01-01T10:15:30Z",
          "level": "ERROR",
          "message": "Database connection error"
        }
      },
      {
        "_index": "logs-index",
        "_id": "def456",
        "_score": 0.987,
        "_source": {
          "timestamp": "2025-01-01T10:16:10Z",
          "level": "ERROR",
          "message": "Timeout error while calling service"
        }
      }
    ]
  },
  "aggregations": {
    "logs_by_level": {
      "buckets": [
        {
          "key": "ERROR",
          "doc_count": 2
        },
        {
          "key": "INFO",
          "doc_count": 15
        }
      ]
    }
  }
}

In this interaction, the client sends a JSON request to Elasticsearch specifying a search query and an aggregation.

Kibana

Since Elasticsearch functions as a JSON request/response server, users often need a convenient interface to interact with it without building a custom UI. Kibana serves this purpose by providing a web-based user interface for searching, visualizing, and analyzing data stored in Elasticsearch.

Kibana enables users to run complex queries, perform advanced aggregations, and generate graphs and charts with ease.

In addition to general data exploration, Kibana is heavily used for log analysis. One of its most powerful features is the Kibana Dashboard, which allows users to combine multiple visualizations into a single, interactive view for monitoring and gaining insights from data.

As stated earlier, Elasticsearch (ES) follows a JSON-based request/response model. Because of this, interaction with Elasticsearch is not limited to Kibana alone.

Users can communicate with ES directly through REST API calls by sending JSON requests over HTTP, or by using client APIs provided for various programming languages such as Java, Python, and JavaScript. These client APIs internally use REST calls, making it easier for applications to integrate with Elasticsearch programmatically.

Logstash and Beats

Logstash and Beats are components of the Elastic Stack that are responsible for feeding data into Elasticsearch in real time using a streaming approach.

A common example is Filebeat, which is installed on servers where application or system logs are generated. Filebeat continuously monitors log files, parses the logs, converts them into a format that Elasticsearch understands, and sends the data to Elasticsearch as it is generated.

Logstash provides more advanced capabilities for data ingestion and processing. It can receive data from multiple machines, apply filtering, transformation, and enrichment, and then push the processed data into Elasticsearch. A very common architecture involves multiple Filebeats sending data to Logstash, with Logstash ingesting that data into Elasticsearch.

It is important to note that Logstash and Beats are not limited to reading log files or ingesting data only into Elasticsearch. They are generic data ingestion systems and can be used to collect data from various sources such as S3, Kafka, and databases, and then forward that data to different systems, not just Elasticsearch.

X-Pack

X-Pack is a paid extension of the Elastic Stack that provides additional enterprise-level features.

These include security for authentication and authorization, alerting to notify users of important events, monitoring of cluster health, reporting capabilities, machine learning for anomaly detection, and graph exploration for relationship analysis.

X-Pack is commonly used in production environments that require enhanced observability and security.

Elasticsearch Basics

There are two fundamental logical concepts in Elasticsearch, the Document and the Index.

A Document is a JSON object that represents a single unit of data and is identified by a unique ID. Any type of data in JSON format can be stored as a document, not just text. The unique ID can either be explicitly assigned by the user or automatically generated by Elasticsearch.

An Index is a collection of related documents and defines the schema or mapping that describes the structure of those documents. Typically, an index contains documents of a similar type or structure, allowing Elasticsearch to efficiently store and search the data.

When compared to a traditional RDBMS, an Elasticsearch cluster can be thought of as a database, an Index as a table, and a Document as a row in that table. However, despite these conceptual similarities, Elasticsearch works very differently under the hood and is optimized for search and analytics rather than transactional operations.

What Is an Inverted Index?

An inverted index is the core data structure used by almost all search engines. Instead of scanning documents sequentially to find matching terms, an inverted index maintains a mapping between terms and the documents in which they appear.

This structure allows Elasticsearch to retrieve matching documents extremely quickly and is fundamental to its high performance. To understand this, consider three simple documents stored in Elasticsearch.

If Elasticsearch were to search these documents sequentially, it would need to read every document fully each time a search is performed. This approach would be very slow as the number of documents grows.

Instead, Elasticsearch builds an inverted index. In this index, each term is stored along with a list of document IDs in which that term appears. Conceptually, the inverted index would look like this:

Now, when a user searches for the word “fast”, Elasticsearch does not scan all documents. Instead, it directly looks up the term “fast” in the inverted index and instantly retrieves the list of matching documents, which are Document 1 and Document 3.

This structure is the foundation of Elasticsearch’s high performance and makes real-time search and analytics possible at scale.

TF/IDF (Term Frequency/Inverse Document Frequency)

Inverse Document Frequency (IDF) is used to measure how important or unique a term is across all documents in an index. The main idea behind IDF is that rare terms are more meaningful than very common ones.

Words that appear in almost every document, such as “is”, “the”, or “and”, do not help much in distinguishing one document from another, so their importance should be reduced.

To understand this, consider an index with four documents.

In this collection, the term “elasticsearch” appears in three out of four documents, while the term “scalable” appears in only one document.

Because “elasticsearch” appears very frequently across documents, its Inverse Document Frequency value is low. This means the term is considered less useful for distinguishing between documents. On the other hand, the term “scalable” is rare, appearing in only one document, so it receives a high IDF value and is considered much more important.

Now consider a user searching for the query “elasticsearch scalable”. Elasticsearch calculates the relevance score for each document using TF/IDF. The Term Frequency (TF) measures how often each term appears within a document, while IDF measures how rare that term is across all documents.

In this case, Document 2 contains both terms, but the rare term “scalable” contributes much more to the final score than the common term “elasticsearch”.

As a result, TF/IDF helps Elasticsearch rank search results so that Document 2 appears at the top of the search results, followed by other less relevant documents.

In its most basic form, IDF is calculated using the following formula:

IDF(t)=log(N / df(t))

Here, N represents the total number of documents in the index, and df(t) represents the document frequency, which is the number of documents that contain the term t.

By giving more weight to rare and meaningful terms and less weight to very common ones, Elasticsearch is able to return documents in order of their relevance to the search query, which significantly improves search quality.