3 min to read

ELK Stack Implementation Guide

Learn how to implement ELK Stack (Elasticsearch, Logstash, Kibana) for log management and analysis.

Featured image


Overview

This guide explores the implementation of ELK Stack (Elasticsearch, Logstash, Kibana) for log collection, processing, and analysis.

What is ELK Stack?

ELK Stack combines three main components: Elasticsearch, Logstash, and Kibana. Recently, EFK Stack (with Fluentd) has also become popular as a lighter alternative.


Architecture(ELK Stack)

Log Pipeline:

Log Pipeline

Metric Pipeline:

Metric Pipeline

APM Pipeline:

APM Pipeline

EFK Stack Log Pipeline:

EFK Stack Log Pipeline

🧩 Key Components

Elasticsearch

Logstash

Kibana

Filebeat


Implementation Guide

Installation Overview

We used the Helm Chart that was archived on May 16, 2023, and installed it in Single Mode for our internal development server.

Installation Methods

1️⃣ Using Archived Helm Charts

Reference repository

Implementation reference

2️⃣ Using Latest Version (ECK Operator)

For the latest version, use the ECK operator:

elk_login


Installation Order

Install components in the following order:

  1. Elasticsearch
  2. Kibana
  3. Logstash
  4. Filebeat

Configuration Examples

filebeat-values.yaml

filebeatConfig:
  filebeat.yml: |
    filebeat.inputs:
    - type: log
      paths:
        - /usr/share/filebeat/app/somaz/dev/app/logs/*  # Mounted log path
        - /usr/share/filebeat/app/somaz/dev/app/logs/**/*
      fields:
        log_source: "dev-somaz-app"  # This becomes Elasticsearch index name
        environment: "dev"
        app: "somaz"
        component: "app"
      fields_under_root: true
      json.keys_under_root: true      # Promote JSON fields to root level
      json.add_error_key: true        # Add error field on JSON parsing failure
      json.expand_keys: true          # Expand nested JSON strings
      processors:
        - decode_json_fields:
            fields: ["data"]
            process_array: true
            max_depth: 2
            target: ""
            overwrite_keys: true
        - script:
            lang: javascript
            source: |
              function process(event) {
                // JSON processing logic
                return event;
              }

logstash-values.yaml

logstashPipeline:
  logstash.conf: |
    input {
      beats {
        port => 5044
      }
    }
    
    filter {
      if [log_source] {
        mutate {
          add_field => { "index_name" => "%{log_source}" }
        }
      }
      # Additional processing configurations...
    }

    output {
      elasticsearch {
        hosts => ["https://elasticsearch-master:9200"]
        user => "${ELASTICSEARCH_USERNAME}"
        password => "${ELASTICSEARCH_PASSWORD}"
        ssl_certificate_verification => true
        cacert => '/usr/share/logstash/config/certs/ca.crt'
        index => "%{index_name}"
      }
    }

Verification

After setup, verify the installation:

# Check Elasticsearch indices
curl -k -u "elastic:password" "http://elasticsearch.somaz.link/_cat/indices?v"

# Sample output:
health status index                          uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .kibana-event-log-8.5.1-000002 0YwfADLHQryK5bYlBpBm5Q   1   1          0            0       225b           225b
yellow open   dev-somaz-app                  9zip7n0sdfasdfsadfsdfsdaf 1   1      24161            0      6.5mb          6.5mb

And if Kibana creates a discover that fits the index, it can be checked as follows. elk_discover


⚠️ Important Considerations

  1. Elasticsearch Health Status
    • Single node clusters can operate safely with yellow status
    • Green status is recommended for multiple node clusters 
      # multiple node cluster will be green
clusterHealthCheckParams: "wait_for_status=green&timeout=1s"
# single node cluster will be yellow
clusterHealthCheckParams: "wait_for_status=yellow&timeout=1s"
  2. Kibana-Elasticsearch Dependency
    • Kibana is completely dependent on Elasticsearch
    • Kibana becomes inaccessible if Elasticsearch fails
  3. Filebeat Load Management
    • Full log collection can cause significant load
    • Recommend collecting logs from specific pods only
  4. Operator Installation
    • When using the operator, carefully analyze eck-operator-crds
    • Follow official documentation for operator setup


Reference