How to Implement Centralized Logging Using ELK Stack for Applications

Introduction

In modern application development, especially in microservices and distributed systems, logs are one of the most important sources of information for debugging, monitoring, and performance analysis. When applications run across multiple servers or containers, logs get scattered, making it difficult to trace issues.

Centralized logging solves this problem by collecting logs from different services and storing them in a single place where they can be searched, analyzed, and visualized.

One of the most popular solutions for centralized logging is the ELK Stack, which consists of Elasticsearch, Logstash, and Kibana.

This article provides a detailed, step-by-step guide on how to implement centralized logging using the ELK stack, along with examples, real-world use cases, advantages, and best practices.

What is ELK Stack?

ELK stands for:

Elasticsearch → Search and analytics engine
Logstash → Data processing and pipeline tool
Kibana → Visualization and dashboard tool

These three components work together to collect, process, store, and visualize logs.

How ELK Stack Works

Flow of Data

Applications generate logs
Logs are collected using agents (e.g., Filebeat)
Logstash processes and transforms logs
Elasticsearch stores and indexes logs
Kibana is used to search and visualize logs

This pipeline enables real-time log monitoring and analysis.

Step 1: Install Elasticsearch

Elasticsearch is responsible for storing and indexing logs.

Bash

sudo apt update
sudo apt install elasticsearch

Start and enable the service:

Bash

sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch

Explanation

Elasticsearch runs as a service
It stores logs in a searchable format
It provides fast querying capabilities

Step 2: Install Logstash

Logstash collects and processes logs before sending them to Elasticsearch.

Bash

sudo apt install logstash

Create a Logstash Configuration

JavaScript

input {
  beats {
    port => 5044
  }
}

filter {
  json {
    source => "message"
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "app-logs-%{+YYYY.MM.dd}"
  }
}

Explanation

Input listens for logs from Beats (like Filebeat)
Filter parses logs into structured JSON
Output sends logs to Elasticsearch

Step 3: Install Kibana

Kibana provides a UI to visualize and analyze logs.

Bash

sudo apt install kibana

Start the service:

Bash

sudo systemctl start kibana
sudo systemctl enable kibana

Access Kibana:

Code

http://localhost:5601

Explanation

Kibana connects to Elasticsearch
It allows searching logs using queries
It provides dashboards and visualizations

Step 4: Install Filebeat (Log Shipper)

Filebeat is used to collect logs from applications and send them to Logstash.

Bash

sudo apt install filebeat

Configure Filebeat

YAML

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log

output.logstash:
  hosts: ["localhost:5044"]

Start Filebeat:

Bash

sudo systemctl start filebeat
sudo systemctl enable filebeat

Explanation

Filebeat reads log files
Sends logs to Logstash
Lightweight and efficient

Step 5: Generate Logs from Application

Example in a .NET application:

using Serilog;

Log.Logger = new LoggerConfiguration()
    .WriteTo.File("logs/app.log", rollingInterval: RollingInterval.Day)
    .CreateLogger();

Log.Information("Application started");

Explanation

Logs are written to a file
Filebeat reads the file
Logs are sent to ELK pipeline

Step 6: Visualize Logs in Kibana

Open Kibana dashboard
Create an index pattern (e.g., app-logs-*)
Use Discover tab to search logs
Build dashboards for monitoring

Example Queries

Search errors: level: "Error"
Filter by date: @timestamp > now-1h

Real-World Use Cases

Monitoring microservices logs in production
Debugging distributed systems
Tracking user activity
Detecting security issues

Advantages of ELK Stack

Centralized log management
Real-time log analysis
Powerful search capabilities
Scalable and flexible

Disadvantages of ELK Stack

Requires setup and maintenance
Can consume significant resources
Learning curve for beginners

Best Practices

Use structured logging (JSON format)
Rotate logs to avoid disk issues
Secure Elasticsearch with authentication
Monitor ELK performance
Use dashboards for better insights

Summary

Centralized logging using the ELK stack is an essential practice for modern applications. It helps developers and DevOps teams collect, process, and analyze logs efficiently. By combining Elasticsearch, Logstash, and Kibana, you can build a powerful logging system that improves debugging, monitoring, and overall system reliability.