Real-time Data Monitoring using Kafka, Logstash, Elasticsearch and Kibana
In this article, the whole process of monitoring log data with ELKK stack is going to be described. This stack consists of Kafka as message broker to produce data into ELK stack.
First, the logs are going to be produced to a topic in kafka; these logs are Json and needed to be parsed. In order to parsing the data in kafka topics and index them in the elasticsearch; then in kibana the indexed logs, would be available for dashboard and monitoring. This document is focused on logstash configs for connection to kafka, filtering and indexing in elasticsearch. Also configuration of kibana for elasticsearch connections.
In this log data monitoring stack, first kafka produces streams of logs to a topic with name of log. The ELKKstack is demonstrated in here:
For the purpose of getting logs from kafka, the logstash needs to be well configured. More information on how to create logstash config:
- https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html
- https://www.elastic.co/guide/en/logstash/current/field-extraction.html
- https://www.elastic.co/guide/en/logstash/current/data-deserialization.html
This project is deployed HERE, which gathered logs from log topic and kafka:9092 broker address. The logstash configuration is developed here. Also the kibana configuration is developed here. This process is deployed with this docker-compose file.
How to run
First go to the meduim directory in the project. Then:
make init
make build_start
after all containers were created:
make topic-log
Then go to the jupyter address in its logs; and run the producer then in kibana discovery logs are shown.
For the purpose of simulation, a jupyter lab is deployed locally to first produce some data to log then by using those data a consume notebook, new data will be produced to log topic. And simultaneously, the logs will parsed with logstash and indexed in elasticsearch. Finally by discovering log_test in kibana; all fields will be shown.
To make sure that the log name that has been configured in logstash is availabe HERE.
This is a sample visualization in kibana:
