Real-time data monitoring with Grafana

In this story the way that data ingested into MySQL and monitored with grafana is going to be explained also the process of monitoring Kafka with prometheus and grafana is also described.

This project is developed here.

Building

I use docker to build the image for kafka, mysql, grafana, prometheus and jupyter. The docker-compose file is helped me to network these images.

Kafka

Monitoring kafka is one the most important issue in tech, with prometheus configuration, we will be to do that:

prometheus.yaml

Grafana data sources

Adding these configuration to grafana and prometheus docker image will help to ingest data from Kafka to prometheus and grafana and by adding this dashboard file to grafana image, a dashboard will automatically be created!

Gold Data Monitor

First the gold data has been gathered from kaggle. These data will be read and transfer with pandas then will be ingested into MySQL with Sqlalchemy.

Reading data

Transform date obj

sqlalchemy to mysql

By adding the mysql in data source and selecting table name: gold_data_v3 in dashboard creation, we will have these results:

docker-compose up --build -d