It sounds really amazing scaling airflow when we want grow and run as many as possible in parallell. Here I just present Airflow setup for scaleing out and the some spark airflow ETL. The source code for this article is in here.
There are several way to scale Airflow workers, one of the best way is to use celery queue. Another way is using Kubernetes Executor but Argo is more better solution if Kubernetes is going to be used for workflow management.
Here I am going to explain about scaling spark on kubernetes task on Airflow:
Here I am going to talk about doing CRUD using Delta Lake and Spark. I heard a lot of good stuff about Delta Lake and I just try to work with it and share my experience. Also it kinda supports ACID transactions on Spark.
Here is the technologies that is going to be used:
The more details and walking tour could be found here:
In this story, a ML pipeline service for training machine learning model is going to be illustrated.
This platform consists of these services:
In this article the implementation of real-time bitcoin prediction and monitoring is going to be explained.
The project for this story has been developed in this repository.
For this job I use one of the cool open api and I am very thankful to them.
In this story, the process of developing a live chatroom with fastapi websocket and build it with docker and deploy it on heroku :))
This repository include all code, and docker files.
This class handles that messages deliver to the member of the specified chatroom.
In this article the development and deployment of a microservice is going to be explained. This microservice consists of:
In the first part the machine learning part is going to be described. This fraud detection project notebook could be found here.
Because of more simplicity of random…
The deployment of the nameko microservice that explained HERE with kubernetes. As it mentioned in part 1, each of microservice component has its own docker image; therefore in this story the process of deploying them is explained.
In this story the whole process of building the microservice app with nameko that has both flask and fastapi as web service, is going to explained.
This project is developed on docker container and is going to be deployed with kubernetes. The repo for this project is here.