Introduction There are lots of benefit of using MySQL as a backend database in Airflow. Main reason is that MySQL is widely used in production instead of SQlite. Also, we can have scalable database system where we can have concurrent requests, high security and well defined permission and roles. From the last few blogs, I’ve […]
airflow
Branching Tasks in Airflow For Dynamic Flow of DAGs
Branching Task in Airflow When do we need to make a branch like flow of a task? A simple example could be, lets assume that we are in a Media Company and our task is to provide personalized content experience. Lets assume that we will have 3 different sets of rules for 3 different types […]
Connecting to MySQL Server in Windows Machine from WSL
Connecting MySQL Server in Windows Machine from WSL What does this mean? In simple sentence, how do we connect to a MySQL server which is hosted in Windows from WSL. It might sound easy but let me tell you, IT IS NOT!!!! I was trying to connect (from WSL) to my local MySQL which was […]
Airflow in WSL and Dynamic Tasks
This blog is a continuation of previous blog Getting Started With Airflow in WSL. Dynamic Tasks in Airflow Sometimes there will be a need to create different task for different purpose within a DAG and those task has to be run dynamically. Not only run but has to be created dynamically also. A simple example […]
Running Airflow In WSL and Getting Started With It
Airflow in WSL: Getting Started With I might stop to write new blogs in this site so please visit dataqoil.com for more cool stuffs. Blog Versions This blog will be updated frequently. 2021-12-01 Written blog. 2021-12-05 Updated contents upto TaskFlowAPI. 2021-12-07 Updated contents upto SubDAGs. 2021-12-13 Updated contents upto TaskGroup. 2022-01-09 Started dynamic tasks in […]