IT Blog

  We are living in the Airflow era. Almost all of us started our scheduling journey with cronjobs and the transition to a workflow scheduler like Airflow has given us better handling with complex inter-dependent pipelines, UI based scheduling, retry mechanism, alerts & what not! AWS also recently announced managed  airflow  workflows. These are truly exciting times and today, Airflow has really changed the scheduling landscape, with scheduling configuration as a code. Let’s dig deeper. Now, coming to a use case where I really dug down in airflow capabilities. For the below use case, all the references to  task  are for  airflow tasks . The Use case The above DAG consists of the following operations: Start an AWS EMR cluster : EMR is an AWS based big data environment. To understand the use case we don’t need to deep dive into how AWS EMR works. But if you want to, you can read more about it  here . Airflow task_id for this operation:  EMR_start...

Introduction If you are starting with Big Data it is common to feel overwhelmed by the large number of tools, frameworks and options to choose from. In this article, I will try to summarize the ingredients and the basic recipe to get you started in your Big Data journey. My goal is to categorize the different tools and try to explain the purpose of each tool and how it fits within the ecosystem. First let’s review some considerations and to check if you really have a Big Data problem . I will focus on open source solutions that can be deployed on-prem . Cloud providers provide several solutions for your data needs, and I will slightly mention them on this article. If you are running in the cloud, you should really check what options are available to you and compare them to the open source solutions looking at cost, operability, manageability, monitoring and time to market dimensions. Big Data Ecosystem Data Considerations (If you have experience with big data, skip to t...