Photo by Proxyclick Visitor Management System on Unsplash

Why do this?

It is often a practice of embedding one’s credentials within code for authentication during development and testing phases. However, such practice is avoided for production grade code as it exposes some vulnerabilities.

While development teams are adept to the practice of embedding credentials in the bash profile of production servers…

Photo by ADITYA PRAKASH on Unsplash

If you have been following the series (Part 1 and Part 2); so far we have established a Lambda based Flask App that handles GET and POST requests. We then configured API Gateway to route suitable requests to the Lambda along with error handling methods.

In our quest to create…

Photo by ADITYA PRAKASH on Unsplash

Part 1 of this series dealt with the steps that one must follow to expose a simple GET method on API Gateway that mapped to a handler function in Lambda.

Part 2 will dwell on the steps(though similar to the GET) that will build up to creating and deploying a…

Photo by ADITYA PRAKASH on Unsplash

Just last week when I was confident on knowing most of AWS Lambda’s Capabilities; my colleague threw me a new one. API!!!!

I metaphorically ran over to my desk (googled) to see if that was even possible. Turns out, yeah!

Well that was enough to throw my work out of…

Photo by Seika I on Unsplash

Snowflake has indeed proven to be a force to reckon with when it comes to SQL based databases, offering much more flexibility minus the overhead of hosting and maintenance. …

Photo by Austin Distel on Unsplash

I have spent the last few weeks digging and hunting for blogs to assist in launching Tableau Server on AWS and found nothing!!!

I thus intend to visually enlighten the steps one must follow to get through installation and configuration in matter of minutes.

First things first; this link is…

Photo by Bence Balla-Schottner on Unsplash

Why do all this?

Out-of-box configuration of Airflow allows us to execute tasks sequentially which is ideal if your DAG depends on it.

However, there are certain use cases which would require for tasks to be run in parallel. …

Photo by Createria on Unsplash

Lately, many organizations have turned towards Airflow to orchestrate their Data pipelines which stresses more on programming than the traditional drag and drop functionality. For this reason and many more, Airflow has proven to be more flexible and robust with regards to kind of pipelines one can build and maintain.

Photo by Scott Sanker on Unsplash

I have often used Hive in conjunction with HDFS to propel my big data jobs on pyspark which is easy to setup and use.

Hive Metastore hosted on the master node made perfect sense with an on-premise setup where the cluster was always up; save for times when the Master/Slave…

Photo by SpaceX on Unsplash

The Beginner’s Guide describes Jupyter Notebook as “The Jupyter Notebook App is a server-client application that allows editing and running notebook documents via a web browser. …

Christopher Lagali

An Enthusiastic Data Eng. who is on a mission to unravel the possibilities of pipeline building with AWS and who believes in knowledge sharing.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store