Master AWS Lambda Functions for Data Engineers using Python

Build Lambda Functions using Python, Lambda Triggers, Deploy using layers and Docker, Validate using Glue and Athena
Master AWS Lambda Functions for Data Engineers using Python
File Size :
5.91 GB
Total length :
13h 11m

Category

Instructor

Durga Viswanatha Raju Gadiraju

Language

Last update

8/2022

Ratings

4.6/5

Master AWS Lambda Functions for Data Engineers using Python

What you’ll learn

Setup required tools on Windows to develop the code for ETL Data Pipelines using Python and AWS Services
Setup Project or Development Environment to develop applications using Python and AWS Services
Getting Started with AWS by creating account in AWS and also configure AWS CLI as well as Review Data Sets used for the project
Develop Core Logic to Ingest Data from source to AWS s3 using Python boto3
Getting Started with AWS Lambda Functions using Python 3 Run-time Environment
Refactor the application, build zip file to deploy as AWS Lambda Function
Create AWS Lambda Function using Zip file and Validate
Troubleshoot issues related to AWS Lambda Functions using AWS Cloudwatch
Build custom docker image for the application and push to AWS ECR
Create AWS Lambda Function using the custom docker image in AWS ECR
Develop Applications using AWS Lambda Functions by adding Python Modules as Layers

Master AWS Lambda Functions for Data Engineers using Python

Requirements

A computer science or IT Degree or 1 or 2 years of IT Experience
Basic Linux Skills with ability to run commands using Terminal
Programming Skills using Python is required
Valid AWS Account to use the AWS Services to learn how to build Data Pipelines using AWS Lambda Functions

Description

Do you want to learn AWS Lambda Functions by building an end-to-end data pipeline using Python as Programming Language and other key AWS Services such as Boto3, S3, Dynamodb, ECR, Cloudwatch, Glue Catalog, Athena, etc? Here is one course using which you will learn AWS Lambda Functions by implementing an end-to-end pipeline by using all the services mentioned.As part of this course, you will learn how to develop and deploy lambda functions using the zip files, custom docker images as well as layers. Also, you will understand how to trigger lambda functions from Eventsbridge as well as Step Functions.Set up required tools on Windows to develop the code for ETL Data Pipelines using Python and AWS Services. You will take care of setting up Ubuntu using wsl, Docker Desktop, and Visual Studio Code along with Remote Development Extension Kit so that you can develop Python-based applications using AWS Services.Setup Project or Development Environment to develop applications using Python and AWS Services on Windows and Mac.Getting Started with AWS by creating an account in AWS and also configuring AWS CLI as well as Review Data Sets used for the projectDevelop Core Logic to Ingest Data from source to AWS s3 using Python boto3. The application will be built using Boto3 to interact with AWS Services, Pandas for date arithmetic, and requests to get the files from the source via REST API.Getting Started with AWS Lambda Functions using Python 3.9 Run-time EnvironmentRefactor the application, and build a zip file to deploy as AWS Lambda Function. The application logic includes capturing bookmarks as well as Job Run details in Dynamodb. You will also get an overview of Dynamodb and how to interact with Dynamodb to manage Bookmark as well as Job Run details.Create AWS Lambda Function using a Zip file, deploy using AWS Console and Validate.Troubleshoot issues related to AWS Lambda Functions using AWS CloudwatchBuild a custom docker image for the application and push it to AWS ECRCreate AWS Lambda Function using the custom docker image in AWS ECR and then validate.Get an understanding of AWS s3 Event Notifications or s3-based triggers on Lambda Function.Develop another Python application to transform the data and also write the data in the form of Parquet to s3. The application will be built using Pandas by converting 10,000 records at a time to Parquet.Build orchestrated pipeline using AWS s3 Event Notifications between the two Lambda Functions.Schedule the first lambda function using AWS EventsBridge and then validate.Finally, create an AWS Glue Catalog table on the s3 location which has parquet files, and validate by running SQL Queries using AWS Athena.After going through the complete life cycle of Deploying and Scheduling Lambda Function and also validating the data by using Glue Catalog and AWS Athena, you will also understand how to use Layers for Lambda Function.Here are the key takeaways from this training:Develop Python Applications and Deploy as Lambda Functions by using a Zip-based bundle as well as a custom docker image.Monitor and troubleshoot the issues by going through Cloudwatch logs.The entire application code used for the demo along with the notebook used to come up with core logic.Ability to build solutions using multiple AWS Services such as Boto3, S3, Dynamodb, ECR, Cloudwatch, Glue Catalog, Athena, etc

Overview

Section 1: Introduction to Mastering AWS Lambda Functions for Data Engineers

Lecture 1 Introduction to Mastering AWS Lambda Functions for Data Engineers

Lecture 2 Resources used for Mastering AWS Lambda Functions for Data Engineers

Section 2: Getting Started on Windows with Required Tools

Lecture 3 Introduction to Getting Started on Windows with Required Tools

Lecture 4 Overview of Powershell on Windows 10 or Windows 11

Lecture 5 Setup Ubuntu VM on Windows 10 or 11 using wsl

Lecture 6 Setup Ubuntu VM on Windows 10 or 11 using wsl

Lecture 7 Setup Docker Desktop on Windows

Lecture 8 Validate Docker on Windows using Command Line leveraging Power Shell

Lecture 9 Review Docker Desktop Resource Configurations

Lecture 10 Install Visual Studio Code on Windows

Lecture 11 Install Remote Development Extension Kit for Visual Studio Code

Lecture 12 Install Python 3.9 and Distutils on Windows using wsl Ubuntu

Lecture 13 Review Tools Installed for Application Development using Python and AWS Service

Section 3: Setup Development Environment to build Data Pipelines using AWS

Lecture 14 Setup Project Folder using Visual Studio Code

Lecture 15 Ensure Python 3.9 for the Project

Lecture 16 Create Python Virtual Environment using Python 3

Lecture 17 Install Required Dependencies for the Project using AWS Services

Lecture 18 Ensure AWS CLI to interact with AWS Services using AWS CLI Commands

Lecture 19 Recommendation to use Personal AWS Account for the course

Section 4: Getting Started with AWS and Review Data Sets

Lecture 20 Setup and Login into AWS Account

Lecture 21 Setup AWS IAM User with Administrator Permissions

Lecture 22 Configure and Validate AWS CLI

Lecture 23 Configure AWS CLI with custom profile as default

Lecture 24 Recap of Date Arithmetic using Python

Lecture 25 Validate Python boto3 to interact with AWS Services

Lecture 26 Setup and Validate Jupyter based Interactive Environment

Lecture 27 Review GHActivity Data Details

Lecture 28 Download GHActivity Data using requests

Lecture 29 Review GHActivity Data using Pandas

Section 5: Core Logic to Ingest Data from Web Service to AWS s3

Lecture 30 Managing s3 using Python boto3

Lecture 31 Overview of AWS Dynamodb

Lecture 32 Create DynamoDB Table for Job Details

Lecture 33 Create DynamoDB Table for Job Run Details

Lecture 34 Get First Run Details to Copy GHActivity Data to AWS s3

Lecture 35 Get Incremental Load Logic for next file

Lecture 36 Understand AWS s3 concepts such as buckets and objects

Lecture 37 Copying or Uploading Files to AWS s3 as objects using Python boto3

Lecture 38 Writing Python Objects or Data as AWS s3 Objects using boto3

Lecture 39 Save GHActivity Data to AWS s3

Lecture 40 Convert Date Time to Integer Unix Epoch using Python

Lecture 41 Save Job Run Details to DynamoDB Table

Lecture 42 Validate Data Copied to AWS s3 and job run details

Lecture 43 Run and Validate End to End Process

Section 6: Getting Started with AWS Lambda Functions

Lecture 44 Introduction to Getting Started with AWS Lambda Functions

Lecture 45 Overview of AWS Lambda and Getting Started using Python 3

Lecture 46 Passing Arguments to AWS Lambda and Processing using Python

Lecture 47 Using Custom Handlers for AWS Lambda Functions using Python 3

Lecture 48 Using AWS Services such as s3 in AWS Lambda Functions

Lecture 49 Recap of handling permissions using AWS IAM Roles and User Groups

Lecture 50 Develop AWS Lambda Function to list objects from AWS S3 Bucket

Lecture 51 Passing Environment Variables to AWS Lambda Functions

Lecture 52 Customizing Resources such as memory used for AWS Lambda Function

Lecture 53 Setup Local Development Environment for AWS Lambda Functions

Lecture 54 Develop logic for AWS Lambda Function using external packages

Lecture 55 Build Zip file to deploy as AWS Lambda Function

Lecture 56 Deploy Application with External Dependencies as AWS Lambda Function

Lecture 57 Understand Problem Statement for Python Application for AWS

Lecture 58 Setup Python Project for AWS Lambda using Visual Studio Code

Lecture 59 Core Logic to upload files to AWS S3 using Python boto3

Lecture 60 Develop Python Application to upload files to AWS s3 using Python boto3

Lecture 61 Build Zip File for Python Application to deploy as AWS Lambda Function

Lecture 62 Deploy Python Application as AWS Lambda Function using Zip File

Lecture 63 Conclusion and request for rating and feedback

Section 7: Build and Deploy AWS Lambda Function using Zip File

Lecture 64 Introduction to Build and Deploy AWS Lambda Function using Zip File

Lecture 65 Update Application Code with Core logic for Ingestion

Lecture 66 Overview of Validating User Defined Functions using Python CLI

Lecture 67 Validate Application using Core Logic to ingest data

Lecture 68 Add Lambda Handler to ingest data to AWS s3

Lecture 69 Build Zip File for Python Application to deploy as AWS Lambda Function

Lecture 70 Upload Python Application Zip File to s3 and deploy as AWS Lambda Function

Lecture 71 Set Custom Handler and required Environment Variables for AWS Lambda Function

Lecture 72 Granting Permissions on AWS s3 and Dynamodb to AWS Lambda Function via Role

Lecture 73 Change Memory and Timeout for AWS Lambda Function and Test

Lecture 74 Recap and Overview of Monitoring Lambda Functions using Cloudwatch

Lecture 75 Limitations of Deploying AWS Lambda Function using Zip file

Lecture 76 Automate Build of AWS Lambda Function using Shell Scripts

Section 8: Build and Deploy AWS Lambda Function using Custom Docker Image

Lecture 77 Introduction to Build and Deploy AWS Lambda Function using Custom Docker Imag

Lecture 78 Create Dockerfile for Custom Docker Image for AWS Lambda Function

Lecture 79 Create Custom Docker Image for AWS Lambda Function using Python 3 Run-time

Lecture 80 Validate Custom Docker Image by creating Docker Container

Lecture 81 Run the application using Python CLI in the Docker Container

Lecture 82 Run the Docker Container with the Credentials and Environment Variables

Lecture 83 Validate AWS Lambda Function Locally using Docker and Curl

Lecture 84 Create AWS ECR Repository for Custom Docker Image

Lecture 85 Push Custom Docker Image for AWS Lambda Function to AWS ECR

Lecture 86 Create AWS Lambda Function using Custom Docker Image in AWS ECR

Lecture 87 Run and Validate AWS Lambda Function created using Custom Docker Image

Lecture 88 Create Shell Script to Build and Push Docker Image to AWS ECR

Lecture 89 Add Command to build script to reconfigure AWS Lambda Function to latest docker

Section 9: Overview of AWS s3 Event Notifications with Lambda or s3 Triggers on Lambda

Lecture 90 Introduction to AWS s3 Event Notifications with Lambda or s3 Triggers on Lamb

Lecture 91 Switching to Different Profile for the demos on AWS s3 Event Notifications

Lecture 92 Setup Project to explore AWS Lambda Triggers or s3 Event Notifications

Lecture 93 Setup Required Datasets for AWS Lambda Triggers or s3 Event Triggers

Lecture 94 Listing AWS s3 Buckets and Objects using Python Boto3

Lecture 95 Listing AWS s3 Objects based on Prefix using Python boto3

Lecture 96 Overview of AWS s3 Events and Event Notification

Lecture 97 Create Simple AWS Lambda Function for s3 Event Notifications or Triggers

Lecture 98 Create Trigger for AWS Lambda Function for s3 Put Event

Lecture 99 Get AWS s3 Object Details from AWS Lambda Arguments

Lecture 100 Develop Read Logic to read from s3

Lecture 101 Convert CSV to JSON Strings using AWS Lambda Function

Lecture 102 Add Logic to Upload JSON File to s3 and Validate AWS Lambda Trigger

Lecture 103 Configure AWS Lambda as AWS s3 Event Notification

Section 10: Develop and Deploy AWS Lambda Function to transform the data

Lecture 104 Introduction to Develop and Deploy AWS Lambda to transform the data

Lecture 105 Basic Approach to convert JSON to Parquet using Python Pandas

Lecture 106 Convert JSON Files to Parquet using Pandas in Chunks

Lecture 107 Setup Job Metadata for JSON to Parquet

Lecture 108 Add Core Logic to transform data to the Application Code

Lecture 109 Develop Wrapper to transform the data from JSON to Parquet

Lecture 110 Validate Function to Convert JSON to Parquet

Lecture 111 Develop Lambda Handler to transform JSON Files to Parquet

Lecture 112 Validate Lambda Handler to transform the data locally using curl

Lecture 113 Push Docker Image to ECR and Update Existing Lambda Function

Lecture 114 Deploy and Run Transformer as AWS Lambda Function

Lecture 115 Perform Validations of AWS Lambda Function to Transform the data

Section 11: Build and Schedule the Pipeline using AWS Lambda Trigger and EventsBridge

Lecture 116 Introduction to Build and Schedule Data Pipeline using AWS Lambda Trigger and

Lecture 117 Review AWS Lambda Trigger and understand arguments

Lecture 118 Add Custom Lambda Handler for s3 Trigger

Lecture 119 Add Trigger to AWS Lambda Function deployed to transform the data

Lecture 120 Validate s3 Lambda Trigger by invoking ingestor

Lecture 121 Understand Internals of AWS Lambda Functions

Lecture 122 Overview of AWS EventBridge Rules

Lecture 123 Overview of AWS EventBridge Rule for Schedule

Lecture 124 Schedule Data Pipeline using AWS EventBridge Rule

Lecture 125 Validate Data Pipeline Run by AWS EventBridge Schedule

Lecture 126 Validate the pipeline after few Scheduled Runs

Lecture 127 Reset Data Pipeline built using AWS Services

Lecture 128 Determine File Not Found in the source using requests

Lecture 129 Troubleshoot the bug in the Data Pipeline

Section 12: Perform Adhoc Analysis of Data using Glue Catalog Tables and Athena Queries

Lecture 130 Introduction to Adhoc Analysis of Data using Glue Catalog Tables and Athena Quer

Lecture 131 Overview of AWS Glue to create Catalog Tables

Lecture 132 Creating AWS Glue Crawler for GHActivity Data

Lecture 133 Create AWS Glue Catalog Table using Glue Crawler

Lecture 134 Getting Started with Amazon Athena to run SQL Queries on s3 and Glue Catalog Tab

Lecture 135 Preview first few records and get count using AWS Athena Queries

Lecture 136 Get Count By Date using AWS Athena SQL Queries

Lecture 137 Setup AWS Profile for the VSCode Project and Install Pyathena

Lecture 138 Getting Started with Pyathena and Pandas

Lecture 139 Additional Athena Queries for adhoc Analysis using Glue Catalog Tables on s3Dat

Section 13: Deploy AWS Lambda Functions using Python Runtime with Layers

Lecture 140 Introduction to Deploying AWS Lambda Functions using Python Runtime with Layers

Lecture 141 Create Lambda Function to explore layers

Lecture 142 Get list of Python Libraries installed in AWS Lambda Runtime

Lecture 143 Add Existing AWS Layer to Lambda Function using Python runtime

Lecture 144 Steps to Add and Configure Custom Layers to AWS Lambda Functions

Lecture 145 Setup Local Environment using AWS Cloud Shell to Create Custom Layer

Lecture 146 Install Required Dependencies for Lambda Layer for Python Runtime

Lecture 147 Create Zip File and Upload to s3 with Python dependencies for AWS Lambda Layer

Lecture 148 Create Lambda Layer using AWS Lambda Console using zip file in AWS s3

Lecture 149 Configure Lambda Function with Custom Layer for Pandas and Requests

Lecture 150 Troubleshoot and Fix the issues related to Lambda Layers for AWS Lambda Functio

Lecture 151 Upload Zip File with Python boto3 to s3 for AWS Lambda Layer

Lecture 152 Create Lambda Layer with latest version of Python boto3 for AWS Lambda Function

Lecture 153 Deploy AWS Lambda Function Sample Application with Layers

University Students who want to learn AWS Lambda Functions with hands on and real time examples,Aspiring Data Engineers and Data Scientists who want to master AWS Lambda Functions for Data Processing with real time examples,Entry Level IT Professionals with prior basic level programming experience who would like to build complete projects using AWS and Python,Experienced Application Developers who would like to explore how to build end to end applications using Python and AWS Services,Experienced Data Engineers to build end to end data pipelines using Python and AWS Services,Any IT Professional who is keen to get started with AWS using some of the key services such as s3, Dynamodb, Lambda, ECR, IAM, etc.

Course Information:

Udemy | English | 13h 11m | 5.91 GB
Created by: Durga Viswanatha Raju Gadiraju

You Can See More Courses in the IT & Software >> Greetings from CourseDown.com

New Courses

Scroll to Top