Dhaval Patel

Software Data Engineer - Transit, Bikes & Scooters Team

Lyft•  July 2021 - November 2022

Technology: Python, SQL, Airflow, Hive, Trino, Spark - Redesigned ETL process to incorporate new events and snapshot data sources for hardware repair dataset. - Led migration initiative to migrate 80+ Airflow DAGs from Hive to Spark as part of cost saving initiative. - Developed datasets to track SIM card activity by working with cross functional teams to prevent paying subscription charges. - Integrated monitoring utilities on Data Pipelines to reduce troubleshooting time during On Call. - Contributed new features into custom ETL framework which helps Data Scientist build ETL pipelines easily.

Software Engineer Intern - Enterprise Data Management Team (Part Time)

NYU IT•  March 2020 - May 2021

Technology: Python, Flask, React.JS, AWS, Snowflake - Developed comparison method to calculate delta changes with respect to data present on upstream data sources. - Deployed Cron job to migrate data from S3 bucket to Snowflake Data Warehouse using Lambda and CloudWatch. - Created API endpoints using Flask to develop API Request Web Portal to deliver information about various databases.

Data Engineer Intern - HBO Max Team

HBO•  January 2021 - April 2021

Technology: Python, SQL, AWS, Airflow, Snowflake, Datadog - Integrated Data Quality Framework on Slack to provide custom messaging for various data pipelines. - Designed monitoring tool for Lambda function failures using CloudWatch metrics to send alerts via SNS topics to Slack. - Supported ETL process using Lambda functions to Snapshot Looker Dashboard data into Snowflake. - Implemented POC for alerting and anomaly detection on critical metrics using Datadog for production releases.

Cloud Technology Intern - Data Analytics and Platform Support Team

HBO•  June 2020 - August 2020

Technology: Python, SQL, AWS, Airflow, Grafana - Designed custom dashboard using Grafana to monitor daily job runs and provide insights for job failures. - Developed rule base alert system to notify Data Engineering and Data Science teams about critical 200+ data sources. - Integrated SQL queries to get information of different Airflow DAGs executions runtime and usage.

Software Engineering Intern

MAQ Software•  January 2019 - May 2019

Technology: Python, SQL, SSIS, SSMS, Power BI - Developed data pipelines to handle Transactional Sales data by integrating 15+ source tables. - Designed Metrics and KPI dashboard using Power BI to give data insights to client and senior management.