keshaw narayan pathak

Data Engineer

Freelancer•  January 2024 - September 2024•  Delhi

Collaborating with multiple international clients to design and implement scalable data pipelines. Developing real-time data processing solutions using AWS Glue and Apache Spark. Integrating data from diverse sources into Redshift and Snowflake for advanced analytics. Utilizing Databricks and ADF to optimize data workflows and enhance data quality.

Data Engineer

Ellow.io•  November 2022 - December 2023

Developed and maintained serverless data processing applications using AWS Lambda. Architected data storage solutions in DynamoDB to ensure high availability and performance. Created interactive dashboards using Node.js and JavaScript to provide real-time data insights.

Data Engineer

Shiprocket•  December 2021 - November 2022

Developed PySpark jobs to fetch data from RDS in incremental mode and store it in AWS S3. Implemented a layered data architecture in S3: Bronze (raw data), Silver (cleaned data), and Gold (processed data). Initially loaded final processed data into Redshift and later migrated to Snowflake after successful POC. Managed CDC logs data in Snowflake, consuming it from Confluent Kafka.

Associate IT Consultant

ITC InfoTech•  November 2020 - December 2021

Led data engineering projects involving Azure Data Factory and Streamsets. Developed data pipelines to migrate and transform data across cloud platforms.

Data Engineer

Freelancer•  September 2018 - October 2020

Worked with a security firm to handle large volumes of data from system logs, antivirus, and firewall protectors. Processed real-time data from system logs using an in-house tool and Kafka, while fetching additional data via API endpoints.

Data Engineer

Equifax India•  October 2017 - August 2018

Worked on Mix Media Marketing project to collect user interactions (touchpoints) with ads, including impressions, clicks, and conversions. Collected raw data in GCS via pixels associated with each ad and additional data from ad networks like DCM and Facebook Ad Networks.

Software Development Engineer

Sigmoid Analytics•  November 2016 - October 2017

Developed ETL pipelines to process and visualize real-time data in Elasticsearch and Kibana using PySpark and Kafka. Consumed real-time data using Kafka and PySpark, storing it in Kibana for visualization.

Software Engineer - Backend

Dailyhunt•  August 2015 - November 2016

Scraped data from 72+ e-commerce websites, storing URLs in MongoDB. Developed and managed jobs to fetch product details from each URL, storing results in MySQL.