Deepanshu Luhach

Software Engineer

Phenom People•  September 2021 - Present

• Spearheaded the comprehensive migration of data from MongoDB to Snowflake, involving the seamless transition of data and re-modeling of data into a Snowflake schema architecture and modification of analytics SQL Queries. • Implemented Apache Spark streaming application, deployed on EMR cluster to process data consumed from multiple Apache Kafka topics and write it into the data lake built on AWS S3. • Designed and implemented efficient ETL pipelines using Apache Airflow to process diverse data (including site traffic, job application, and hiring data) present in the data lake and load it into fact and dimension tables on Snowflake, enabling better decision-making and insights for the organization. • Used Snowflake features like Secure Data Sharing and Cross region data replication to solve complex data migration and data sharing problems. • Improved Analytics Dashboard performance by 3X through optimization of SQL queries.

Software Engineer

Barclays•  July 2019 - September 2021

• Worked on a data migration project to migrate data from on-premises storage to on-cloud Hadoop to achieve PCI-DSS compliance. • Automated the process with a history migration data pipeline using configs to streamline the effort and minimize human errors. Utilized SAS and shell scripts to ingest data and Apache Spark to process and load data into Hadoop. • Implemented multiple custom SAS Functions (such as email) to limit and audit the use of Unix functions on SAS.