Karan Naidu

Data Engineer

accenture•  January 2022 - Present

● Led the development and management of 100+ specialized ETL pipelines leveraging Azure Data Factory and Azure Databricks for a US based Utility company (Electricity, Oil & Gas), enabling seamless data integration and transformation data. ● Loaded and transformed large sets of semi structured data likes XML, JSON, Avro, Parquet, resulting in a 40% increase in data processing speed and enhanced data-driven decision making and business insights. ● Ensuring the scalability, security and availability of the data infrastructure collaborating with data architects and 5+ cross-functional teams to understand business requirements and develop data-driven solutions. ● Conducted data analysis to determine requirements for 100+ scenarios and integrated appropriate queries and programs to meet those needs. ● Deployed robust Azure security & monitoring practices for data platform, reducing data breaches and optimizing costs by 20%.

Data Engineer Associate

accenture•  July 2021 - January 2022

● Engineered the migration of 500 TB of data from HDFS to Azure Blob, leveraging Azure Data Factory for efficient transfer and implementing automated validation processes, achieving a 99.7% accuracy for datasets. ● Transformed PySpark code for 30% high-speed data processing and performed SCD to meet business requirements for Azure Synapse Analytics. ● Developed Python scripts to automate data processing and quality checks, reducing manual effort by 40% and improving data accuracy by 25%. ● Analyzed 50+ functional documents with data scientists designed by the Shareholders and Development team. ● Scripted CI/CD pipelines with Jenkins for data transformation, achieving 20% reduction in manual intervention.

Data Science Intern

Great Place IT Services•  July 2020 - July 2021

● Spearheaded the creation of an innovative Theme Extraction system using NLP and Python programming techniques, enabling seamless automated sentiment analysis. This resulted in a published research paper. ● Designed 4 data analysis frameworks using Spark and Power BI enabling text mining, natural language processing, and deep learning techniques to analyze collected data and generate insightful reports.