Priyanshu Naidu

Software Engineer

capgemini•  January 2022 - March 2024•  Mumbai

Designed and maintained ~400 CI/CD data pipelines, processing up to 4 billion semi-structured data records daily using ETL process in ADF Spark on the Power Bi dashboard Developed end to end data pipelines. Utilized complex SQL queries to Extract, Manipulate, and Transform data from multiple sources and use in in Azure Data Factory. Contributed in Design Discussions to implement the best practices of Data Modelling and Data Warehousing to build robust and highly scalable pipelines and continuous monitoring. Created Data Flows and corrected Errors occurring in existing pipelines and data flows for high speed processing, Improving business performance and overall efficiency. Collaborated with cross functional team and product owner in Agile environment to deliver an entire new Master pipeline with multiple execution pipelines and its Data Flows covering development, testing, validation, CI/CD deployment.