uzaif quadri

Data Engineer

Webangel Tech•  December 2023 - Present•  Bhopal

Developed Databricks notebooks and Job Compute. Designed and implemented a data lake solution on Azure using Azure Data Lake Storage and Azure Databricks to enable real-time data processing and analysis. Developed data ingestion pipelines and data transformation workflows using Python and Apache Spark. Responsible for creating and maintaining data pipelines in Azure Data factory, Pipeline components such as Integration Runtime, Linked Services, Datasets and Activities. Created nested pipelines with various triggering mechanisms. Transforming data with Azure Data Flows. Pipelines migration among different workspaces. Created script to perform ETL operation on unstructured files in Azure Databricks. Optimized data pipelines for performance and scalability. Implemented data quality checks and data validation routines to ensure accuracy and consistency of data. Collaborated with cross-functional teams to gather requirements and design data models for data warehousing projects.