Badges
Certifications
sontakkeujwal511 has not earned any certificates yet.
Work Experience
Data Engineer
HCL Technologies• February 2022 - Present• Pune, Maharashtra
• Designed and implemented a scalable AWS-based data processing pipeline for handling large transaction data of 3 million daily record. • Utilized Apache Spark-Scala for efficient batch processing & minimizing execution by 20 times. • Employed Hive for Data warehousing for structure level optimization like partitioning and bucketing majorly for highly optimized JOIN queries for 5 normalized tables. • Coordinated ETL workflows for data integrity and quality assurance. • Implemented incremental load techniques in big data pipelines, enhancing data processing efficiency and minimizing latency. • Implemented performance tuning techniques such as broadcast variables, caching, persist and dynamic allocation of resources for Spark jobs to enhanced throughput and resource utilization. • Leveraged optimized file formatting like CSV for its simplicity in data exchange and interoperability across systems additionally Avro for flexible schema evolution along with implementing Parquet for optimized columnar storage, enhancing nested query performance and minimizing storage overhead in big data environments.
Education
sontakkeujwal511 has not updated education details yet.