Kartikeya Singh

Associate Software Engineer II - Data

Optum•  June 2022 - Present

Created a high-performance ETL script with AWS Glue for the anonymization of 1 Billion records of PHI and PII, leading to enhanced data integrity and privacy compliance. Overhauled a complex Java JAR Spark job into a streamlined PySpark data pipeline, resulting in a 20% reduction in processing time while simplifying the creation of 50 tables and enhancing maintainability of the codebase. Optimized data processing by consolidating two Big Data pipelines into a single, efficient pipeline, utilizing dataframes to achieve a savings of 15TB of storage in Amazon S3. Spearheaded development of sample datasets, converting large datasets into efficient n% subsets, significantly reducing testing time for developers. Improved testing coverage and workflow efficiency reducing testing time by 50%. Optimized and reduced the runtime of 2 Spark applications using techniques such as shuffle partitions, checkpoint(), cache(), repartition(), broadcast(), and Window functions to enhance performance and efficiency.