SAKSHAT JOSHI

Software Engineer

american chase•  July 2024 - Present•  Indore

* Designed and implemented enhancements in AWS Glue and YAML configurations to support evolving business requirements, optimising data model performance. * Added new columns to AWS Glue tables and optimised dataset caching, significantly improving data pipeline efficiency. * Automated dataset management by implementing a YAML-based purging mechanism, reducing storage costs and improving operational workflows. * Developed a robust testing base by facilitating seamless financial dataset migration from acceptance to development environments. * Upgraded a difference-checking tool to enable attribute-level comparisons at folder levels, improving efficiency for large-scale data validations. * Executed YAML scripts to generate high-quality parquet files from golden sample datasets, accelerating data validation cycles. * Streamlined data pipeline operations by refining dataset caching mechanisms, achieving significant performance gains. * Enhanced YAML configuration efficiency by automating dataset lifecycle management, improving maintainability and storage utilisation. * Designed and implemented mechanisms to validate control totals, ensuring the integrity of financial datasets across pipelines. * Spearheaded improvements in data migration processes, enabling accurate testing and validation across environments. * Refactored YAML scripts to optimise data frame operations, resulting in faster and more reliable data transformations. * Implemented mechanisms for unused dataset purging in YAML configurations, streamlining data lifecycle management. * Improved performance of ETL pipelines by redesigning dataset caching strategies and eliminating redundancies in processing workflows.