Atul Kumar

Data Engineer

Rapido•  September 2022 - Present

Developed and maintained data pipelines using Spark, Scala, GCS, Airflow, Presto, and GoCD to provide tailored datasets for cross-functional teams, enabling data-driven decision-making. Facilitated knowledge sharing through comprehensive documentation and code reviews, and mentored junior team members to enhance team efficiency. Identified opportunities to optimize existing datasets, implementing cost-effective solutions that improved overall data processing efficiency.

Senior Software Engineer

Sigmoid Analytics•  January 2022 - August 2022

Modularising, structuring and optimising the execution time of code from Pandas and R to Pyspark the execution time. Exploring Azure Cloud Services.

Software Engineer

Sigmoid Analytics•  January 2021 - December 2021

Migrated and optimise code from pandas to PySpark to improve computation cost. Built a system to optimise the forecast input Cost of raw commodities to predict the future opportunity or risk.

Associate Software Development Engineer

Sigmoid Analytics•  May 2019 - December 2020

Built end to end ETL pipelines using Apache Spark, Hadoop, MySQL, MongoDB, Scala to process TBs of data. Provided Big Data solutions to various customers with minimal query serve time. Optimize the data processing cost and no. of partitions to save the data for indexing.