Badges
Certifications
balaji_kottana05 has not earned any certificates yet.
Work Experience
Lead Data Engineer
AT & T (End Client)• November 2023 - Present
● Lead the team in migrating existing on-prem OLAP solution to Snowflake using Databricks. ● Worked extensively in Informatica designer to identify the source analysis of the dimension tables and transformed them to Snowflake through Databricks transformation jobs. ● Orchestrated pipelines to transform data from AWS S3 buckets to Snowflake using Databricks Jobs. ● Performed Code reviews for the pipelines designed by the team and ensured error free code and readable code is deployed in the production environment. ● Optimized the query performance of Views by working on micro partitions in Snowflake. ● Proposed implementation guidelines when using cloud resources which avoid underutilization of resources while developing pipelines. ● Tested the implemented pipelines for correct results with on-prem on cloud warehouse. ● Worked in Teradata learn the source OLAP system. ● Ensured the pipelines are documented for further enhancements.
Senior Data Engineer
National Informatics Centre• January 2018 - August 2022
Public Distribution System ● Served as a SME to the PDS department to handle end to end activities of household and beneficiary data maintained in Ration Card Management System (RCMS), distribution of commodities to beneficiaries at ePOS, and Supply Chain Management of commodities from Stock Keeping Units (SKUs) to Fair Price Shops(FPS). ● Implemented Real Time Governance System (RTGS) integrating data of different welfare departments based on household information of RCMS which provided a clear picture of how different welfare schemes applied to each household in the state. This helped to government plan their budget of each department and reduced over estimation. ● To create this RTGS data platform, to implement the datalake, used Alteryx to extract data from different source systems and ingest data into Azure blob storge. ● Created Databricks Notebooks to load data from the datalake into Databricks DBFS. In this process I worked on extracting data in different structured and semi-structured formats. ● Created and managed compute resources in Databricks to perform the transformations on extracted data. This operation optimized the cost of Databricks cloud resource. ● Transformed the delta tables using PySpark libraries and SQL following madallion architecture processing data from Bronze layer to Gold layer in Databricks and saved into on-prem OLAP systems ● Implemented data pipelines and orchestrated using Databricks Workflows. ● Implemented the data warehouse architecture following Kimball principles using start schema and took at-most care in implementing slowly changing dimensions. ● Defined a process for implementing ration card mutation services, which benefited millions of households to avail the benefits of welfare schemes. ● Optimized the performance of database system by finetuning the frequent running queries. This reduced the processing time to populate the dashboards. ● For the requirement of low latency dashboards having complex joins and aggregations, in-order to take the advantage of dynamic scalability of both compute and storage resources, architected cloud-based data warehouse solution in snowflake to perform near real time data analysis over household data. This provided a clear picture in near real-time analysis of distribution, the performance of dealers at FPS and also helped in identifying the migration patterns of beneficiaries. ● Led the migration on-premises PDS OLAP data solutions into Snowflake, ensuring zero data loss and improved scalability. ● Worked extensively in query profiles and partitioning to reduce the overlap of micro partitions, this improved the performance of the system. ● Created data visualizations to monitor the performance of FPS, SKUs, Allotment, ration card mutation services, beneficiary migration patterns and distribution using Tableau for better decision making. ● Forecasted the demand of essential commodities during festival distribution. This eliminated the last-minute allotment and reduced additional transportation challenges. ● Designed and led the implementation of Aadhar Data Vault in PDS environment to ensure data governance guidelines to implement PII policies for public data. ● Designed and developed data models for implementing doorstep delivery through Mobile Dispensing Units (MDUs) during Covid-19 to eliminate contact violations.
Data Engineer
National Informatics Centre• August 2011 - December 2017
Planning and Monitoring Department ● Architected on-premises data warehouse for integrating, transforming and monitoring budget information. ● Implemented batch data loads to calculate budget deductions, procurement balances. ● Led highly impactful on-premises data platforms traditional technologies. ● Designed and implemented inventory management system and integrated to budget monitoring system for monitoring vendor penalties. ● Highlights: implemented customized Resource Management data solution to pin picture the budget status for the department. ● Utilized PL/SQL procedures and views, Pentaho Data Integrator and Tableau.
Software Engineer
BirlaSoft• July 2007 - August 2009
Developed and implemented FinDashboard application using MVC- struts java to perform assert management
Education
State University of New York at Binghamton
Data Analytics, MS• August 2022 - July 2023
JNTU, Hyderabad (Jawaharlal Nehru Technological University)
Information Technology, B.Tech• September 2003 - May 2007
Skills
balaji_kottana05 has not updated skills details yet.