Badges
Certifications
Work Experience
Data Engineer
8bit• August 2022 - Present
Building and maintaining data pipelines with ETL processes, performing API testing, designing data warehouse solutions. Successfully optimized data pipelines, reducing processing time by up to 50%, by reducing inefficiencies in the data pipelines and using caching mechanisms. Completed a data migration project which involved sourcing and switching data providers, constructing database, and transferring and transforming data from legacy systems to a new database (Postgres). This project resulted in more than 50% of savings compared to the legacy system. Conducted a data integration project which involved setting up data pipeline using data integration platform (Hevo) to load data from Postgres to Google Bigquery. Designed and constructed pipeline from scratch to load hourly stock price data to Google Bigquery, which involved sourcing of API, extracting and transforming data, and loading it by batches into Google Bigquery in the right format for the quant team. This pipeline is orchestrated using Google Cloud Task and Google Cloud Scheduler. Set up CI/CD for data pipeline using CircleCI. Collaborated with software developers team to identify and resolve data-related bugs and issues, as well as conduct documentation to improve quality in the production process. Cross-collaborated with content and design team to set up AI prompts to generate contents and images for Goodwhale app's lesson modules (Quest) feature.
Education
Universiti Tunku Abdul Rahman
Biotechnology, BS• October 2011 - September 2014
Links
Skills
chongxe19911 has not updated skills details yet.