Dipal Prajapati

Technical Lead

Mobileum•  September 2018 - Present•  Ahmedabad, GUJARAT

Analysed and processed EDR data, including HTTP and flow data, from 12 distinct probes. Engineered a real-time data pipeline utilizing Kafka and Spark, capable of processing semi-structured data at a rate of 200K FPS. Built Daily and Monthly Spark aggregation jobs. Developed scalable modules in Go programming language to enhance platform performance. Designed Clickhouse database architecture and devised effective data modelling strategies for a 2*2 cluster configuration. Implemented logic for identifying device change events and new user entries, accommodating a user base of over 1 million. Implemented robust S3 read and write functionalities to optimize data management processes. Orchestrated a seamless migration to AWS cloud infrastructure within a one-week timeframe. Constructed a robust Data Ingestion pipeline using Talend to optimize efficiency. Executed fine-tuning of Spark streaming jobs to maximize performance and efficiency. Facilitated report generation in Tableau by creating tableau schema atop Clickhouse. Built the pipeline from RDS Aurora mysql to MSK for Change Data Capture (CDC) using Debezium MySQL Connector.

Software Developer

Infodesk India Pvt. Ltd•  December 2015 - September 2018•  Vadodara, GUJARAT

Designed various Spark Transformers. Involved in writing Spark transformers to transform one form of input to desired output. Processed XML files, splitting each sentence into words, and emitting word's lemmas. Updated business view model (metadata.xmi) in Pentaho. Created reports in Pentaho report designer and deploying them on BI-Server. Revamped ETL jobs and transformations in data integration tool Spoon to streamline data loading into the warehouse. Processed data on Apache Solr to improve search functionalities. Designed multithreaded tool in Spring Boot using Java and Solr JAPI to fetch. Wrote XSLT for transforming raw feeds to NewsML G2 XMLs for more than 200.

Software Engineer

Guavus Network Systems Pvt. Ltd•  June 2012 - October 2015•  Ahmedabad, GUJARAT

Worked on various aspects in multiple projects beginning with aggregation of raw data as required using java collections followed by designing web. Analysed of raw data & data aggregation. Designing and developing web services with Servlets or BlazeDS to interact with front-end and back-end. Optimizing data aggregation process, webservice performance. Introduced Caching mechanism and crafted optimized database queries to enhance performance. Tested End to End Workflow, Observing BlazeDS Code, Changing web. Provided onsite Support to team for deployment. Coding in Scala for processing Radius file. Implemented IP sessions stitching. Wrote Oozie jobs to schedule execution of Spark jobs. Achieved to have this project in production. Performed PoC version and production version. Implemented Servlets to communicate with Netezza (was used to excel queries on large data. Worked on Netezza, modifying tables and stored procedures.