Kajal Gaikwad

Data Engineer

IBM India Pvt Ltd•  January 2020 - Present

Designed and implemented real time data pipeline to process semi structure data by integrating near about 1 TB data per day from different sources using Kafka, Logstash and stored it in GPFS Designed and implemented Logstash HA (High Availability) script using shell script , Elasticsearch watcher. Built different types of visualization for real time data using Kibana Implemented script to convert NMON data to csv format using Python Implemented monitoring of data using Python Elasticsearch client Automated process of generating report by writing python script to provide real time count of the IPs from which we are getting data. Integrated data from multiple third party API using Logstash, Kafka. Worked on Anomaly Detection in Kibana. Implemented python script to create dummy data for given interval of time. Led team of 2 employees