Badges
Certifications
Work Experience
Software Development Engineer 2
Sigmoid Analytics•  March 2019 - Present
Working on building a data pipeline which will be used to perform ETL using Pyspark. Build a server-less pipeline using Python and AWS components like Lambda, API gateway, SNS, SQS, EMR cluster and Step functions which will run the ETL, training (ML models) and prediction jobs. The data pipeline which we are building here will be able to handle around 70-80k request in a day, given that each job will take around 35-40 mins of time.
Product Engineer
Stylumia Intelligence and Technologies Pvt Ltd•  July 2018 - February 2019
Build a large-scale end-to-end multi-threaded distributed web crawler from scratch single-handedly. The product is based on the micro-services architecture where the services talk to each other using REST API calls or Asynchronously using Pub/Sub services. Build a distributed job scheduler using Apache Kafka. Apart from that my work also includes automating the whole framework, writing shell scripts (to take backups, to automate some processes, to archive the data etc), writing unit test cases, integration testing of the whole product, product infrastructure set up and making system level improvements.
Software Engineer
Promptcloud Technologies Pvt Ltd•  August 2016 - May 2018
Working on a Large Distributed System in the domain of Web crawling. Worked on a new asynchronous version of our web crawling product. Writing complex web crawlers based on client’s requirement Worked with infrastructure team to optimise the performance ofexisting system by making the system highly available and scalable. My work at Promptcloud includes building distributed system, writing crawlers, writing shell scripts, writing unit test cases, working with git, Jenkins, fixing code issues, system level improvements.
Education
ozishan91 has not updated education details yet.
Links
Skills
ozishan91 has not updated skills details yet.