Essential Duties and Responsibilities:

Responsible for systems analysis - Design, Coding, Unit Testing and other SDLC activities

• Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities

• Requirement gathering and understanding, Analysing and converting functional requirements into concrete technical tasks and able to provide reasonable effort estimates.

• Create Spark jobs for data transformation and aggregation.

• Produce unit tests for Spark transformations and helper methods.

• Design data processing pipelines

• Monitoring performance and advising any necessary infrastructure changes

• Work proactively, independently and with global teams to address project requirements, and articulate issues/challenges with enough lead time to address project delivery risks.

In depth Knowledge and Experience in the following areas:

• Experience with Apache Spark streaming and batch framework

• Experience in Azure cloud platform and Data Bricks

• Experience with Pyspark

• Spark query tuning and performance optimization

• Experience with various messaging systems, such as Kafka

• Manage cloud deployment & configuration of our clusters.

• Experience in Git