Lead Big Data Engineer
Essential Duties and Responsibilities:
Responsible for systems analysis - Design, Coding, Unit Testing and other SDLC activities
• Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities
• Requirement gathering and understanding, Analysing and converting functional requirements into concrete technical tasks and able to provide reasonable effort estimates.
• Create Spark jobs for data transformation and aggregation.
• Produce unit tests for Spark transformations and helper methods.
• Design data processing pipelines
• Monitoring performance and advising any necessary infrastructure changes
• Work proactively, independently and with global teams to address project requirements, and articulate issues/challenges with enough lead time to address project delivery risks.
In depth Knowledge and Experience in the following areas:
• Experience with Apache Spark streaming and batch framework
• Experience in Azure cloud platform and Data Bricks
• Experience with Pyspark
• Spark query tuning and performance optimization
• Experience with various messaging systems, such as Kafka
• Manage cloud deployment & configuration of our clusters.
• Experience in Git