hero

Current opportunities at Zetta portfolio companies

Zetta Venture Partners
Zetta Venture Partners
45
companies
224
Jobs

Lead Big Data Engineer

skan.ai

skan.ai

Data Science
Bengaluru, Karnataka, India
Posted on Tuesday, August 22, 2023

Essential Duties and Responsibilities:

Responsible for systems analysis - Design, Coding, Unit Testing and other SDLC activities

• Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities

• Requirement gathering and understanding, Analysing and converting functional requirements into concrete technical tasks and able to provide reasonable effort estimates.

• Create Spark jobs for data transformation and aggregation.

• Produce unit tests for Spark transformations and helper methods.

• Design data processing pipelines

• Monitoring performance and advising any necessary infrastructure changes

• Work proactively, independently and with global teams to address project requirements, and articulate issues/challenges with enough lead time to address project delivery risks.

In depth Knowledge and Experience in the following areas:

• Experience with Apache Spark streaming and batch framework

• Experience in Azure cloud platform and Data Bricks

• Experience with Pyspark

• Spark query tuning and performance optimization

• Experience with various messaging systems, such as Kafka

• Manage cloud deployment & configuration of our clusters.

• Experience in Git