Data Engineer

The role requires a skilled Data Engineer with expertise in Apache Spark and distributed computation techniques. Candidates should possess advanced SQL optimization skills and a solid understanding of caching strategies. Familiarity with the Cloudera ecosystem will be advantageous for big data management. The ideal candidate will work independently, take initiative in solving technical problems, and demonstrate excellent communication skills to collaborate with both technical and non-technical teams.Main Responsibilities:Distributed Data Processing: Develop and optimize distributed data pipelines using Apache Spark, ensuring efficient processing of large datasets.Caching & Computation Optimization: Implement advanced caching techniques and optimize distributed computation workflows.Partition Key Join Optimization: Enhance query processing speed and overall system performance.SQL Query Optimization: Improve performance and scalability of SQL queries in distributed database systems.Cloudera Ecosystem: Manage and scale data workflows when applicable.Key Requirements:Expertise in Apache Spark.Advanced skills in SQL optimization.Thorough understanding of distributed computation techniques.Familiarity with the Cloudera ecosystem for managing big data.Strong problem-solving skills and initiative.Excellent communication skills.Nice to Have:Knowledge of CI/CD pipeline automation.Experience with version control and branching.Deployment automation, artifact management, and containerization.Interest in DevOps methodologies.Other Details:This role involves working in an Agile environment alongside a team of five developers. It includes participation in daily stand-ups and sprint planning. The position may require occasional DevOps responsibilities, encompassing tasks such as automation of deployment processes and managing containerized applications.