Lead Data Engineer

Responsibilities:Enhance existing data pipelines using industry-standard tools and practices.Redesign and implement pipelines leveraging AWS managed components.Set up monitoring systems to proactively detect and alert on pipeline issues.Investigate and resolve pipeline-related issues.Document solutions and operational guidelines for data pipelines.Develop unit and integration testing for pipeline components.Requirements:Minimum 6 years of hands-on experience in data engineering, particularly designing and building ETL pipelines.Strong background in project execution and technical expertise.Proficiency in Python.Solid knowledge of SQL.Experience with shell scripting (Linux).Familiarity with orchestration tools such as Airflow or similar.Experience with AWS services including S3, Glue, EMR, Lambda, and Step Functions.Exposure to AWS Batch, Athena, file formats like Parquet, NoSQL databases, and tools like AWS OpenSearch.Ability to work onsite when requested by the project team.