Data Engineer

Data Pipeline Development & ManagementDesign, build, and maintain robust data pipelines using AWS GlueImplement ETL/ELT processes for data ingestion from multiple sourcesOptimize data workflows for performance and scalabilityMonitor and troubleshoot data pipeline failures and performance issuesData Infrastructure & Engineering (with IT Department)Manage and optimize AWS Redshift data warehouse operationsConfigure and maintain data storage solutions (AWS S3, data lakes)Implement data partitioning, indexing, and compression strategiesSupport Infrastructure as Code (IaC) for data infrastructure deploymentCI/CD & DevOps for Data (with AWS partners and IT Department)Develop and maintain CI/CD pipelines for data workflows using GitLabImplement automated testing for data pipelines and data qualitySupport version control and deployment strategies for data assetsConfigure AWS Lambda functions for data processing automationMonitoring & Support (with IT Department)Set up monitoring and alerting for data pipeline healthProvide technical support for data-related issuesCollaborate with technical teams on data architecture requirementsOptimize query performance and database operationsDocumentation & Reporting (with IT Department)Document data pipeline architectures and technical specificationsMaintain runbooks and operational proceduresConduct monthly progress meetings (1 hour) to report on system healthTrack engineering tasks through SHIP-HATS JiraMaintain technical documentation on SHIP-HATS ConfluenceRequired Skills & Experience:Strong background in data engineering and data pipeline developmentProficiency in SQL, Python, and shell scriptingExtensive experience with AWS data services (Redshift, S3, Glue, Lambda, CloudWatch)Data warehouse design and optimization experienceStrong CI/CD pipeline knowledge (GitLab preferred)Infrastructure as Code (IaC) experience (Terraform, CloudFormation)Knowledge of data modeling and database design principlesStrong troubleshooting and performance optimization skills