Data Engineer

Position Responsibilities Work in a dedicated data reporting and analytics team building a world-class data platform to produce data-driven insights for Transact Campus and our clients Analyze, interpret and orchestrate complex data across disparate sources comprising unstructured, semi-structured and structured datasets in streaming and batch modes Design and develop real-time data pipelines using the latest Databricks and Delta Lake Azure cloud technologies Collaborate with data consumers (reporting, analysis, or data science) to provide metrics that meet their needs Contribute to standards for data producers streaming data into the Lakehouse Test commercial software products using both manual and automated testing processes Support the application lifecycle during QA, UAT, and post-release phases Comply with and contribute to consistent development guidelines (coding, change control, build, versioning) Participate in peer code reviewsRequired Skills Bachelor’s degree in Computer Science, IT or related field OR equivalent related work experience – preferably with a focus on Data Analytics 5+ years of experience in software enterprise-level Data Engineering Expertise in big data workloads Experience with data lakes and scale-out processing Relational database design and best practices Hands-on experience designing and developing Spark data pipelines Strong SQL and Python skills Understanding of cloud-based big data workloads Experience with ETL/ELT patterns, preferably using Databricks jobs Excellent technical documentation skills Experience with data lakes (HDFS, Azure Data Lake or AWS S3) Experience with source code management systems such as Git/TFS/SVN Experience working in Agile teams (Scrum, XP, Kanban) Ability to present ideas and insights to business stakeholders Fluency in written and spoken EnglishPreferred Skills Good understanding of Azure Data Services (Azure Databricks, Azure Data Factory, Azure Data Lake Gen 2) Experience with Databricks Delta Lake, Delta Sharing, and Delta Live Tables Experience with Spark Structured Streaming Experience with NoSQL databases Experience with Infrastructure as Code technologies such as Terraform or ARM Experience in Data Science and ML methodologies Experience with Azure services for streaming data (EventHub, EventGrid) Understanding of Data strategy including Data Governance and Data management