Senior Big Data Engineer

We are seeking a highly skilled and experienced Senior Big Data Engineer to join an exciting project focused on rearchitecting an on-premises Ruby application, ETL pipelines, and underlying data storage and reports to a cloud-native AWS environment.Our client is a world-renowned technology company from Silicon Valley, focused on revolutionizing data systems. You will work with world-class experts and technologies in the heart of Silicon Valley, developing products and services that billions of people use every day. ResponsibilitiesDesign and implement scalable ETL pipelines in a cloud-native AWS environmentDevelop and maintain data processing workflows using Python, Spark, and AirflowOptimize data storage solutions and implement advanced data management strategies using AWS and RedshiftEnsure the platform meets non-functional requirements by defining robust testing strategies and execution plansCollaborate with cross-functional teams to create and maintain decision and change logs, ensuring consistent communication and tracking of technical decisionsProactively identify opportunities to enhance the quality of the product while managing scope effectivelyMonitor and fine-tune data pipelines for performance, scalability, and maintainabilityProvide technical guidance and mentorship to team members, promoting best practices in data engineeringTroubleshoot and resolve complex issues in data workflows and pipelines Requirements3+ years of experience working in Big Data engineering rolesStrong data engineering and overall engineering skills, including expertise in testing and quality assuranceProficiency in PythonBackground in Spark, Airflow, and AWS technologies like RedshiftProven experience in decision-making processes with the ability to document and maintain decision and change logsUnderstanding of strategies to meet non-functional requirements such as performance, scalability, and testingCapability to design solutions that enhance product quality without significantly increasing scopeExcellent command of English (B2+ level), both written and spoken Nice to haveKnowledge of Ruby, Iceberg, Trino, Cassandra, and Tableau We offerInternational projects with top brandsWork with global teams of highly skilled, diverse peersHealthcare benefitsEmployee financial programsPaid time off and sick leaveUpskilling, reskilling and certification coursesUnlimited access to the LinkedIn Learning library and 22,000+ coursesGlobal career opportunitiesVolunteer and community involvement opportunitiesEPAM Employee GroupsAward-winning culture recognized by Glassdoor, Newsweek and LinkedIn