Senior Automation QA Engineer (Big Data Platform)

We are seeking a seasoned Automation QA Engineer with a strong focus on Big Data to play a pivotal role in modernizing a massive Enterprise Data Platform. You will be instrumental in migrating an existing on-premises HDFS/Hive ecosystem to a modern Cloud Object Storage solution.ResponsibilitiesMigration Support: Actively participate in the migration of the Enterprise Data Platform from HDFS/Hive to Cloud Storage.Pipeline Development & Validation: Implement and validate Spark-based ETL/ELT jobs (Scala/Python) to facilitate data movement to cloud storage.Automation: Develop and maintain robust automated tests and integration scripts, primarily utilizing Python.Data Integrity: Conduct functional testing and code validation to ensure the quality and stability of data pipelines.Verification: Test migration outcomes to confirm data accuracy, completeness, and reliability; perform critical data quality checks across systems.Analysis: Use SQL and analytics tools (e.g., Jupyter Notebooks, Apache Superset) to analyze and validate processed or migrated data and interpret performance metrics.Collaboration: Partner closely with Data Engineers to troubleshoot issues, validate fixes, and maintain exceptionally high data quality standards.RequirementsProven experience as an Automation QA Engineer, ideally within Big Data or large-scale data platform projects.Strong background in functional testing, code validation, and end-to-end data verification.Big Data Tech Stack: Hands-on experience with Apache Spark, Flink, HDFS, and a strong preference for experience with Apache Iceberg.Cloud Proficiency: Familiarity with AWS or equivalent major cloud environments.Coding: Proficiency in Python for writing and maintaining robust integration and automation tests.Solid understanding of data analytics—capable of analyzing results and validating transformations using SQL.Experience in validating the quality and consistency of data and tools used within pipelines.Excellent communication and collaboration skills within distributed team settings.We offerOpportunity to work on bleeding-edge projectsWork with a highly motivated and dedicated teamCompetitive salaryFlexible scheduleProfessional development opportunitiesAbout UsGrid Dynamics (NASDAQ: GDYN) is a leading provider of technology consulting, platform and product engineering, AI, and advanced analytics services. Fusing technical vision with business acumen, we solve the most pressing technical challenges and enable positive business outcomes for enterprise companies undergoing business transformation. A key differentiator for Grid Dynamics is our 8 years of experience and leadership in enterprise AI, supported by profound expertise and ongoing investment in data, analytics, cloud & DevOps, application modernization and customer experience. Founded in 2006, Grid Dynamics is headquartered in Silicon Valley with offices across the Americas, Europe, and India.