AQA Engineer with LLM Testing experience

The company is a pioneering AI-driven research company providing in-depth stock analysis globally in multiple languages. Our mission is to bridge the knowledge gap in the investment world, empowering investors with AI-powered insights. By integrating our advanced AI technology into brokerage platforms and financial institutions, we enable millions of investors to make informed investment decisions.We are looking for a Mid-level Automation QA Engineer to join our team in Poland. The role will primarily focus on API testing (80%) with UI testing (20%), alongside testing LLM-powered AI solutions. You will help design, enhance, and maintain automation frameworks, ensuring efficient and scalable testing—especially for AI/LLM-driven features.Key Responsibilities:Automation Framework & StrategyContribute to designing and improving automation testing strategies.Develop and maintain automation frameworks for API, UI, and LLM validation.Ensure scalability, maintainability, and reusability of automation code.Test Development & ExecutionAPI Testing: Develop automated API tests in Python using the Requests library.UI Testing: Create and maintain UI tests with Playwright.LLM Testing: Design and execute automated test suites for **Large Language Model (LLM)**based features, including prompt validation, output accuracy, and bias/error detection.Integrate automated tests with Postman collections where applicable.Execute performance, regression, and functional tests.Continuous Integration & DeploymentIntegrate automation tests into CI/CD pipelines (Jenkins preferred).Work with Docker and Kubernetes to manage and scale test environments.Collaboration & DocumentationCollaborate with developers, QA, and DevOps teams to improve the QA process.Write test plans, strategies, and design documents.Track issues using Jira and manage test cases (Xray optional).Analyze logs, investigate failures, and troubleshoot in staging and production.Requirements:5 years of experience in test automation or a related QA role.Strong programming skills in Python.Solid experience in API automation (Requests library) and UI automation (Playwright).Hands-on experience testing LLM-based applications (prompt engineering, output evaluation, hallucination/bias testing).Experience with CI/CD tools (Jenkins preferred).Experience with Docker & Kubernetes for test environments.Strong analytical, problem-solving, and debugging skills.Experience with Jira for tracking and reporting.Ability to work independently and proactively in a fast-paced environment.Nice to Have:Experience with Kafka.Familiarity with Xray for test management.Knowledge of AWS cloud services.Experience with ML/AI model evaluation metrics.