AI Quality Specialist

Location: RemoteReports to: Head of EngineeringThe RoleWe're looking for a Machine Learning Quality Specialist to own the quality and performance of Tapi's AI-powered invoice extraction system. You'll create, test, and maintain the prompts that enable our AI to accurately extract critical information from invoices, ensuring our customers get reliable data every time.This role sits at the intersection of AI operations, quality assurance, and customer success. You'll combine analytical thinking with strong communication skills, working closely with our AI engineering team to optimise extraction accuracy and with customer success to resolve issues quickly.You'll be part of our platform team, directly impacting the quality and reliability of a core product feature that thousands of customers depend on daily.About TapiTapi is an AI-native SaaS platform reimagining how property maintenance is done. We bring property managers, tenants, and trades into one seamless experience, using AI to reduce admin, speed up workflows, and create better outcomes for everyone.We already support more than 450,000 properties across New Zealand and Australia, and we're just getting started. With global expansion ahead of us and an ambitious roadmap in front, this is a pivotal time to join.What You'll DoCreate and refine AI prompts for new invoice types and data extraction requirements across different suppliers and formats.Test prompt variations against real invoices to validate accuracy and identify edge cases before deployment.Monitor extraction quality through dashboards, identifying patterns in errors and opportunities for improvement.Maintain and update existing prompts, responding to customer issues and changing invoice formats.Triage customer issues related to invoice extraction, determining whether they require prompt updates, data fixes, or engineering support.Be available to work between 14:00 and 22:00 NZDT.Build and maintain monitoring dashboards that track extraction accuracy, error rates, and system performance.Use data analysis tools to investigate extraction issues and measure the impact of prompt changes.Document prompt changes, test results, and quality metrics to inform continuous improvement.What You'll BringExcellent command of written English with strong attention to detail and nuance.Analytical mindset with the ability to spot patterns, diagnose issues, and test hypotheses.Natural curiosity and eagerness to learn emergent AI technologies and best practices.Strong organisational skills with ability to manage multiple priorities and customer issues simultaneously.Clear communicator who can explain technical concepts to non-technical stakeholders.Comfortable working with data and metrics to drive decisions.Experience with SQL for data analysis (highly desirable but not required).Familiarity with observability and monitoring tools (highly desirable but not required).What Great Looks LikeFocus AreaMeasure of SuccessExtraction Quality - High accuracy rates maintained across all invoice types and suppliersIssue Resolution - Customer extraction issues resolved within agreed timeframesProactive Monitoring - Issues identified and addressed before customers report themPrompt Performance - Regular testing and optimisation showing measurable improvementsSystem Reliability - Dashboards and monitors maintained, alerts configured and responsiveKnowledge Growth - Demonstrable upskilling in AI/ML technologies and prompt engineering techniquesA 10/10 Candidate WillTake full ownership of extraction quality, treating every inaccuracy as an opportunity to improve.Be naturally curious, staying ahead of AI developments and bringing new ideas to the team.Think like a detective, methodically investigating issues and testing solutions.Communicate clearly with engineering and customer success, keeping everyone informed.Balance speed and quality, knowing when to iterate quickly and when to test thoroughly.Champion the customer experience, understanding that extraction accuracy directly impacts their workflows.Why Join TapiBe part of a SaaS company reinventing property maintenance.Work with a talented, supportive team that moves fast and values ownership.Work remotely with flexibility and autonomy.Make a real impact by ensuring the quality of AI features that thousands of customers rely on daily.