Data Engineer – Web Scraping & Data Pipelines

Data Engineer – Web Scraping & Data Pipelines$500 - $600HybridOur client is looking for a Data Engineer – Web Scraping & Data Pipelines to join their Data Engineering team. The role involves building and maintaining critical data pipelines that power AI-driven products for accounting professionals.Key ResponsibilitiesDesign, develop, and maintain web scrapers for alternative datasets such as regulations, tax laws, and financial dataClean, transform, and process data using Python (BeautifulSoup, lxml for parsing; Pandas for tabular data)Ingest and manage scraped data within databases and data warehousesSchedule and orchestrate scraping jobs using Airflow or similar toolsImplement quality control checks to ensure data accuracy and availabilityInvestigate and resolve data incidents to ensure smooth daily operationsCollaborate closely with engineering and product teams to define data requirements and optimize workflowsMust-Have QualificationsBachelor’s degree in Computer Science or a related field (or equivalent practical experience)1–3 years of software development experienceStrong Python skills with hands-on web scraping experience (BeautifulSoup, lxml, Scrapy)Solid SQL and database skills, including query writing and schema designExperience working with HTML, JavaScript, APIs, and modern web technologiesStrong experience in data cleaning and manipulation (Pandas or similar libraries)Daily user of AI tools such as ChatGPT, Claude, or other code assistantsGood communication skills and the ability to work independentlySelf-motivated with a strong interest in automation and problem-solvingPreferred QualificationsExperience with cloud platforms such as Azure or AWSFamiliarity with advanced scraping tools (Selenium, Playwright, XPath)Experience containerizing applications using DockerExposure to CI/CD pipelines (GitHub Actions, GitLab CI, Jenkins)Familiarity with data orchestration tools such as Airflow or DagsterIf you are interested, send your CV to careers@talentstudio.io