We are seeking a senior Web Scraping Engineer focused on automation and anti-bot techniques to build resilient data-collection systems. This role is critical for reliably extracting data from dynamic web applications while navigating anti-bot defenses and maintaining ethical, compliant practices.
The position offers deep hands-on work with headless browsers, proxy and CAPTCHA solutions, and cloud-deployed scraping architectures using JavaScript/Node.js, Python, and .NET. You will lead technical design, solve complex evasion challenges, and transfer knowledge to internal teams.
Responsibilities
- Design and implement web scraping and browser automation solutions using headless browsers.
- Develop strategies to handle dynamic content (AJAX, infinite scroll) and complex DOM interactions.
- Implement anti-bot evasion techniques, including CAPTCHA integration, fingerprint spoofing, IP rotation, and proxy management.
- Emulate human-like behavior (mouse movement, delays) to reduce detection.
- Build reliable authentication flows and manage secure credential storage and sessions (OAuth, SAML, MFA).
- Integrate and manage third-party CAPTCHA solvers and proxy services.
- Deploy and containerize scraping services for scalable execution in cloud functions or containers.
- Analyze network traffic and reverse-engineer web app interactions to identify stable data sources.
- Collaborate with stakeholders to transfer knowledge and ensure ethical, compliant scraping practices.
Requirements
- Senior-level experience in web scraping, browser automation, and anti-bot strategies.
- Proficiency with headless browsers and automation libraries: Puppeteer, Playwright, or Selenium.
- Strong skills in JavaScript/Node.js and Python; experience with .NET also required.
- Experience parsing and extracting content using BeautifulSoup, lxml, or Scrapy.
- Experience performing direct HTTP requests using requests or httpx.
- Practical experience with CAPTCHA solving workflows and services such as 2Captcha, Anti-Captcha, or CapMonster.
- Experience with proxy services and IP rotation strategies using providers like Bright Data, Oxylabs, or Smartproxy.
- Familiarity with deploying scraping workloads to cloud functions (AWS Lambda, Azure Functions) and containerizing with Docker.
- Deep understanding of authentication flows (OAuth, SAML, MFA), secure credential handling, and session management.
- Awareness of legal and ethical boundaries for scraping and familiarity with data privacy considerations (e.g., GDPR, CCPA).
Nice to Have
- Experience reverse-engineering web applications and analyzing API calls via browser dev tools.
- Experience with browser fingerprinting tools such as Multilogin.
- Familiarity with bot detection platforms like Cloudflare Bot Management or PerimeterX.
