1. All jobs
  2. Data and Analytics
  3. Data Engineer

Web Crawling/scraping Engineer (5+), Google, Twitter platforms to $5000

IOIX Ukraine

Bypassing Google CAPTCHA and Twitter's hidden bans for data collection through crawling the search results on these platforms.

 

Preference will be given to candidates with ready-made solutions who can demonstrate their work. Respond to this job offer only if you comply.

 

Work Experience: Minimum of 5 years of experience in web crawling and automation.

 

Key responsibilities include collecting statistics on various types of blocks and developing automated tests to check the responses of websites. The specialist also works on automating data structure management, such as user profiles.

 

An important part of the job is simulating user behavior on social networks, such as Google and other sites, including the use of artificial intelligence. This specialist should not just be a consultant-analyst but also possess the ability to independently gather and analyze data, rather than just requesting prepared information about the frequency of CAPTCHA blocks and the types of CAPTCHAs shown to different users.

 

Candidate Requirements:

 

Experience:

  • Proven experience in developing web scrapers and crawlers for collecting data from complex web applications (5+ years).
  • Deep understanding of the principles of HTTP, HTML, CSS, and JavaScript.
  • Experience with HTML parsing tools (Beautiful Soup, lxml, etc.).
  • Experience with browser automation tools (Selenium, Puppeteer).
  • Experience in bypassing blocks and CAPTCHAs.

  • Experience with proxy servers and VPNs.

     

Skills:

  • Ability to analyze the structure of web pages and APIs.
  • Ability to develop effective and reliable blocking bypass algorithms.
  • Ability to work with asynchronous requests.
  • Ability to write clean, maintainable, and well-documented code.

 

📊 Average salary range of similar jobs in analytics →