Python Software Engineer, Web Scraping Expert

Experience

I'm a Python/Golang Software Engineer and web scraping expert. I have deep experience in developing web applications using Django framework. Also, I have 3 years of web scraping & data processing experience. In my projects, I used Scrapy framework and Scrapyd to deploy my spiders. I scraped a lot of websites which using antibots, Cloudflare and complex JS. For more productive data processing, I used the technology of de-coupling processes and asynchrony. As a Go developer, I built highly loaded projects using the microservice architecture

Skills

Python Scrapy Django web scraping Golang Docker asyncio WebSockets REST API PostgreSQL Scrapyd Data Scraping & Processing Selenium Redis Kafka Linux XPath

Highlights

As a web developer (Python/Django):
- creating complex web applications using such technologies as celery, redis, nats, kafka, jwt auth, and other
- TDD approach
As web scraping specialist:
- Creating spiders to crawling sites of any* complexity
- Bypass captcha, bypass Cloudflare protection, NOT selenium approach to sites with a complex Js, use rotating proxies
- creating pipelines for the collected data (checking for duplicates, format of fields, etc.)
As Golang/Go developer:
- creation of high-load projects using the microservice architecture
- processing large amounts of data through web sockets api

Looking for

At the moment I am interested in working with web scraping and data processing