Job summary
This freelance senior engineering role focuses on developing and managing sophisticated Python-based web scraping workflows to extract and process data from complex, dynamic websites. The position involves ensuring high data quality and reliability, adapting to evolving web technologies, and leveraging cloud and automation tools. Part-time hours are estimated at 10-20 per week during active project phases, compensated up to approximately $45 per hour based on expertise and contribution pace. The role is fully remote and requires strong English proficiency.
Location and work setup
- Location
- Norway
- Work setup
- Remote
Salary
USD 45.00–45.00 hour
Responsibilities
Lead complete data extraction workflows across intricate websites to deliver structured and accurate datasets. Utilize specialized scraping tools and custom methods to streamline data collection, validation, and delivery. Address challenges such as JavaScript-rendered content and dynamic site changes, implement data quality controls including consistency checks and formatting adherence, and scale operations efficiently using batching or parallelization techniques. Monitor system stability and handle anti-bot mechanisms effectively while maintaining reliability.
Qualifications
A minimum of five years experience in data engineering, web scraping, automation, or software development is required. Relevant academic credentials such as a Bachelor's or Master's degree in Engineering, Computer Science, Applied Mathematics, or related technical fields are advantageous. Proven technical expertise with Python scraping libraries (BeautifulSoup, Selenium), handling dynamic content, APIs with proxies, as well as skills in data cleaning, normalization, and validation are essential. Experience with cloud platforms (AWS), Docker containerization, and the application of large language model frameworks (LangChain, OpenRouter) in automation tasks is needed. Candidates must be detail-oriented, independent problem solvers with demonstrated English proficiency at an upper-intermediate level or higher.