Job summary
Seeking a Senior Data Infrastructure Engineer to design and scale robust web scraping and data ingestion systems in a fully remote position based in Norway. The role focuses on building infrastructure that collects large-scale social media data daily while overcoming advanced anti-bot challenges to support analytical and AI workflows.
Location and work setup
- Location
- Norway
- Work setup
- Remote
Responsibilities
Lead the development and maintenance of scalable data collection systems ensuring reliable and efficient data ingestion. Design web scraping infrastructure utilizing residential proxies, TLS fingerprinting, browser automation, and session rotation. Build fault-tolerant, high-throughput scraping pipelines targeting platforms such as Instagram, TikTok, YouTube, and X. Develop distributed queues and workflow orchestration using tools like Temporal, Ray, Apache Kafka, and Celery. Architect data storage environments for structured and unstructured data with technologies including Parquet, Apache Iceberg, Snowflake, and Amazon S3. Implement monitoring, automated alerts, and recovery mechanisms to maintain platform reliability and minimize downtime. Enable downstream AI and analytics workflows through robust data infrastructure.
Qualifications
At least 4 years of experience in high-volume web scraping, data engineering, or reverse engineering. Strong expertise in handling advanced anti-bot protection (Cloudflare, Akamai, PerimeterX) via TLS impersonation, browser automation, and proxy management. Advanced programming skills in Python or Go. Proficient with browser automation and scraping frameworks such as Playwright, Puppeteer, Scrapy, or Selenium. Experience with distributed systems and task queues including Temporal, Ray, Kafka, and Redis. Strong SQL abilities and experience with modern analytical data lakes and warehouses. Ability to design scalable, fault-tolerant architectures and resolve complex data ingestion issues. Preferred experience includes extracting social media video content and user metadata, and integrating scraping results with vector databases and real-time AI processing.