Data Engineer (Data Collection Team)

As a Data Engineer in the Data Collection Team at Cube, you will leverage various methods to transform raw data into reliable and scalable data systems. You will focus on improving efficiency by aligning data architecture with business goals, while being hands-on in building data collection infrastructure. This role emphasizes automation, pipeline orchestration, and cloud-based solutions.

To succeed in this position, you should have prior experience in large-scale public data collection from the web using open APIs and other tools. A strong understanding of relevant terms of service, guidelines, and technical considerations governing such data collection is essential.

In addition, you should be familiar with multiple programming languages and have a basic understanding of machine learning methods. If you are detail-oriented, highly organized, and experienced in this field, we'd love to hear from you!

Responsibilities

• Build and maintain scalable data pipelines to process and integrate e-commerce data from multiple sources
• Develop production-ready Python scripts for API consumption, web scraping, and data transformation
• Design and implement cloud-based data architecture using AWS services such as S3, EC2, and Lambda
• Orchestrate and monitor scheduled data collection workflows using Apache Airflow
• Evaluate and implement tools for high-performance data transformation and analysis (e.g., Polars, DuckDB, PySpark)
• Collaborate with data analysts to deliver accessible, well-structured datasets for reporting and advanced analytics
• Identify opportunities to improve data quality, reliability, and performance across all data collection endpoints

What You'll Love About This Role

• Build from Scratch: Be part of a team creating foundational data systems and processes, shaping the future of our platform
• Learn by Doing: Gain hands-on experience with modern tools, cloud technologies, and real-world data challenges
• Work on Complex Projects: Tackle challenging problems from integrating large-scale e-commerce data to optimizing pipelines for performance and scalability

Requirements

• Proficiency in programming languages such as Java and Python
• Hands-on experience with SQL and database design
• Previous experience as a Data Engineer or in a similar role
• Strong technical knowledge of web data collection architectures, including proxy and header management
• Excellent numerical and analytical skills
• Willingness to learn and adapt to new tools and technologies
• Proactive, curious, and highly tenacious, with a strong drive to grow and stay ahead of industry trends
• Comfortable navigating a fast-paced, ambiguous environment with a high degree of independence
• A Bachelor's degree in a quantitative field (e.g., Computer Science, Engineering, Information Systems) is preferred