AI/ML Data Engineer

Who we are and what we do

Contra Labs is a human-centered AI lab focused on creative and multimodal outputs, where human taste defines the next generation of AI capabilities.

We build industry-leading creative preference datasets that power benchmarking, evaluation, and post-training for the world's leading AI models and applications. Our work helps define what "good" looks like across design, video, imagery, and beyond.

Built on top of Contra, the professional network for independent creatives, Contra Labs connects frontier AI labs with a global network of top creative talent. Together, we turn human judgment into the training and evaluation layer that enables AI models and tools to power the next generation of human creativity.

Why this role exists

Contra Labs produces evaluation data that AI labs and creative tool companies use to make high-stakes decisions about their models and products. That data needs to be rigorous, defensible, and clearly communicated. This role owns the analytical layer: you take raw evaluation data from client projects and turn it into benchmark results, statistical analyses, leaderboard methodology, and the quantitative backbone of everything we deliver. You make our work credible.

What you'll do

• Own the quantitative output of Contra Labs: benchmark results, model evaluation reports, regression tracking, and Creative Arena leaderboards
• Run statistical analysis on human evaluation data, including inter-rater reliability, preference distributions, confidence intervals, and annotation quality metrics
• Build and maintain the analytical methodology behind the Human Creativity Benchmark (HCB) and client deliverables
• Produce data visualizations and research reports that become client-facing deliverables and public-facing publications
• Partner with the Special Project Lead to ensure project data is clean, analysis is sound, and deliverables are rigorous
• Feed insights back to GTM to support sales conversations, case studies, and positioning

About you

• Strong in Python, SQL, and statistical analysis. You've done real analytical work, not just dashboards
• You understand how AI models are evaluated: preference data, pairwise comparisons, ranking systems, evaluation benchmarks. You don't need to build models, but you need to understand what the data means
• Clear communicator: you can turn complex analysis into a clean chart, a tight summary, or a compelling slide
• High ownership, bias toward action. You ship analysis quickly, flag issues proactively, and iterate
• AI-native: you actively use AI tools to automate your own workflows and are always looking to eliminate manual work

Experience

• 3-5\+ years in data analysis, data science, research operations, or applied analytics, ideally in AI/ML, tech, or creative industries
• Proficiency in statistical methods: hypothesis testing, regression, sampling, inter-rater reliability, experimental design
• Hands-on experience producing analytical deliverables for external clients or stakeholders
• Familiarity with AI/ML evaluation concepts (preference data, RLHF, benchmarking) or strong willingness to learn

Bonus

• Experience working with human evaluation or annotation data at scale
• Background in research operations, ML evaluation, or benchmark design
• Experience at an AI lab, data labeling company, or research org (Scale, Surge, Anthropic, Google DeepMind, etc.)

Requirements

• Based in NYC, USA
• In-office 5 days/week (Williamsburg)

Total Comp

• Salary: $180,000-$220,000 USD \+ Equity
• Medical, Dental, Vision Benefits
• 401k Matching
• We will provide you with a company laptop on your start date

Interview Process

1. Interview with Recruiting Team (20 minutes)
2. Interview with CEO \& Co-Founder (30 minutes)
3. Interview with VP of Product (45 minutes)
4. Paid Case Study and Presentation (60 minutes)

Note: Contra communicates with applicants through @contra.com domains only. We never ask for money from potential employees. For the latest job postings, visit Contra Careers.