Data Scientist Job at Mercor, Remote

U2JhQ2E5WTFBN2FWbkFyUk9jNGhZd1E0aEE9PQ==
  • Mercor
  • Remote

Job Description

Job Description: AI Task Evaluation & Statistical Analysis Specialist

Role Overview

We're seeking a data-driven analyst to conduct comprehensive failure analysis on AI agent performance across finance-sector tasks. You'll identify patterns, root causes, and systemic issues in our evaluation framework by analyzing task performance across multiple dimensions (task types, file types, criteria, etc.).

Key Responsibilities

  • Statistical Failure Analysis : Identify patterns in AI agent failures across task components (prompts, rubrics, templates, file types, tags)

  • Root Cause Analysis : Determine whether failures stem from task design, rubric clarity, file complexity, or agent limitations

  • Dimension Analysis : Analyze performance variations across finance sub-domains, file types, and task categories

  • Reporting & Visualization : Create dashboards and reports highlighting failure clusters, edge cases, and improvement opportunities

  • Quality Framework : Recommend improvements to task design, rubric structure, and evaluation criteria based on statistical findings

  • Stakeholder Communication : Present insights to data labeling experts and technical teams

Required Qualifications

  • Statistical Expertise : Strong foundation in statistical analysis, hypothesis testing, and pattern recognition

  • Programming : Proficiency in Python (pandas, scipy, matplotlib/seaborn) or R for data analysis

  • Data Analysis : Experience with exploratory data analysis and creating actionable insights from complex datasets

  • AI/ML Familiarity : Understanding of LLM evaluation methods and quality metrics

  • Tools : Comfortable working with Excel, data visualization tools (Tableau/Looker), and SQL

Preferred Qualifications

  • Experience with AI/ML model evaluation or quality assurance

  • Background in finance or willingness to learn finance domain concepts

  • Experience with multi-dimensional failure analysis

  • Familiarity with benchmark datasets and evaluation frameworks

  • 2-4 years of relevant experience

Job Tags

Remote job, Full time,

Similar Jobs

Dentsu

Associate Project Manager Job at Dentsu

 ...Elevate the work. You turn ideas into well-run projectsbringing clarity, momentum, and...  ...About the Role As an Associate Project Manager at Amplifi North America, part of the dentsu...  ...retrospectives. Create and manage JIRA tickets, epics, and boards to track progress and provide... 

Careerscape

Content Writer Job at Careerscape

 ...blog posts, whitepapers, case studies, and website copy. The content team works with clients in software, cybersecurity, and cloud computing industries. The Content Writer will work remotely with flexibility to manage their own schedule while meeting weekly content... 

Halo Land Management

Document Specialist Job at Halo Land Management

 ...what success looks like in every role. This Document Specialist position is a key part of that...  ...compress, and annotate PDFs for internal review and external delivery. Build clean...  ...values both speed and precision. Remote flexibility with steady project volume and... 

ESR Healthcare

Industrial Maintenance Technician - 3rd Shift Bolingbrook, IL Job at ESR Healthcare

 ...Industrial Maintenance Technician - 3rd Shift Bolingbrook, IL Job Summary/Company: Our client is seeking a skilled Industrial Maintenance Technician to join our team in Bolingbrook, IL on third shift working Monday through Friday 11:00pm - 7:00am. As an Industrial... 

Business of Home

Staff Designer (Junior Designer) Job at Business of Home

Job Post Staff Designer (Junior Designer) Apply Company: Loczi Design Duration: Full-time...  ...often referred to as a Junior Designer in our industry) to join our team in San Francisco. This is not an entry-level position. This role is central to how our studio...