Technical Program Manager, AI Evaluation Specialist

About the Role

We’re hiring an AI Evaluation Specialist to strengthen how Chime governs, evaluates, and improves AI systems across Operations. As part of Speech Analytics, you will own the human-in-the-loop review processes that measure model accuracy, reliability, and alignment with Chime’s standards for quality and member trust.

Your work provides the trust layer that ensures models behave as expected — identifying gaps, failure modes, and opportunities for improvement. You’ll partner closely with Speech Analytics, Data teams, Enablement, and Model Owners to ensure AI systems operate safely and consistently in production.

The base salary offered for this role and level of experience will begin at $105,000 and up to $145,000. Full-time employees are also eligible for a bonus, competitive equity package, and benefits. The actual base salary offered may be higher, depending on your location, skills, qualifications, and experience.

In this role, you can expect to:

• Own the Human-in-the-Loop evaluation process for all AI models supporting Operations.
• Run recurring sampling and reviews to assess accuracy, consistency, and failure modes.
• Score, tag, and document cases where AI systems misclassify, hallucinate, skip steps, or generate incomplete outputs.
• Maintain structured rubrics and guidelines to ensure reviewer alignment and scoring consistency.
• Conduct deeper investigations into error patterns and root causes.
• Translate insights into recommendations for model owners and partner teams.
• Track and report key evaluation metrics such as accuracy, recall, coverage, and error types.
• Maintain thorough documentation for evaluation procedures, sampling logic, and scoring definitions.
• Collaborate with cross-functional teams to integrate evaluation findings into dashboards and tuning workflows.
• Support scaling governance processes and strengthening model-health standards across Operations.

To thrive in this role, you have:

• 3–5+ years in QA, evaluation, operational analytics, HITL programs, or model monitoring.
• Experience reviewing unstructured text and applying rubrics or scorecards.
• Understanding of how AI supports operations (classification, summarization, categorization, automation).
• Ability to identify patterns, edge cases, and failure modes from qualitative and quantitative data.
• Familiarity with QA frameworks or content-review workflows.
• Experience with SQL, Looker, Snowflake (nice to have).
• Strong attention to detail and high consistency standards.
• Clear communication and documentation skills.
• A passion for improving member experience by ensuring AI is safe, fair, and reliable.
• COPC or Lean Six Sigma experience is a plus.

#LI-Remote #LI-EI1