AI Product Manager, Insights

The Role

We are looking for a highly analytical and strategic thinker to take ownership of our model evaluation analysis and insight generation. Our analysis has established a high standard for deep-dive analysis of model evaluation. We need someone who can not only maintain this cadence but elevate it, turning raw result data into a roadmap for model improvement.

Responsibilities

• Own the creation of model evaluation from initial hypothesis, data scraping to final publication. 
• Go beyond aggregate metrics (e.g., "Accuracy is 85%"). deeply analyze why the model failed on the other 15%. Identify semantic patterns, edge cases, and systemic hallucinations in raw model outputs.
• Review raw data sets, meeting transcripts, and research notes to identify the "so what?" We need to turn these findings into a logical hierarchy 
• You will act as the bridge between the data and the narrative by structuring findings into a logical hierarchy where the most critical "hook" lands first, followed by the supporting evidence

Who You Are (Requirements)

• Experience: You have 5 - 10 years of experience in DS, ML, AI research and analysis
• Structured Thinker: You organize your writing logically.
• High Tolerance for Ambiguity: You can take a messy pile of notes and organize it into a coherent outline without needing your hand held.
• Executive Presence: You are comfortable interviewing senior leaders and pushing back when an "insight" isn't actually insightful.
• Cross Functionality: Be able to work cross functionally across ML researchers to clients.

Nice to Have

• Experience in Model Evaluation, ML Engineering or Technical Research.
• Experience designing or curating datasets (RLHF, SFT data)