AI product eval/ applied LLM eval/Human-in-the-Loop Evaluation / Annotation Advisor(1-3hr pre week/potential co founder)
Looking for a Part-Time Evaluation Advisor (human annotation/llm eval) The direction method is already validated by multiple ai leaders experts and ux researcher, there’s early buyer interest /pre pilot from big tech ai team , and the immediate goal is to turn the current build into a small, stable, pilot-ready product. We’re building an early evaluation and workflow product for AI teams. The current focus is helping teams structure and operationalize real-world failure cases, especially in scenarios where an assistant recommends too early, becomes overconfident, or fails to verify what matters before responding. The short-term wedge is a lightweight regression and review workflow. The longer-term opportunity is much bigger: infrastructure for how AI systems are tested, reviewed, and controlled in production decision flows. Looking for someone who can help with: • human annotation design • label guideline writing • evaluation schema • translating Best fit: someone with experience in human eval, annotation design, ranking/review quality, AI eval, or related areas. Bonus if you’ve worked on shopping, marketplace, trust, search, ranking, or agent systems. • Part-time / advisor to start. Best fit: someone with experience in human eval, annotation design, ranking/review quality, AI eval, or related areas. Bonus if you’ve worked on shopping, marketplace, trust, search, ranking, or agent systems. • Part-time / advisor to start. offer equity or cash