AI Research Scientist - Evaluation, Handshake AI

Your impactHandshake is building the future of human data for AI.We partner directly with top AI labs to power large language model (LLM) training and evaluation with high-quality, expert-generated data. As AI models become more sophisticated, the demand for specialized human input continues to grow—and Handshake is uniquely positioned to meet it. We power career platforms at 92% of the top U.S. universities, giving us direct access to verified expert talent across a wide range of domains.Our AI team is rapidly building a new generation of human data products—from expert annotation platforms to AI interviewers and seamless payout infrastructure—all designed to accelerate research and improve model performance.We’ve assembled a world-class team from YC, Notion, Scale, Coinbase, Palantir, and more, and we’re working directly with many of the world’s leading AI research labs. This is a unique opportunity to join a fast-growing team shaping the future of AI through better data, better tools, and better systems—for experts, by experts.We’re seeking Research Scientists to join our Handshake AI Research team, where you’ll conduct pioneering research that pushes the boundaries of LLM understanding and evaluation. You’ll work at the frontiers of model comprehension, advanced evaluation methodologies, and the intersection of human and AI knowledge systems, with the goal of building the future of how we understand and measure AI capabilities.As a Research Scientist, you’ll collaborate closely with engineers and cross-functional teams to transform fundamental insights into breakthrough evaluation frameworks and understanding paradigms. Whether you’re developing novel approaches to probe model understanding, creating sophisticated benchmarks that reveal emergent capabilities, or establishing new methodologies for measuring AI-human knowledge alignment; you’ll help define how we comprehend and assess the next generation of AI systems.Location: San Francisco or New York CityYour roleDesign and conduct original research in LLM understanding, evaluation methodologies, and the dynamics of human-AI knowledge interactionDevelop novel evaluation frameworks and assessment techniques that reveal deep insights into model capabilities and limitationsCollaborate with engineers to transform research breakthroughs into scalable benchmarks and evaluation systemsPioneer new approaches to measuring model understanding, reasoning capabilities, and alignment with human knowledgeWrite high-quality code to support large-scale experimentation, evaluation, and knowledge assessment workflowsPublish findings in top-tier conferences and contribute to advancing the field’s understanding of AI capabilitiesWork with cross-functional teams to establish new standards for responsible AI evaluation and knowledge alignmentYour experiencePhD or equivalent research experience in machine learning, computer science, cognitive science, or a related field with focus on AI evaluation or understandingStrong background in LLM research, model evaluation methodologies, interpretability, or foundational AI assessment techniquesDemonstrated ability to independently lead post training and evaluation research projects from theoretical framework to empirical validationProficiency in Python and deep experience with PyTorch for large-scale model analysis and evaluationExperience designing and conducting experiments with large language models, benchmark development, or systematic model assessmentStrong publication record in post training, AI evaluation, model understanding, interpretability, or related areas that advance our comprehension of AI capabilitiesAbility to clearly communicate complex insights about model behavior, evaluation methodologies, and their implications for AI developmentNice to HaveExperience with RL, agent modeling, or AI alignmentFamiliarity with data-centric AI approaches, synthetic data generation, or human-in-the-loop systemsUnderstanding of the challenges in scaling foundation models (e.g., training stability, safety, inference efficiency)Contributions to open-source AI libraries or research toolingInterest in shaping the societal impact, deployment ethics, and governance of frontier modelsWhy Join UsThis is a rare opportunity to help define how the world’s top labs build, test, and evaluate cutting-edge AI systems. You’ll be working with a uniquely high-talent team, tapping into a network of 18 million students and 500K+ PhDs, and shaping foundational infrastructure at a critical moment in the field. If you're excited to build from first principles—and want your work to directly accelerate frontier AI—we'd love to talk.What we offerAt Handshake, we'll give you the tools to feel healthy, happy and secure.Benefits below apply to US employees in full-time positions. Equity and ownership in a fast-growing company. 16 Weeks of paid parental leave for birth giving parents & 10 weeks of paid parental leave for non-birth giving parents. Comprehensive medical, dental, and vision policies including LGTBQ+ Coverage. We also provide resources for Mental Health Assistance, Employee Assistance Programs and counseling support. Generous learning & development opportunities and an annual $2,000 stipend for you to grow your skills and career. Financial coaching through Origin to help you through your financial journey. Monthly internet stipend and a brand new MacBook to allow you to do your best work. Monthly commuter stipend for you to expense your travel to the office (for office-based employees). Free lunch provided 5x a week in office. ️ Free gym access in San Francisco office building. Referral bonus to reward you when you bring great talent to Handshake. Team outings throughout the year to stay connected to each other. 401k Match: Handshake offers a dollar-for-dollar match on 1% of deferred salary, up to a maximum of $1,200 per year. All full-time US-based Handshakers are eligible for our flexible time off policy to get out and see the world. In addition, we offer 13 standardized holidays, and 2 additional days of flexible holiday time off. Lastly, we have a Winter #ShakeBreak, a one-week period of Collective Time Off. Handshake offers $500 home office stipend for you to spend during your first 3 months to create a productive and comfortable workspace at home. Family support: Parental leave coaching and support provided by Parentaly. We partner with Maven Clinic to provide a lifetime coverage up to $15K for expenses related to fertility and family forming! Lifestyle Savings Account: We offer you an annual stipend of $500 to use for purchases such as fitness classes, gym memberships, work-from-home setup, and more.Looking for more? Explore our mission, values and comprehensive US benefits at joinhandshake.com/careers.Handshake is committed to providing reasonable accommodations in our recruitment processes for candidates with disabilities, sincerely held religious beliefs or other reasons protected by applicable laws. If you need assistance or reasonable accommodation, please let your recruiter know during initial communications.

Apply Now