Evaluations - Member of Technical Staff
- Location
- San Francisco, CA
- Employment
- Full Time
- Work model
- On-Site
- Level
- Staff
- Salary
- $200k – $400k/yr
- Posted
Skills
About this role
About the Company
Pilots don't train with real passengers. Actors don't rehearse with real audiences. Yet, the most consequential decisions in society are often pushed straight to production. Simile is changing that. We have built the first AI simulation of society, populated by generative agents based on real humans. Our research pioneered the field of AI-based simulation, proving it is possible to model human behavior with high accuracy. Today, we are developing a Foundation Model to predict human behavior in any situation, at any scale. We have raised $300M in total funding, with a $2B post-money valuation in our latest round. Our investors include Greenoaks, Index Ventures, Hanabi, A*, Bain Capital Ventures, Factory, CVS Health Ventures, and Definition.
About the Role
As a Member of Technical Staff, Model Evaluations at Simile, you will help define how simulations of human behavior should be tested and when their predictions are useful enough to guide real decisions. You will develop measurement methods, challenge existing assumptions, and turn evidence into priorities for modeling and product. This is an opportunity to help establish evaluation practices for an emerging field while building systems that people use. Evaluation at Simile brings together model evals, statistics, behavioral science, research methodology, product quality, and human judgment. Because our models simulate people, populations, markets, and groups, our evals must account for distributions, noisy human ground truth, uncertainty, qualitative outputs, and customer decisions. You will work with rich human data, including surveys, long-form interviews, customer studies, qualitative research, transactions, product interactions, and other real-world behavioral traces. Questions you could investigate include: How can we tell whether an interview elicits faithful information? When do simulated responses to interventions match observed behavior? Can we predict an individual’s future choices better than population baselines? You will turn questions like these into studies, working tools, and evidence that guides what we build. The core question for this role is simple: How do we know when a simulation of human behavior is good enough to trust? Areas we are especially hiring for We are building a team with complementary strengths. Depth in one of the four areas below can be a strong fit; we do not expect expertise in all four. We also welcome specialists and generalists in broader model and product evaluation, research engineering, and related fields. Your workstream will reflect your strengths and the team’s priorities. Human Data and Measurement: Build reliable systems for collecting and evaluating human data. Assess whether interviews and other collection methods elicit faithful, sufficiently rich accounts of people’s experiences and behavior, and whether leading questions or unsupported detail distort the resulting data. Design rubrics, expert review, and annotation workflows; measure data quality, validity, and rater agreement; and turn these into repeatable programs that can run at scale. Behavioral Experimentalist / Builder: Design and build creative experiments across diverse populations, settings, and behavioral tasks. Develop tools and collection workflows to run studies at scale, test whether simulations reproduce human responses to interventions, and assess how findings generalize beyond each study. Behavioral Measurement and Causal Inference: Develop scalable strategies to learn causal relationships from large, diverse behavioral datasets. Make identification assumptions explicit and test robustness through sensitivity analyses and validation. Use this evidence to evaluate whether simulations reproduce intervention effects and support defensible counterfactual predictions across populations and settings. Individual and Longitudinal Evaluation: Evaluate whether simulations capture stable differences between individuals and changes in their