EngineeringHybrid

Senior Software Engineer, AI Evals

Sentry · Hybrid

Apply

Last seen by MeritLog September 10, 2026Source: AshbySource version: ashby-public-job-posting-v1

MeritLog read this listing from Sentry's Ashby job board and last checked it on September 10, 2026.

Source: the employer's Ashby job board. Open the original listing for current details.

Job details

Work model: Hybrid
Salary: $155K - $400K
Location: San Francisco, California
Occupation: Software Developers(O*NET 15-1252.00)
Company website: blog.sentry.io

Hiring context

How this role compares at Sentry

Sentry has 41 live roles in MeritLog’s catalog across 6 job families, and 25 of them are in engineering. 41 of those listings publish a pay range, a disclosure rate of 100%.

This role's posted range of $155K - $400K sits above 50% of the 26 other Sentry roles quoted over the same currency and period.

Sentry concentrates this hiring in:

Counted across the job boards MeritLog tracks, at the time this page was served. Pay comparisons use only listings that publish a complete range in the same currency and period.

What the role asks for

What you'd do

Design and build robust evaluation frameworks to measure accuracy, reliability, regressions, and edge cases in AI systems
Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data
Build automated test harnesses and metrics pipelines to continuously evaluate models, prompts, and agentic workflows
Partner closely with applied AI engineers and product leaders to define what “good” looks like and translate it into measurable criteria
Own the evaluation lifecycle for major AI initiatives, from early experimentation through production monitoring
Care deeply about correctness, rigor, and measurement in AI systems
Enjoy turning fuzzy product goals and model behavior into concrete tests and metrics
Like building foundational infrastructure that unlocks faster iteration and higher confidence for the entire AI team
Thrive in cross-functional environments and enjoy influencing model design through better evaluation

What they're asking for

Minimum 5+ years of professional experience with a Bachelor’s degree in computer science, machine learning, or a related fieldEducation
Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred)SkillPreferred
Comfort writing production-quality code (we use Python and TypeScript)Skill
Experience working with structured and unstructured datasets, labeling workflows, or data quality pipelinesSkill
Familiarity with modern ML systems and evaluation techniques (e.g., offline metrics, online evaluation, regression testing for models or prompts)Skill
Bonus: experience evaluating LLMs, agentic systems, or AI-assisted developer toolsSkillPreferred

Parsed by MeritLog from the employer’s own posting. The full description follows below.

Job description

ABOUT SENTRY Software runs the world and the pace is faster than ever. Sentry helps developers fix errors and performance issues before users notice, so teams can spend less time firefighting and more time building. Trusted by 200,000+ organizations, Sentry is today’s application monitoring standard and our team is building its AI-native future. ABOUT THE ROLE As a Senior Software Engineer on Sentry’s AI/ML team, you’ll be responsible for building the evaluation infrastructure that measures the accuracy, reliability, and real-world performance of our AI systems. This role is critical to ensuring that our debugging agents and AI-powered features behave correctly, safely, and predictably as they scale. You’ll design datasets, benchmarks, and test harnesses that turn ambiguous AI behavior into measurable signals, helping the team ship AI with confidence. IN THIS ROLE YOU WILL - Design and build robust evaluation frameworks to measure accuracy, reliability, regressions, and edge cases in AI systems - Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data - Build automated test harnesses and metrics pipelines to continuously evaluate models, prompts, and agentic workflows - Partner closely with applied AI engineers and product leaders to define what “good” looks like and translate it into measurable criteria - Own the evaluation lifecycle for major AI initiatives, from early experimentation through production monitoring YOU’LL LOVE THIS JOB IF YOU - Care deeply about correctness, rigor, and measurement in AI systems - Enjoy turning fuzzy product goals and model behavior into concrete tests and metrics - Like building foundational infrastructure that unlocks faster iteration and higher confidence for the entire AI team - Thrive in cross-functional environments and enjoy influencing model design through better evaluation QUALIFICATIONS - Minimum 5+ years of professional experience with a Bachelor’s degree in computer science, machine learning, or a related field - Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred) - Comfort writing production-quality code (we use Python and TypeScript) - Experience working with structured and unstructured datasets, labeling workflows, or data quality pipelines - Familiarity with modern ML systems and evaluation techniques (e.g., offline metrics, online evaluation, regression testing for models or prompts) - Bonus: experience evaluating LLMs, agentic systems, or AI-assisted developer tools The base salary range (or hourly wage range, if applicable) that Sentry reasonably expects to pay for this position is $155,000 to $400,000 USD. A successful candidate’s actual base salary (or hourly wage) amount will be determined by a variety of relevant factors including, without limitation, the candidate’s work location, education, work and other relevant experience, skills, and job-related knowledge. A successful candidate will be eligible to participate in Sentry’s employee benefit plans/programs applicable to the candidate’s position (including incentive compensation, equity grants, paid time off, and group health insurance coverage). See Sentry Benefits https://sentry.io/careers/ for more details about the Company’s benefit plans/programs. EQUAL OPPORTUNITY AT SENTRY Sentry is committed to providing equal employment opportunities to its employees and candidates for employment regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, veteran status, or other legally-protected characteristic. This commitment includes the provision of reasonable accommodations to employees and candidates for employment with physical or mental disabilities who require such accommodations in order to (a) perform the essential functions of their jobs, or (b) seek employment with Sentry. We strive to build a diverse team, with an inclusive culture where every teammate can thrive. Sentry is an open-source company because we believe that everyone, everywhere, should have the ability and tools to make great software. Software should be accessible. That starts with making our industry accessible. If you need assistance or an accommodation due to a disability, you may contact us at accommodations@sentry.io. Want to learn more about how Sentry handles applicant data? Get the details in our Applicant Privacy Policy https://sentry.io/careers/applicantprivacy/.

Keep exploring

More Engineering roles

Search all jobs