Work
Agentic Eval Suite
Task-based evals that benchmark AI context products against a no-context baseline and inform launch and roadmap decisions.
The goal
"The demo looked good" is not a launch bar. We needed objective answers to two questions. Does our context actually make agents better at building with us? And when a developer asks an agent to build something in our category, how do we do?
What shipped
My team and I built task-based evals for Google Maps Platform. We benchmark launches against a no-context baseline and use the delta to inform launch and roadmap decisions. The same tasks let us compare retrieval, skills, and agent integrations against one developer job.
What I learned
Measurement turns developer experience investments into decisions. Once an evaluated change has a baseline and a delta, the team can put the effort where the result moves. The operating method starts with real tasks, not polished demos.