Field Notes
Builder Platforms Grow by Owning the Agent Loop
Give coding agents a tested path into your platform, measure what works, and use the results to improve activation and retention.
The best model will change: and so will the agent wrapped around it. A builder platform cannot anchor its strategy to either one. It needs to own the feedback loop that keeps its critical developer journeys working as models and agents change.
This is a growth problem, not just an AI quality problem. When a developer asks an agent to build with your platform, that session becomes part of your activation funnel. I see it in the traces our team reviews: when the agent chooses the wrong API, misses an authentication step, or produces code that does not compile, the developer experiences that failure as your platform.
The model is not the strategy
A coding agent is a model plus a harness. The harness controls instructions, tools, repository search, context, and how the agent decides it is done. Change either and the behavior changes.
Models and harnesses will keep moving. Your platform can still own four parts of the loop:
- Context: current product knowledge, examples, constraints, and workflows.
- Evals: representative tasks and a clear definition of success.
- Distribution: portable integrations that put proven context into the agent environments developers already use.
- Outcome measurement: attributed signals for task success, activation, retention, and expansion.
The loop: distribute tested context, measure activation, retention, and expansion, then review selected traces to improve the skills and evals.
Evals test the context. Distribution puts the tested path in more hands, while outcome signals show where it still falls short. After review, some failures become new tasks and some successes become better examples. Usage does not automatically become training data for a model company, and it should not.
Why verifiers matter
Pretraining gives a model broad capability. Post-training turns that capability toward a job, through human demonstrations and preferences or through rewards, and coding makes some rewards unusually concrete: a compiler can check the build, tests can check specified behavior, and static analysis can catch some security and API errors. Reinforcement learning from verifiable rewards, or RLVR, trains directly against those checks.
The stack: pretraining builds broad capability. Post-training can use examples, preference rewards, or verifiable rewards. The runtime harness supplies current context, tools, and execution.
SWE-Gym packages real repository tasks with executable environments and tests. That is what a DevX team needs: a real job and a trustworthy check.
Start with deterministic checks
For each critical developer journey, define the checks that can be objective:
- Does the project install and compile?
- Does it complete the task with the current API?
- Are credentials handled through the correct boundary?
- How much time, tool use, and token cost did the successful run require?
Add human judgment where the task needs it. For qualities like UI taste and system simplicity, start with a human-written rubric, and calibrate any model grader against repeated human review. A judge that agrees with itself is not evidence that it agrees with developers.
The eval: deterministic checks and calibrated judgment lead to a ship, improve, or hold decision. Held-out tasks stay outside tuning.
Keep a meaningful slice of tasks and answers out of the context and tuning loop. Otherwise the system can learn the test instead of the job. Recent audits of public coding benchmarks show how broken tasks and gameable tests can hollow out a familiar score. Audit the eval before blaming the model.
First-party signals close the product loop
A builder platform needs a first-party measurement path, either through a surface it owns or a partner harness with clear permission and privacy boundaries. The point is to govern what gets measured and shared, not to replace every coding agent.
Record the exact model, harness, context, execution environment, verifier, and run budget, then connect the run to a product outcome. A passing eval only says the path can work. From there, activation marks first value; retention shows whether it lasted; expansion shows the platform earning more of the developer's work.
Do not collect raw source code or private conversations by default. Keep only the signals that can change a product decision, and review traces through an explicit privacy and security process.
Distribution makes the learning compound
Our team is running a version of this loop for Google Maps Platform through portable agent skills and a task-based eval suite. We use eval results to guide launch and distribution decisions, and we are still learning what works.
Ship versioned context in a portable form, and run the same tasks in the tools, prompts, and execution boundaries developers receive. Treat every new model or harness as another row in the test matrix, not a reason to restart the strategy.
Open source the skills, examples, and selected evals you want agent builders and model teams to learn from, but keep a separate held-out set. Share the harness where useful, not every test case and answer.
Build one complete loop
Start with ten critical developer journeys. For each one:
- Write the task in the developer's language.
- Define the deterministic checks and the small amount of calibrated judgment it needs.
- Package the best current context as a versioned, portable skill or workflow.
- Test it across representative model and harness versions.
- Measure outcomes and review failures. Reuse traces only when they are authorized and cleared through privacy and security review.
Run it again. Models and harnesses will move, and nobody has this figured out yet, including us. Keep learning, keep experimenting, keep shipping, and let the loop get better every time they do.
Discussion
Comments
Comments are GitHub Discussions rendered by giscus. Sign in with GitHub inside the widget to post or react.