Notes

Builder Platforms Grow by Owning the Agent Loop

2026-07-18

Give coding agents a tested path into your platform, measure what works, and use the results to improve activation and retention.

When a developer asks a coding agent to build with your platform, that session becomes part of how they experience your product.

Our team learned this while running agent skills and evals across Google Maps Platform: if an agent picks the wrong API, misses an authentication step, or writes code that fails to compile, the developer doesn't blame the model: they experience that failure as your platform, and they leave.

The best model will change, and so will the agent harness wrapped around it. Don't anchor your platform's developer experience to either one. Own the feedback loop that keeps core developer tasks working as models and agents move around it.

The model is not the strategy

A coding agent pairs a base model with an execution harness. The harness controls instructions, tools, repository search, context, and stopping criteria. Change either component and the runtime behavior changes.

Models and harnesses evolve continuously. Your platform can still own four parts of the loop:

1. Context: current product knowledge, examples, constraints, and workflows.
2. Evals: representative tasks and a clear definition of success.
3. Distribution: portable integrations that put proven context into the agent environments developers already use.
4. Outcome measurement: attributed signals for task success, activation, retention, and expansion.

The loop: distribute tested context, measure activation, retention, and expansion, then review selected traces to improve the skills and evals.

Use evals to test your context. Distribution puts tested paths in developer hands, while outcome signals show where they break. After review, failures become new eval tasks and successes become better examples. Usage doesn't automatically become training data for a model company, and it shouldn't.

Why verifiers matter

Pretraining gives a model broad capability. Post-training shapes that capability for a job through human demonstrations and preferences or through verifiable rewards. Coding makes some rewards unusually concrete. A compiler can check the build, tests can check behavior, and static analysis can catch security and API errors. Reinforcement learning from verifiable rewards, or RLVR, trains directly against those checks.

The stack: pretraining builds broad capability. Post-training can use examples, preference rewards, or verifiable rewards. The runtime harness supplies current context, tools, and execution.

SWE-Gym packages real repository tasks with executable environments and tests. That's what a DevX team needs: a real job and a trustworthy check.

Start with deterministic checks

For each task, define the checks that can be objective:

• Does the project install and compile?
• Does it complete the task with the current API?
• Are credentials handled through the correct boundary?
• How much time, tool use, and token cost did the successful run require?

Add human judgment where the task needs it. For qualities like UI taste and system simplicity, start with a human-written rubric, and calibrate any model grader against repeated human review. A judge that agrees with itself isn't evidence that it agrees with developers.

The eval: deterministic checks and calibrated judgment lead to a ship, improve, or hold decision. Held-out tasks stay outside tuning.

Hold out a representative slice of tasks and answers from the context and tuning loop. Otherwise the system learns the test instead of the job. Recent audits of public coding benchmarks show how broken tasks and gameable tests can hollow out a familiar score. Audit the eval before blaming the model.

First-party signals close the product loop

Build a first-party measurement path, either through an owned surface or a partner harness with clear permission and privacy boundaries. The point is to govern what gets measured and shared, not to replace every coding agent.

Record the exact model, harness, context, execution environment, verifier, and run budget, then connect the run to a product outcome. A passing eval only proves the path can work. Activation proves the developer found value, and retention confirms they stayed. Expansion follows when the platform earns more of their work.

Don't collect raw source code or private conversations by default. Keep only the signals that can change a product decision, and review traces through an explicit privacy and security process.

Distribution makes the learning compound

Our team runs this loop across Google Maps Platform using portable agent skills and a task-based eval suite. When a new model ships, we run the evals, review the failure traces with product and engineering, and update the context.

Ship versioned context in a portable form, and run the same tasks in the tools, prompts, and execution boundaries developers receive. Treat every new model or harness as another row in the test matrix, not a reason to restart the strategy.

Open source the skills, examples, and selected evals you want agent builders and model teams to learn from, but keep a separate held-out set. Share the harness where useful, not every test case and answer.

Build one complete loop

Start with ten core developer tasks. For each one:

1. Write the task in the developer's language.
2. Define the deterministic checks and the small amount of calibrated judgment it needs.
3. Package the best current context as a versioned, portable skill or workflow.
4. Test it across representative model and harness versions.
5. Measure outcomes and review failures. Reuse traces only when they're authorized and cleared through privacy and security review.

Start small and let the loop compound. If you are building developer platforms for an agent-first world, how are you structuring your evaluation and distribution loops? Share your approach in the comments below.

Share LinkedIn Email

← All notes