Key Takeaways
- • AWS argues that the cost of agentic coding assistants should be benchmarked on a team's own repositories, since public leaderboards and provider benchmarks do not price private code.
- • In a sample run of 19 models over 21 tasks in one repository, the 27B open dense model delivered 18.8 quality points above the baseline for $0.60 more per task, and was described as the best value on the frontier.
- • Standardizing on the top closed frontier model cost $11.95 per task, roughly eight times the 27B option, for 2.7 more points than the 27B model.
- • Harness choice produced the larger and more consistent saving: one model cost about $8 per task under a lighter harness versus about $24 under a heavier, subagent-based harness, with no matching gain in quality.
- • AWS released open-source benchmarking code under MIT-0 at aws-samples/sample-agentic-coding-harness-benchmarks and a skill called swe-router that surfaces the cost-quality frontier to developers before each task.
Summary
AWS published guidance on measuring and managing the cost of agentic coding assistants such as Claude Code, Codex, Cursor, and Kiro. The post argues that model inference for coding has become a material budget line as usage scales across engineering organizations, and that the common default of selecting the strongest available model produces spend that was never an explicit decision. AWS recommends building a cost-and-quality frontier in each organization's own environment, since a provider benchmark or public leaderboard does not price a team's private code, and refreshing that frontier as new models are released or the mix of work changes.
The post explains why agentic workloads differ from conventional pricing assumptions. Non-agentic guidance assumes an input-to-output token ratio of around 5:1, whereas a coding agent replays a growing transcript on every turn across tens or hundreds of turns, which may push the ratio to a few hundred to one. AWS notes that output-token-based pricing therefore understates the bill for input-heavy work, and that a task running 98% input tokens is not inexpensive. For self-hosted GPUs, AWS proposes pricing runs by GPU-seconds, calculated as total processed tokens, including cache reads and writes, divided by measured throughput. The post cautions that throughput must be measured on the workload being priced, since cache-heavy runs process tokens faster than cold benchmarks.
AWS identifies three levers: the model, the harness (the coding agent that reads the repository, plans changes, and writes code), and the hosting location. In a sample run covering 19 models over 21 tasks in one repository, six models sat on the cost-quality frontier. An open MoE model with 35B total and 3B active parameters scored 59.2 at $0.26 per task, while an open dense 27B model scored 78.5 at $1.47. A closed frontier model scored 82.8 at $11.95 per task. AWS characterizes the 27B step as the best value on the frontier. In a separate comparison of 12 models under a lighter and a heavier harness, the lighter harness used fewer tokens, usually finished sooner, and retained the same score. The heavier harness spawned subagents that inflated token counts, cost, and wall-clock time.
On hosting, the post distinguishes three paths: native Amazon Bedrock per-token access for frontier models from labs such as OpenAI and Anthropic, Bedrock through a proxy to open-weight models via an OpenAI-compatible endpoint, and self-hosted vLLM on Amazon EC2 billed at a fixed hourly rate. AWS suggests starting with a managed per-token path and moving to self-hosted GPUs only once token volume makes fixed hardware cheaper. To deliver the findings to developers, AWS released swe-router, a skill that installs into the coding assistant, estimates the consequence of an error, sets a quality floor, and recommends the cheapest measured model that meets it. The post notes that the sample code is provided for demonstration and that users should review scripts and IAM permissions before running it in sensitive environments.
The post explains why agentic workloads differ from conventional pricing assumptions. Non-agentic guidance assumes an input-to-output token ratio of around 5:1, whereas a coding agent replays a growing transcript on every turn across tens or hundreds of turns, which may push the ratio to a few hundred to one. AWS notes that output-token-based pricing therefore understates the bill for input-heavy work, and that a task running 98% input tokens is not inexpensive. For self-hosted GPUs, AWS proposes pricing runs by GPU-seconds, calculated as total processed tokens, including cache reads and writes, divided by measured throughput. The post cautions that throughput must be measured on the workload being priced, since cache-heavy runs process tokens faster than cold benchmarks.
AWS identifies three levers: the model, the harness (the coding agent that reads the repository, plans changes, and writes code), and the hosting location. In a sample run covering 19 models over 21 tasks in one repository, six models sat on the cost-quality frontier. An open MoE model with 35B total and 3B active parameters scored 59.2 at $0.26 per task, while an open dense 27B model scored 78.5 at $1.47. A closed frontier model scored 82.8 at $11.95 per task. AWS characterizes the 27B step as the best value on the frontier. In a separate comparison of 12 models under a lighter and a heavier harness, the lighter harness used fewer tokens, usually finished sooner, and retained the same score. The heavier harness spawned subagents that inflated token counts, cost, and wall-clock time.
On hosting, the post distinguishes three paths: native Amazon Bedrock per-token access for frontier models from labs such as OpenAI and Anthropic, Bedrock through a proxy to open-weight models via an OpenAI-compatible endpoint, and self-hosted vLLM on Amazon EC2 billed at a fixed hourly rate. AWS suggests starting with a managed per-token path and moving to self-hosted GPUs only once token volume makes fixed hardware cheaper. To deliver the findings to developers, AWS released swe-router, a skill that installs into the coding assistant, estimates the consequence of an error, sets a quality floor, and recommends the cheapest measured model that meets it. The post notes that the sample code is provided for demonstration and that users should review scripts and IAM permissions before running it in sensitive environments.
Go deeper than the headline
Research every firm in this story, track the ones that matter and get the daily briefing — free for 7 days.
