When an AI agent writes the code, does an application framework help it or get in its way? The question matters commercially: engineering leaders are being asked to adopt agentic development and reduce delivery cost at the same time, and the tooling decisions made now will shape codebases for years.
We ran a controlled experiment to answer it with measurements rather than opinion. One agent, one frozen task, two stacks: Jmix with its framework skill pack, and bare Spring Boot with generic instructions. Three scored runs each. The results were not the ones we expected, and one of them is unfavourable to the framework.
Summary of Findings
Cost was identical. Token consumption and dollar cost were statistically indistinguishable between the two stacks. Variation between runs of the same stack exceeded the gap between stacks. Any claim that a framework makes agentic development cheaper is not supported by this data.
Defect patterns were not identical. Two of three framework runs passed acceptance with no findings. Zero of three bare-stack runs did. One specific class of defect, incorrect authorization rules, appeared in every bare-stack run.
Agents cannot self-verify permissions. In both arms, the agent tested its own work while authenticated as an administrator, which makes role-specific defects structurally invisible. This is an operational finding independent of stack choice.
Output was more predictable on the framework. Code volume varied by 7% across framework runs and by 24% across bare-stack runs.
Method
The task. A loan application pipeline with four entities, CRUD screens, two roles with distinct permissions, deterministic scoring by an exact formula, localized labels and persistent data. The specification was frozen before any run and was byte-identical for both stacks. Technology choices within each stack were left to the agent.
The environment. GLM-5.2 running in Codex CLI, provider pinned to a single first-party endpoint for every scored run. Java 21, Gradle 8.14, HSQLDB with Liquibase on both sides. The bare-stack arm received a generic instruction file; the framework arm additionally received a skill pack of 20 framework-specific instructions covering entities, views, roles and migrations.
Acceptance. Each run was evaluated against a seven-point checklist plus a hidden scoring oracle of twelve frozen scenarios executed through the application's own service layer. Findings were issued in a single batch, then re-verified after correction. Role permissions were verified by authenticating as each non-administrative role separately.
Scale. Three scored runs per stack, thirteen runs in total including pilots and calibration. Total cost of the benchmark: approximately $45.
Results
Cost and Acceptance
Cost ranges overlap entirely. The distinguishing variable is the number of findings at acceptance, not the price of generation.
| Stack | Run | Cost | Tokens | Findings | Outcome |
|---|---|---|---|---|---|
| Bare Spring | 1 | $3.14 | 10.2M | 2 | accepted after corrections |
| Bare Spring | 2 | $3.90 | 12.4M | 3 | accepted after corrections |
| Bare Spring | 3 | $3.94 | 13.3M | 3 | accepted after corrections |
| Jmix | 1 | $3.28 | 11.1M | 0 | accepted first pass |
| Jmix | 2 | $3.71 | 12.0M | 0 | accepted first pass |
| Jmix | 3 | $4.02 | 12.8M | 1 | accepted after correction |
Code Volume
Application sources only, tests and tooling excluded, median of three runs:
| Metric | Jmix | Bare Spring |
|---|---|---|
| Total lines | 1,826 | 2,304 (+26%) |
| Variation across runs | 7% | 24% |
| Frontend code | 0 | 524 to 858 |
The volume difference is secondary. The variation figure is the operationally significant one: on the framework, three independent runs converged on comparable output, which is what makes estimation possible.
Analysis
Where Wrong Guesses Surface
Neither arm had reliable knowledge of its stack. Both produced incorrect API usage. The stacks differed in when that became visible.
On the framework, incorrect usage typically failed to compile, producing an immediate correction loop that cannot be skipped. On the bare stack, incorrect usage compiled successfully and failed in runtime behaviour, most often in security configuration where a rule matched nothing or a filter chain was ordered incorrectly. Runtime failures of this kind are detected by testing, if the testing happens to cover them.
Authorization as a Recurring Defect Class
Authorization defects appeared in all four bare-stack runs, scored and pilot. The pattern was consistent: permissions expressed as URL patterns with an overly permissive fallback. In one run a user in the credit officer role could create and delete client records over REST while the user interface correctly hid the corresponding controls, producing an application that appeared correct in the browser and was not.
The security configuration file was rewritten as many as fourteen times within a single run, as the agent reshaped a structure that nothing in the stack constrained.
The framework arm expresses the same rules as declarative role definitions validated by the framework itself. Two of three runs produced no authorization findings.
The general principle is worth stating separately, because it applies beyond this benchmark: a mechanism that enforces an invariant provides a guarantee; an instruction that recommends one only reduces probability. We verified this within our own skill pack, where the instruction that closed a defect by construction eliminated it in all three runs, while the instruction that merely advised against a pattern saw that pattern recur in one run of three.
The Limits of Agent Self-Verification
Every agent in this benchmark tested its own work and reported success. Every agent performed that testing while authenticated as an administrator, because that was the credential available to it. Role-specific authorization defects cannot be detected from an administrative session.
For teams deploying agents on production work, the operational conclusion is direct: agent self-verification does not cover authorization, and cannot be made to cover it without an explicit multi-role test procedure. This finding is independent of framework choice.
Threats to Validity
We are the vendor of one of the two stacks under test, and we state the limitations accordingly.
Sample size. Three scored runs per arm. Differences of a single finding are not statistically meaningful and no conclusions are drawn from them.
Confounded variable. In practice the skill pack is inseparable from the framework and its project skeleton. The comparison is therefore "framework plus skeleton plus skill pack" against "bare Spring Boot plus generic instructions," not a domain skill pack evaluated in isolation.
Task shape. A CRUD application with role-based access control is a workload frameworks are designed for. A different domain could produce a different result.
Authorship. We wrote the skill pack. It was frozen at a pinned commit before the scored runs, but we are not a neutral party.
One run exceeded the token cap during a correction round. It is included in the table rather than omitted.
What This Means for Engineering Leaders
The benchmark does not support a cost argument, and we do not make one. It supports a different argument, which is about risk.
When an agent works on a bare stack, its failures are structural, repeat across attempts, and are invisible to the agent that produced them. Detecting them requires a human reviewer who knows to authenticate as a non-administrative user. When an agent works within a framework that enforces its invariants, most failures surface at compile time and resolve without human involvement.
The question to ask of any stack you intend to hand to agents is therefore not which one produces less code. It is which one makes the agent's mistakes visible before your users encounter them.
Organizations evaluating agentic development for enterprise applications should test this property directly on their own workload: run the same specification twice, verify permissions under each role rather than as an administrator, and count the findings rather than the lines.
The frozen task specification, acceptance checklist, scoring oracle and raw run data are available on request.
