Abstract illustration of legacy systems being modernized with AI coding agents

Legacy Modernization With AI Coding Agents: Where They Help, Where They Fail

AI coding agents can speed up understanding, testing and migrating old systems, but only inside controls that prove the new code behaves like the old. Here is where they help, where they fail and what to put in place.

By Shayan Salehi 8 min read

Image: ColdAI

Legacy modernization with AI works when coding agents are used to understand, document and test old systems, and when every change they make is checked against what the old system actually does. It fails when agents translate code and ship it on the strength of a plausible diff. Four controls separate the two outcomes: characterization tests, behavioral equivalence checks, incremental strangler fig migration and human review.

Why legacy systems are a risk worth fixing now

The scale of the problem is documented. In GAO-25-107795, published July 17, 2025, the US Government Accountability Office reviewed the legacy systems that federal agencies said most needed modernization and narrowed them to 11. Those systems range from 23 to 60 years old. Eight use outdated programming languages, four rely on unsupported hardware or software, and seven have known cybersecurity vulnerabilities.

The same report shows how planning lags behind the need. Nine of the 11 systems have modernization plans, but only three plans include all three elements GAO looked for: milestones, a description of the work and the planned disposition of the legacy system. GAO warned that incomplete plans raise the likelihood of cost overruns, schedule delays and project failure.

The federal government is one example, but the pattern applies to any organization with a decades-old core system. The scarce resource is usually not the new platform. It is knowledge of what the old system does, written down nowhere, and the time to prove a replacement does the same thing.

Where AI coding agents help

Agents are most useful where their output can be checked cheaply. That describes four jobs in a modernization program.

Code comprehension and documentation

An agent can read a large codebase, trace a field from the screen to the database, and summarize what a batch job does in plain language. A senior engineer can verify that summary against the code in minutes, which is far faster than writing it from scratch. The output is a draft for human review, not a source of truth. Treat every generated explanation as a hypothesis until a test or a domain expert confirms it.

Test generation

Legacy systems rarely have tests. Agents can propose test inputs, run the old code and record the results. IBM researchers describe a related approach in "Automated Testing of COBOL to Java Transformation" (arXiv, April 2025). Their framework uses symbolic execution to generate unit tests for the original COBOL programs, mocks external calls, and converts the tests into JUnit tests that check whether the translated Java behaves the same way.

Mechanical migration

Google's experience report on LLM-assisted code migrations (arXiv, January 2025) is the best public evidence on what agents can do at scale. For a migration of Google Ads identifier types from 32-bit to 64-bit integers, the authors report that 80 percent of the code modifications in landed changelists were fully AI-authored. Engineers estimated the total time was reduced by 50 percent compared with a similar migration done without LLM help, including review and landing time.

In a JUnit 3 to JUnit 4 test migration, the same paper reports 5,359 files and more than 149,000 lines modified in three months. About 87 percent of the AI-generated code was committed without change. The authors note that the bottleneck was the speed at which engineers could review the changes.

Translation, with caution

Translating one language to another is the task executives ask about most, and it is where the evidence is weakest. The next section explains why.

Where AI coding agents fail

The Google paper is candid about limits. For the identifier migration, the authors found that in most cases a human needed to revert at least some of the model's changes because they were incorrect or unnecessary. The model was good at the bulk of the work and unreliable at the edges, which is exactly where production incidents live.

Language translation is harder. In "Lost in Translation" (arXiv, August 2023), researchers tested LLMs on about 1,700 code samples across C, C++, Go, Java and Python. Correct translations ranged from 2.1 percent to 47.3 percent depending on the model, and the authors cataloged 15 categories of translation bugs. Those models are now old, so treat the figures as a floor rather than a forecast. The lesson that has not aged is the bug taxonomy: translations that compile and look right but behave differently.

Several failure modes follow from how legacy systems are built:

  • Implicit behavior. Decimal rounding, date handling, character encodings and error codes are often defined by the platform, not the source file, so a literal translation changes results.
  • Hidden dependencies. Batch schedules, shared files, database triggers and operator procedures sit outside the code an agent reads.
  • Dead and contradictory code. Agents tend to treat everything in the repository as intended behavior, including paths nobody has executed for years.
  • Review fatigue. Large, plausible diffs are hard to read carefully, and reviewers approve what they cannot fully check.

The 2025 DORA report, "State of AI-assisted Software Development," points to the same dynamic at the organizational level. Based on a survey of nearly 5,000 technology professionals conducted between June 13 and July 21, 2025, it found that 90 percent use AI at work and more than 80 percent believe it has raised their productivity. It also found that 30 percent report little to no trust in AI-generated code. AI adoption now improves software delivery throughput, DORA reports, but it still increases delivery instability. DORA's summary is that AI is an amplifier: it magnifies the strengths of high-performing organizations and the dysfunctions of struggling ones.

What benchmarks do and do not tell you

Vendors often cite SWE-bench scores to show that agents can fix real software problems. Read those numbers with care. The original SWE-bench paper describes 2,294 task instances drawn from 12 Python repositories. A benchmark of well-maintained open-source Python tells you little about a 30-year-old COBOL or PL/SQL estate with no tests.

The benchmark also has documented quality problems. In "SWE-Bench+" (arXiv, October 2024), researchers analyzing SWE-Agent with GPT-4 found that 32.67 percent of successful patches benefited from the solution being stated in the issue report or comments. Another 31.08 percent passed tests too weak to verify correctness. After filtering these out, the resolution rate fell from 12.47 percent to 3.97 percent. The authors report that similar problems appear in SWE-bench Lite and SWE-bench Verified, and that over 94 percent of the issues were created before the models' knowledge cutoff dates, which raises contamination concerns.

The takeaway is not that agents are weak. It is that a public benchmark score is not evidence that an agent will preserve the behavior of your system. Only a test suite built from your system can show that.

Controls that make it safe

Four controls turn agent output from a plausible draft into something you can deploy.

  1. Characterization tests. Before changing anything, record what the legacy system does for a wide range of real inputs, including its mistakes. These tests describe actual behavior, not intended behavior. Agents can help generate inputs, but the expected outputs must come from running the old system.
  2. Behavioral equivalence checks. Run old and new implementations side by side on the same inputs and compare outputs. Where the new path is exposed to live traffic, shadow it: send production requests to both, serve the old answer, and log every difference. Treat each difference as a defect or a deliberate, documented change.
  3. Incremental replacement. Move one capability at a time behind a routing layer, so each step is small, reversible and measurable.
  4. Human review sized to the risk. Limit how many changes an agent can generate per week so reviewers can read them, as Google did to avoid overwhelming its engineers. Require a domain owner to sign off on anything touching money, safety or regulated data.

IBM's published work on COBOL-to-Java translation shows why the first two controls matter together. A second IBM paper, "Quality Evaluation of COBOL to Java Code Transformation" (arXiv, July 2025), describes a system that combines analytic checkers with an LLM acting as a judge, aimed at continuous integration and large-scale benchmarking. Even there, the design relies on several different checks rather than a single score.

Strangler fig migration and the role of the agent

Martin Fowler's page on the strangler fig pattern, dated August 22, 2024, describes the approach this way: build new functionality alongside the legacy system and move behavior over piece by piece. He notes that full rewrites often fail, partly because reproducing existing behavior, including behavior nobody wants, is hard. He also notes that legacy systems usually lack clean seams, so teams must create them, and that the temporary architecture which lets old and new coexist is an accepted cost.

This pattern fits agent-assisted work well. Each slice is small enough for an agent to handle and for a human to review. Each slice has a clear boundary at which to compare old and new outputs. If a slice goes wrong, the routing layer sends traffic back to the legacy path. A big-bang cutover removes all three safeguards.

Fowler's point about organizational change also holds. DORA's finding that AI amplifies existing dysfunction means the migration will inherit the team's habits. A team that does not review carefully or deploy in small batches will not start doing so because an agent is writing the code.

Where to start

You can begin without committing to a full program.

  1. Pick one bounded capability with real business value and a clear interface, such as a pricing calculation, a statement run or a single report.
  2. Capture its behavior. Record inputs and outputs from the live system, including edge cases, and store them as characterization tests.
  3. Use an agent to document the capability and to draft the replacement, then compare the draft to the recorded behavior.
  4. Run the new implementation in shadow mode and review every difference with a domain owner.
  5. Cut over that one slice behind a routing layer, keep the rollback path for a defined period, and measure review time, defect rates and differences found.

Start modernization work this way: with the old system's recorded behavior as the standard, and with agents confined to tasks whose output can be verified. Once the first slice runs cleanly, you have a measured basis for deciding how much more to hand to agents and how much review the next slice needs.

Tags:legacy-modernizationai-coding-agentsstrangler-figcharacterization-testscode-migrationsoftware-engineeringtechnical-debt

Your starting point

What would you like to move forward?

Start with the outcome that feels closest. We'll help give the next step a useful shape.

Let's find your next move

One conversation. A clearer direction.

Tell us what you want to improve. A few sentences are enough to start.

shayan@coldai.org