The day after cutover, what will tell you whether the next AI change is safe to accept?

That question belongs at the center of any legacy modernization evaluation. Judge tools on how well they convert your code, and on what they leave behind: evidence your reviewers and auditors can accept at cutover, and a baseline your team can use for every change after it.

The question gets harder as AI takes on more of the work. For mainframe exit projects specifically, Gartner predicts that more than 70% of those started in 2026 will fail to deliver their intended benefits, because organizations overestimate what generative AI tooling can do. Its analysts describe “a widening gap between the marketing promise of GenAI and its real-world ability to transform and migrate complex legacy code.”[1]

That gap doesn’t show up in a demo, where conversion looks fast. It shows up at acceptance, when reviewers, risk teams, and auditors have to agree that the converted system can be trusted. And it doesn’t end there. As AI agents become part of maintaining the converted system, each subsequent change needs the same discipline around evidence, scope, and acceptance that the migration did. So the migration should leave behind more than converted code: an accepted baseline of the system’s business rules, relationships, and logic, derived from source, and evidence the team can reuse when reviewing future changes.

What is a legacy modernization evaluation? An assessment of how an approach converts legacy code, and what evidence it leaves behind, both for acceptance and for every change after it.

This guide focuses on that evidence question. Security, data migration, performance, resilience, deployment, and commercial fit need their own evaluation. It covers three shifts that change how modernization tools should be evaluated, then a compact toolkit for running the evaluation:

• What each approach leaves behind for governing changes after cutover

• How to measure a program by what gets accepted, not what gets converted

• What “deterministic” actually means, layer by layer

• The evaluation toolkit: must-pass questions, proof-of-value tests, and a map of the market

Start with the questions. Download the vendor evidence questions: five recommended must-pass questions and six proof-of-value tests on one page, with room to record each vendor’s answers.

What happens after cutover: evaluating modernization tools for ongoing AI change

Modernization evaluations treat the migration as a project with an end date, with “done” defined as cutover. But the converted system keeps changing, and AI agents are increasingly the ones changing it. Each change raises the questions the migration did: what changed, what was preserved, and who accepted the difference.

If the migration leaves only converted code and test reports, governing those changes starts over. The team has to reconstruct its understanding of the system from the code, the test suite, or the memory of whoever ran the migration. That’s the work the migration was supposed to make unnecessary.

That’s why the question at the top of this guide belongs in every vendor conversation:

The day after cutover, what will tell us whether the next AI change is safe to accept?

The answers vary by approach. Individual products can span more than one approach, so read each row as a starting point for questions, not a verdict on a vendor. Source-derived evidence isn’t an alternative to the others: it can accompany any of them, so it appears last for comparison.

Runtime evidence, requirements, analysis of the source code, and recorded decisions are complementary. No single approach captures everything a system does. Asking for this during the evaluation costs nothing. Retrofitting it after cutover means working out what the system does all over again.

A worked example

Consider a hypothetical interest calculation that rounds half-cent results up, used by three batch jobs.

1. Original behavior: the COBOL source shows the rounding rule and the three jobs that depend on it.

2. Converted implementation: the Java version uses BigDecimal with HALF_UP rounding.

3. Evidence: the analysis records the rule as preserved, traces it to its COBOL source and all three jobs, and lists one item as unresolved: a year-end path that no test data exercised. A reviewer checks that item, and the team accepts the baseline.

4. A later change: a year after cutover, an AI coding assistant refactoring a shared helper switches the rounding to HALF_EVEN. Checked against the accepted baseline, the change appears as a difference in an accepted rule that affects all three jobs, and it’s held for review instead of merging unnoticed.

For systems integrators, the same question is practical. An accepted baseline supports acceptance discussions with the client, hands over application knowledge at the end of the program, speeds up investigation of later changes, and creates a basis for offering ongoing governance as a continuing service.

How to measure a legacy modernization program when conversion is fast

Reviewer and auditor capacity doesn’t grow with AI output. A program can convert code faster than anyone can accept it, and the usual measures of progress, lines converted and percent automated, track the part that got faster. They say little about whether the program is getting closer to acceptance.

Five measures track acceptance instead:

• Accepted changes per reviewer-hour: shows whether review capacity, not conversion, is the real bottleneck.

• Share of the system accounted for: how much of the system’s business logic has a recorded outcome. Gaps are what auditors flag.

• Unresolved items open, by count and age: known unknowns waiting for a decision. Aging items signal acceptance risk.

• Time from conversion to acceptance: the lag between code delivered and code accepted, which shows where the program actually stalls.

• Analysis and evidence reproducibility: whether rerunning the analysis on the same code version, with the same analysis version and rules, produces the same findings and evidence. Evidence that can’t be reproduced can’t be re-verified for audit.

These belong in the program dashboard, and in the contract, alongside or in place of volume measures. When a vendor reports progress, ask for them. The same measures keep working after cutover, when the work shifts from migrating the system to governing the changes made to it.

What “deterministic” means in code conversion

“Deterministic” now appears across modernization marketing. IBM, Google, CAST, Swimm, Moderne, CloudFrame, and Heirloom Computing all use it to describe their approach. They don’t all mean the same thing. The word can describe three different layers, and each guarantees something different.

Some vendors span more than one layer, or use the word differently; the table reflects how each describes itself. Where Holonic uses models to write plain-English explanations or summaries, those are identified separately from the deterministic evidence.

The distinction matters because the layers don’t inherit from each other. A deterministic analysis feeding an AI generator can still produce different code on every run: the model of the system is consistent, but the output isn’t. And deterministic transformation tells you the output is repeatable, not that it’s complete. So when a vendor says “deterministic,” ask which layer they mean, then run the same input twice and see. For how source-derived and prompt-derived approaches differ more broadly, see what replaces the two-year parallel run.

Repeatability isn’t correctness, either. Identical results can still be wrong. Reproducibility makes independent verification possible; it doesn’t establish on its own that converted code behaves like the original.

Evaluation criteria for legacy modernization tools in regulated industries

Holonic recommends five criteria, drawn from the questions reviewers, auditors, and examiners raise when a modernized system goes up for acceptance. We suggest treating all five as must-pass: a weak answer to any one is likely to resurface at acceptance, however fast the conversion was. Each pairs with a test you can run in a proof of value. Two shortcuts can stand in for this evidence without providing it: treating conversion coverage as completeness (the coverage trap) and relying on sampled review (why sampling falls short).

Before a proof of value, the same questions work in vendor meetings and RFPs; how a vendor answers is itself useful evidence. For a step-by-step testing checklist for COBOL to Java migrations specifically, see COBOL to Java migration testing. For what risk teams look for at sign-off, see what risk officers ask for.

Give your evaluation team one standard. Download the vendor evidence questions: the five recommended must-pass questions and six proof-of-value tests on one page, with room to record each vendor’s answers. [Download the questions]

The legacy modernization landscape in 2026

The approaches below are the main ways to convert or move legacy code. The vendors are examples from each category, not endorsements or a complete list.

Rehosting and wrapping avoid conversion risk altogether, at the cost of keeping the legacy language and the skills it requires.

Every approach in the table answers how the code gets converted. Source-derived evidence answers what lets you accept the result, and every change after it. Holonic modernizes code with CodeIntent, and CodeIntent can govern target code produced by other modernization approaches when the source code, the converted code, and the required analysis inputs are available.

This market changes every quarter. Get a short briefing when it does: vendor moves, new approaches, and changes in evaluation practice.

Where CodeIntent fits

CodeIntent derives a baseline of a system’s business rules, relationships, and logic from source, each traced back to where it came from, along with the evidence behind it. People make it an accepted baseline: the team reviews it, resolves or explicitly accepts unresolved items, and authorizes intentional changes. The accepted result becomes the system of record for future changes. Holonic uses it to modernize code, recording what was preserved, what changed, and what remains unresolved.

The baseline outlasts the migration. CodeIntent can be integrated into the change-governance and CI/CD workflow so proposed changes, including same-language changes such as Java to Java, can be checked against the accepted baseline. When a check finds a change in what the code does, the team decides whether it’s acceptable. Accepted changes are recorded and become the new baseline; others are fixed or held for review. In CodeIntent Studio, it reports how much of the system is accounted for, and open items by count and age. For how that works on everyday AI-written changes, see how to detect unintended behavior changes in AI-generated code.

In CodeIntent Studio, the Trace Explorer traces converted code back to its source and the evidence behind it.

The evidence has limits, and we state them. Behavior outside the analyzed scope, such as runtime configuration, stays explicitly unverified rather than assumed. Run the tests in this guide on Holonic as you would on any other approach, and see how CodeIntent works.

Frequently asked questions

What should you look for in a legacy modernization tool?

Beyond conversion speed, look for evidence a regulated organization can accept: results that reproduce, outputs that trace to their source, a complete accounting of what changed and what’s unresolved, and evidence someone other than the vendor can verify. Ask, too, what the approach leaves behind for governing changes after cutover.

What does “deterministic” mean in code conversion?

It depends on the layer. A deterministic analysis produces the same model of the code every time; a deterministic transformation produces the same converted code from the same inputs and configuration; deterministic evidence produces the same record of findings. A deterministic analysis feeding an AI generator can still produce different code on each run.

How do you govern AI changes after a migration?

Evaluate each change against an accepted baseline of the system’s business rules and logic, derived from source, alongside tests and code review. Record a decision for every difference, including the ones still under review, so every difference has a recorded outcome.

Which tools are used for mainframe modernization?

The main approaches are AI-assisted conversion (IBM Bob for Z, AWS Transform, Google Cloud), rule-based conversion (CloudFrame, TSRI, Astadia, Heirloom Computing), parallel-run verification (Mechanical Orchard), rehosting (Rocket Enterprise Server, Raincode), and analysis tools (CAST, Swimm). Source-derived evidence platforms such as Holonic CodeIntent can govern code produced by these other approaches.

The question that outlasts the evaluation

Every modernization evaluation ends with a choice of approach. The better question is what that choice leaves behind: converted code, or an accepted baseline for checking every future change.

Bring the evidence question to every vendor. Download the vendor evidence questions for your evaluation team, and opt in to a short quarterly briefing on how legacy modernization is changing. [Get the questions]