Skip to content

LLM Refactoring Adoption Patterns

Developer-initiated ChatGPT refactors are mostly adopted as-is; when modified, the change falls into one of five patterns driven by prompt context and refactor complexity.

Scope and caveats

Patterns come from Schön et al., 2026 (PROMISE 2026), which analyzed 169 commits and 440 files from the DevGPT dataset — ChatGPT (GPT-3.5 / GPT-4), July to August 2023.

The result is qualified, not universal:

  • Single-repo concentration: 143 of 169 commits come from one project (tisztamo/Junior) (Schön et al., 2026).
  • Adoption-biased sample: DevGPT only captures commits where developers shared the ChatGPT link. Rejected conversations are absent (Schön et al., 2026).
  • Automation confound: "in one of the projects, there is a shell script" that prompts ChatGPT, "extracts the code suggestion, updates the relevant file, and commits the changes". The paper does not say which project. The authors attribute most of the observed full-adoption rate to this automated pipeline, not to developer judgment (Schön et al., 2026).
  • Refactoring-only: not ambient completion (~10% useful raw inference, see Suggestion Gating) and not AI review feedback (16.6% adoption per arxiv:2603.15911).

Read the patterns as a vocabulary for modification work, not as proof that suggestions are reliable.

The headline

Token Match Rate density concentrates above 0.9 — most refactored files closely mirror the suggestion, and developers typically reach the goal in 1 to 4 prompts (Schön et al., 2026). The authors qualitatively inspected 190 datapoints: all 96 files across the full 440-file dataset that showed any token-level modification, plus a complementary sample of 94 files with a perfect token match. That sample is stratified, not random, so 96 of 190 is not a modification rate — the dataset-wide share of files with any token-level modification is 96 of 440, about 22% (Schön et al., 2026).

Refactoring activity distribution (Schön et al., 2026):

Activity Count
Rename 44
Documentation 37
Restructure 36
Logic splitting 33
Code cleaning 29
Simplification 25
Data type changes 7

These are mostly local, single-file transformations.

The five modification patterns

graph TD
    S[ChatGPT suggestion] --> Q{Did you change it?}
    Q -->|No| P1[1. Complete adoption]
    Q -->|Yes| W{What did you change?}
    W -->|Removed extra code| P2[2. Remove erroneous parts]
    W -->|Project-specific edits| P3[3. Integration adjustments]
    W -->|Kept logic, restructured| P4[4. Structural redesign]
    W -->|Kept structure, rewrote logic| P5[5. Structure preservation, content disregard]

1. Complete adoption without modification

The suggestion is committed verbatim (similarity = 1.0). This concentrates on low-risk, local tasks — variable rename, file reorganization — and on prompts that paste the entire file as context (Schön et al., 2026). Some of this is a prompting effect: when the prompt encodes the integration points, the suggestion lands as-is. But the authors trace much of the full-adoption rate to an automated commit pipeline in one unnamed project, not to developer review (see Scope and caveats).

2. Removal of erroneous parts

Developers keep the suggestion's core but delete additions that were never asked for — extra if statements, error handling, new helper functions, copy-to-clipboard logic. Deletions dominate the modifications (Schön et al., 2026). The pattern surfaces a recurring LLM tendency: scope inflation.

3. Integration adjustments for project compatibility

The suggestion is correct in isolation but wrong against the project — wrong file paths, wrong function names, wrong import conventions. Developers make token-level edits to align with project conventions, common in restructuring and modularity work (Schön et al., 2026) — the cost of partial-context prompts.

4. Structural redesign while preserving core logic

Developers keep the suggested logic but reshape it — combine multiple suggested functions, split one function across files, add comments, rename for consistency (Schön et al., 2026). Substance survives; form does not. Developer judgment about code organization outpaces what the model infers.

5. Structure preservation, content disregard

The inverse of pattern 4. Developers keep the suggestion's layout — function decomposition, file structure — but replace the logic with their own approach. Token Match Rate is low; the file is reorganized around different logic. In the paper's words, developers "preserve the layout or function decomposition in the suggestion, while disregarding the logic" (Schön et al., 2026). The suggestion functioned as a scaffold, not a solution.

What drives which pattern

graph LR
    C[Context completeness] -->|Full file| P1[Pattern 1]
    C -->|Partial| P3[Pattern 3]
    X[Refactor complexity] -->|Low| P1
    X -->|Medium| P2
    X -->|High| P4P5[Patterns 4–5]

Two axes explain which of the five modification patterns a refactor lands in (Schön et al., 2026):

  • Context completeness — full-file context drives pattern 1; partial context drives pattern 3.
  • Refactor complexity — simple transformations land in pattern 1 or 2; complex transformations (logic split, restructure) trigger patterns 4 and 5 because, as the authors note, "more complex suggestions often cause errors or add unwanted behavior" that requires more substantial modification (Schön et al., 2026).

Implications for practice

Name the pattern when you modify. Deleting code you did not ask for is pattern 2: push back on suggestion scope in the prompt. Translating names and paths is pattern 3, so give the model more file context next time, since full-file context is the cheapest way to land pattern 1. Rewriting logic under a kept structure is pattern 5 — the model gave you a scaffold, not a solution.

Do not generalize past the dataset. Findings cover developer-initiated, single-file, local refactors. Cross-module refactors, interface changes, and review-feedback adoption show different rates (compare Human-AI Review Synergy, where AI review suggestions adopt at 16.6%).

Example

Two excerpts from the paper, one a commit and one a suggestion, show what the metric does and does not see (Schön et al., 2026).

Pattern 2, removal of erroneous parts. ChatGPT suggested a GenerateButton component whose handleGeneratePrompt function called copy(response.prompt) and setPrompt(htmlPrompt) after generating a response. The developer kept the suggestion's core logic but removed both calls before committing — behavior nobody asked for, edited back out.

Not a pattern: a limit of the metric. The paper introduces this one with "on the other hand", against pattern 5. Here the developer does apply the suggestion, and similarity stays low because command-line operations "are not captured by the similarity measures". A suggestion to reorganize a project's file layout included:

mkdir -p src/frontend/service/helpers/
mv src/frontend/services/helpers/getComparison.js src/frontend/service/helpers/getComparison.js
rm -r src/frontend/services/

Token Match Rate does not see mkdir, mv, or rm, so a suggestion of this shape scores as heavily modified whatever the developer does with it. The paper prints the suggestion alone, with no committed file beside it, so it evidences the blind spot rather than any particular adoption.

The paper names no repository for either excerpt. Read them as illustrations, not as a claim about how often either case occurs.

Key Takeaways

  • Most committed ChatGPT refactors in the DevGPT dataset are adopted with high similarity; only about 22% of the 440 files show any token-level modification, and even that is a floor, since a perfect token match can still hide a deletion
  • The five patterns (complete adoption, remove erroneous parts, integration adjustments, structural redesign, structure preservation with content disregard) are driven by prompt context completeness and refactor complexity, not LLM capability per se
  • Findings are qualified by single-repo concentration (143 of 169 commits), an automated commit pipeline in that repo, and adoption bias (rejected conversations excluded) — generalization requires explicit conditions
  • The result does not extend to AI-generated review feedback (16.6% adoption — arxiv:2603.15911) or to ambient code completion (~10% useful raw inference — see Suggestion Gating)
Emerging — single recent source Last reviewed: 2026-08-30