The Patchwork Problem in LLM-Generated Code¶
LLM-generated code can compile, pass tests, and clear SAST yet be structurally incoherent — each patch locally valid, the whole globally broken.
The patchwork problem is single-artifact structural incoherence: an LLM emits a change that is locally correct but violates a consistency invariant the rest of the repository already holds. A generated endpoint reads a config key nobody declared. An import targets a package that exists in no registry. A new route drops the authentication guard every sibling endpoint carries. Each patch is plausible on its own, so it passes review that reads it in isolation (Mothukuri & Parizi, arXiv:2607.08981).
What it looks like¶
The paper formalizes structural coherence as consistency invariants over eight graph views of a repository — import, call, dependency, configuration, schema, resource, control-flow, and routing — and names eight failure categories (arXiv:2607.08981):
| Category | The incoherence |
|---|---|
| Symbol resolution (SRF) | A referenced name resolves nowhere in the module graph |
| Phantom internal API (PIA) | A call uses a signature the real declaration does not have |
| Dependency hallucination (DHI) | An import names a package absent from registries and the manifest |
| Build/config incoherence (BCI) | Code assumes environment variables or config never declared |
| Resource coherence (RCF) | A required file, return, or schema is missing |
| Control-flow coherence (CFC) | Unreachable blocks or contradictory conditions |
| Cross-file contract (CCV) | Producer and consumer disagree on a field name or shape |
| Security structural regression (SSR) | A route omits a guard its sibling routes enforce |
Why standard CI misses it¶
Type checkers, tests, and SAST reason about local correctness, not repository-scale consistency. In the paper's evaluation, type checking and linting caught 2 of 67 structural findings, test execution caught none, and SAST caught none — the vast majority evaded every baseline (arXiv:2607.08981). The risk concentrates in exactly the changes agents are asked to make: cross-cutting tasks failed at about 44.6% versus 13–16% for single-file work.
Why it works¶
The mechanism is convention over ground truth. An LLM generates token by token from training-data patterns, not from a symbolic model of the target repository. When wiring a change requires consistency across configuration, middleware, and schema boundaries the model cannot see in context, it emits the plausible continuation (a config key, field name, or method signature that matches naming conventions) instead of one grounded in the repo's actual declarations (arXiv:2607.08981). Each token is locally probable, so each patch is locally valid; incoherence only appears at the scale tests and per-file linters never check. The same mechanism produces hallucinated dependencies: one 576,000-sample study found commercial models invent packages in about 5.2% of outputs and open-source models in 21.7% (Spracklen et al., arXiv:2406.10279) — names an attacker can register to poison the supply chain (Socket).
This is distinct from its neighbours. Shadow tech debt is cumulative drift across many merged PRs; assumption propagation is a wrong interpretation that solves the wrong problem. The patchwork problem is one artifact's structural break, detectable at generation time by static graph analysis.
Mitigation¶
Add repository-scale structural checks that standard CI lacks. The paper's hybrid framework delegates symbol and signature checks to mature static analysis where it already excels, then adds purpose-built detectors for the cross-cutting invariants existing tools underserve — configuration coherence, dependency validation, and cross-file contracts (arXiv:2607.08981). Practically: resolve every import against the manifest and registry before merge, diff a new route's guards against its siblings, and validate config keys against the declared schema.
When this backfires¶
Repository-graph checking is not free, and the failure it targets is uneven. Skip or scope it down when:
- the codebase is greenfield or throwaway — there is no established structure to be incoherent with;
- the stack is strongly typed and well tooled — a strict TypeScript, Rust, or Java build already surfaces most SRF, PIA, and DHI as compile errors;
- the change is a small single-file diff a reviewer sees whole — the paper's own data puts simple-task failure at 13–16%, so a detector mostly adds triage cost;
- the code is dynamic or loosely typed — graph checks over-approximate on Python and JavaScript and can flood reviewers with false positives, the classic weakness of static analysis, where detectors fail through near-zero recall or overwhelming false positives (Du et al., arXiv:2601.18844).
Failure patterns also diverge between models, so a detector tuned on one model's profile under-covers another's (arXiv:2607.08981).
Key Takeaways¶
- The patchwork problem is code that is locally valid but globally incoherent — undeclared config keys, phantom imports, missing auth guards — and standard CI is nearly blind to it.
- The cause is convention over ground truth: LLMs emit plausible continuations, not repository-grounded ones, so incoherence appears only at repository scale.
- Catch it with repository-scale structural checks: resolve imports against the manifest, diff route guards against siblings, validate config against schema.
- The value is highest on cross-cutting changes in large, dynamically typed repos and lowest on small diffs in strongly-typed, well-tooled stacks.