Skip to content

Explained Feedback for LLM Vulnerability Repair

Explaining a scanner finding as an attack mechanism with ordered repair steps beats the raw diagnostic for strong instruction-followers on single-weakness findings.

Hand the model the scanner's own diagnostic first. Averaged over the eight models Yan et al. tested, that alone cuts the target-vulnerability rate by 1.7 to 5.5 percentage points depending on the benchmark, though individual models show no change (Yan et al., 2026). Rewriting it into an explanation buys another point or two, and on one of their three benchmarks nothing. Even their best prevention hints left "a non-trivial residual of vulnerable code", and repair lowered the rate rather than clearing it (Yan et al., 2026). The detector and the merge gate carry the security. Feedback wording is a tuning knob inside that loop.

When this applies

Rewrite the finding only when all three hold. Otherwise pass the raw diagnostic through and spend the effort on detection coverage.

  1. The repairing model executes multi-step instructions reliably. The gain tracks instruction-following, not security knowledge. GPT-4o and GPT-3.5-turbo are the study's strongest instruction-followers and "translate the explanations into coherent patches", while smaller open-weight models "often fail to coordinate the necessary updates" on longer, multi-CWE explanations (Yan et al., 2026).
  2. Each finding is one weakness with a localized fix. The benefit "diminishes as the required changes become more distributed or configuration-heavy" (Yan et al., 2026).
  3. A model stronger than the repairer writes the explanation. A GPT-4o teacher wrote every explanation in the study; the self-explained variant regresses, covered below.

What the explanation has to add

The raw diagnostic says where. The explanation says why it is exploitable and what to change. Yan et al. draw four requirements from their case study: name the attack mechanism and its security impact, give fine-grained repair guidance rather than leaving the model to infer remediation from tool output, state the security invariant instead of an isolated syntactic change, and order the steps (Yan et al., 2026).

Averaged over eight models against a vanilla prompt, teacher explanations move the target-vulnerability rate by −2.9 points on SecCodePLT and −5.8 on SecurityEval, against −1.7 and −4.7 for the raw diagnostic. The authors size that gap themselves: "the marginal value of an explanation on top of a precise detector is modest to begin with, and whether it is net-positive depends on the model's ability to use it" (Yan et al., 2026).

Why it works

Naming a weakness and removing it are separate capabilities, and the raw diagnostic only exercises the first. Yan et al. state the gap directly: "raw diagnostics identify where a vulnerability exists, but provide insufficient guidance on how to safely remediate it." A model given only the finding produces edits that "appear security-related" while leaving the weakness reachable. Stating the invariant instead of the symptom changes what the model edits. In their archive-extraction case the same model "does not merely patch the flagged line, but restructures the extraction logic to enforce a global security invariant." Their summary of the study is the mechanism: "recognizing a vulnerability is not the same as being able to act on it" (Yan et al., 2026).

When this backfires

  • The model explains its own finding. Self-explained feedback, with no external teacher, averaged −2.0 points on SecurityEval against −4.7 for the untouched diagnostic, and pushed several models above the do-nothing baseline: Llama3.1-8B rose 3.1 points on the target rate and 10.9 on the any-vulnerability rate (Yan et al., 2026). Huang et al. report the same shape outside security: models "struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction" (Huang et al., 2024).
  • Several weaknesses land in one task. CyberSecEval averages 3.6 distinct CWEs per sample against 2.8 elsewhere, so the explanation runs to roughly 300 words and asks for coordinated edits in one pass. Averaged over all eight models it came in below the raw diagnostic on both rates, −4.2 against −5.5 on the target rate (Yan et al., 2026).
  • The repairer is a weak instruction-follower. On that same benchmark, swapping the terse diagnostic for the explanation moved StarCoder2-15B from −10.9 to −4.0 and DeepSeekCoderV2-16B from −4.7 to −0.6 (Yan et al., 2026). Partly applying a long instruction leaves the code globally insecure where the terse diagnostic had kept the fix narrow.
  • Fixing the weakness needs configuration-level reasoning. Path traversal (CWE-22) and XML external entity processing (CWE-611) resisted both prevention and repair throughout the study.
  • CodeQL never raises the weakness. Every result here is measured with that scanner, which "fails to capture dynamic vulnerabilities or those that manifest under specific runtime conditions" (Yan et al., 2026). Nothing it misses gets feedback of any grade.
  • The repair breaks the feature. Vulnerability rates are all the study reports, so a fix that clears the finding and regresses behavior scores as a success. Keep the functional suite in the loop alongside the scanner.

Example

Yan et al. walk through a tar extraction that lets a crafted archive entry escape the target directory. Handed the raw CodeQL finding, the model applied surface fixes that left containment unenforced. Handed feedback that named the attack mechanism, stated the containment invariant, and ordered the steps, the same model added a check validating each entry before extraction and aborted on a traversal attempt, restructuring the extraction path rather than patching the flagged line (Yan et al., 2026).

The four requirements above come from that single case study. The authors call them "consistent with the empirical improvements observed for explained feedback in our larger-scale experiments" rather than separately measured (Yan et al., 2026).

Key Takeaways

  • Pass the raw scanner diagnostic back first. It is the oracle-free condition in the study and it captures most of the available reduction on its own (Yan et al., 2026).
  • Budget the rewrite as roughly 1 to 1.5 percentage points, not as the control. Gate the merge on a re-scan.
  • Never let the repairing model explain its own finding without a stronger explainer. That variant regressed past the do-nothing baseline on several models (Yan et al., 2026).
  • Split multi-weakness findings into one explanation each. The advantage disappears at 3.6 CWEs per task (Yan et al., 2026).
  • Track functional correctness separately. These benchmarks measure vulnerability rates only.
Emerging — single recent source Last reviewed: 2026-09-08