Skip to content

Auditing Agent Tool Chains for Silent Partial Success

A tool call can return success with most of the answer missing, and nothing in the response says so.

Audit a tool integration for silent partial success when two conditions hold together. The tool exposes a richer surface than the one your agent calls, such as a web interface or a documented API the wrapper only partly covers. And a downstream claim depends on having retrieved everything. Where both hold, compare the surfaces and record what each layer drops. An audit of 15 scientific tools inside ToolUniverse validated 91 such failures, defining one as an interaction where "(1) the tool invocation appears successful, (2) the returned information is incomplete, transformed, ambiguous, or otherwise insufficient for the intended task, and (3) the limitation is not adequately disclosed to the downstream user or agent" (Gopalan et al., arXiv:2609.26836v1).

Where to look

The audit names seven points in the chain from the resource to the agent. Four of them are differentials, so each needs a reference surface to compare against. The API is missing a capability the tool has, the API covers it only partly, no wrapper exposes it, or the wrapper "partially omits parameters, fields, or formats which the API returns" (§1). The other three sit outside that comparison: an inherent limit in the resource, an output the agent cannot use, and an output the agent misreads.

Of the 91 validated failures, 51 sat at the API layer and 25 at the wrapper layer. The authors could not place three, and put the other 12 outside those two layers. The agent usability point accounted for no confirmed failures in any of the 15 tools, and the authors write that "Failures at the agent-level loci were comparatively uncommon in this analysis" (§3). Read that as a pointer, not a proportion. The limitations section says "the candidate generation may influence the proportion of failures observed and hence proportion of failures are not to be considered as representative of typical silent failures in agent-tool interaction" (§6).

Completeness was the most affected dimension, 28 of the 134 occurrences the 91 cases produced, and missing data or fields the most common issue type at 38 (§3). The recurring shapes are mundane. EMDB caps its rows parameter at 1000 "despite the database containing substantially more records". BiGG search results are "silently capped with no documented page size", and its documentation "reports API v1.3.0 while live responses report v1.6.0". The ClinicalTrials.gov wrapper "returns a success status with empty outcomes" where the web interface shows an explicit no-results state (§4, §5).

Running it

Candidates came from documentation, not from traffic. The authors assembled each tool's docs, web interface detail, API docs, manuals, release notes, and reported issues, then looked for "conflicts, contract disagreements, implementation issues, and Web/API/wrapper inconsistencies" (§2).

The test-design rule is the part worth copying. Each query named a concrete identifier, asked the agent to report what came back, and never asked it to grade itself: "Queries were designed to test the target failure rather than ask the agent to independently determine whether a failure existed" (§2). An agent asked whether its own output is complete answers from the same truncated payload that caused the problem.

Compare computed results with rendered text

Extend the audit through the formatter. A tool can compute the right result, then print text that hides its limits. Who Verifies the Tools? studies this boundary in one archive research deployment, across 4 selected defect families.

Compare the rendered response against the internal result or authoritative data. The study's cases suggest four checks:

  • State the search scope, including sources or records the query excludes.
  • Distinguish returned records from the total, with any result cap.
  • Label an estimate and preserve its uncertainty.
  • Identify the namespace of each identifier before using it to retrieve provenance.

These checks follow the disclosure checklist in section 5. Namespace labels also need enforcement; the study finds that clearer wording alone leaves provenance errors largely intact.

For verifier testing, hold an incorrect answer fixed and vary only the tool text supplied with it. The paper uses 30 false answers, 10 each for scope, completeness, and uncertainty, judged by 3 verifiers. All 90 paired verdicts change from acceptance under misleading text to rejection under clear text. These are repeated judgments of selected answers, not 90 independent tasks. Section 4.2.

Provenance is the counterexample: verifier acceptance changes only from 18/30 to 14/30. In the agent experiment, provenance overclaim remains at 94% under clear wording. Repairs also combine factual disclosures with behavioral instructions, so that experiment cannot isolate disclosure alone. These stress tests do not estimate how often formatter defects occur. Results and limitations.

The shared-premise verifier failure can therefore enter through tool output as well as a task description. Give the verifier an independent reference. In selected scope and completeness cases, the paper's control shows verifiers rejecting errors when given the missing facts and a correctness question. The control does not establish general verifier competence. Verifier control, Table 1.

Why it works

When a response omits its limits, the agent cannot infer completeness from a success status alone. The paper's illustration is an API that "returns its first 1,000 records (out of 10000 records)" while the agent reports the records (§1). The cost arrives one layer later, in what the authors call silent amplification: "An upstream limitation that is relatively benign at the tool or API layer can become a more consequential error when the agent interprets the partial output as complete evidence" (§1). That is why task-success benchmarks miss this class. They score the answer, and the answer is well formed.

Auditing the chain checks whether the output preserves what the authors call contextual reliability, "the extent to which an Agent–Tool interaction preserves and communicates the information, qualifiers, provenance, scope, and meaning required to support the intended scientific conclusion" (§5).

When this backfires

  • No richer surface exists. An internal service your own team wraps has nothing to compare against. The four differential checks return nothing, and you are left with the three that need runtime evidence.
  • Completeness is not load-bearing. An agent drafting prose or triaging a queue can lose a field at no cost.
  • The tool ships weekly. An audit is a snapshot of a documentation state, and the authors concede "The APIs, wrappers, interfaces, and data evolve, so individual observations may change with tool versions" (§6).
  • The failure leaves no documentary trace. Candidate discovery reads documentation, so undocumented behavior is invisible to it. A synthesis of 27 papers keeps tool invocation and parameter-level errors as its own top-level failure cluster (arXiv:2607.05775v1), and this method found almost nothing at the agent.
  • Location may be the wrong axis. An eight-week study of one production runtime cuts silent failures by mechanism instead, reporting that "the longest-lived failures lived in the seams between components", where no test runs by construction. In a system "defended by 4,286 unit tests and 827 declarative governance checks", it found that "roughly 70% of silent failures were ultimately caught by human user-view observation of system output, not by unit tests, health checks, or governance audits" (Wu, arXiv:2606.14589v1). If your defenses are already mechanism-shaped, a map of the chain adds vocabulary rather than a fix.

Key Takeaways

  • Treat a successful tool call as evidence the call ran, and nothing more. Check whether the output discloses scope and limits before relying on it.
  • Pick the tools to audit by whether they expose more than your wrapper reaches, and by whether a missing field would change the answer.
  • Write test queries that name a concrete identifier and ask what came back, then check the answer against the richer surface yourself. Asking the agent to judge its own completeness reads the same short payload twice.
  • The 51-to-25 API-to-wrapper split is an observation from one environment, not a prior. The authors say so themselves.
  • Expect result caps, undocumented page sizes, a success status returned over an empty result, and a documented API version that disagrees with the live one. Those four shapes cover the audited examples.
Emerging — single recent source Last reviewed: 2026-09-24