Runnable Documentation as Agent Verification¶

Convert inline code examples into standalone files that CI executes on every build — catching doc rot with the same signals that catch broken code, and cutting stale-context failures in agents that retrieve docs via RAG.

Related lesson: Golden Journeys, a hands-on lesson with quizzes covers this concept.

The mechanism¶

Every inline code example is a hidden assertion that the API works as shown. A reader checks that assertion only when they copy the snippet and find it broken. Runnable documentation promotes the assertion to a test.

The pipeline has four steps (LangChain: How We Made Our Docs Test Themselves):

Extract inline snippets into standalone source files. Add the setup and teardown that make them executable.
Mark the documentation-visible region with snippet delineators. Bluehawk uses :snippet-start: / :snippet-end: with :remove-start: / :remove-end: for test-only code.
Inject the extracted snippet back into the published doc through an include mechanism. Options include Mintlify snippets, MDX partials, and reStructuredText includes.
Run the source files in CI and file a ticket when a run fails.

This is the same principle that fences application code: treat samples as code that must pass tests (LangChain).

Prior art¶

The pattern predates agents. Python's doctest runs interactive sessions embedded in docstrings to verify they work as shown (Python docs). sphinx.ext.doctest runs code blocks embedded in reStructuredText (Sphinx docs). pytest --doctest-glob applies the same mechanism to arbitrary text files (pytest docs).

Doctest-style tools cover single-expression examples and short REPL transcripts. They do not cover multi-step flows that spin up clients, call tools, and assert on structured output. For those longer examples, the agent-era shift makes the extract-and-test pipeline worth the setup cost, because agents that retrieve docs as context inherit every stale snippet.

Why this matters for agents¶

Agents that retrieve docs over RAG pull whatever the retriever scores highest. A stale doc that semantically matches the query still scores near the top, because relevance grading does not detect staleness (kapa.ai: RAG Gone Wrong). The agent then generates code from the stale snippet.

Runnable documentation is the upstream fix. The stale snippet never ships, because CI fails the build when its source file stops running. The pattern complements continuous documentation by preventing drift between audit runs.

Pipeline¶

graph LR
    A[Inline code in docs] --> B[Extract to src/code-samples/]
    B --> C[Add setup + teardown]
    C --> D[Mark snippet region]
    D --> E[CI runs file]
    E --> F{Passes?}
    F -->|yes| G[Extract snippet]
    G --> H[Include in published doc]
    F -->|no| I[File ticket / block merge]

Extraction is the expensive step. LangChain handed it to a docs-code-samples Deep Agents skill that moves inline code, adds setup and teardown, inserts delineators, runs the tests, and wires up the include (SKILL.md). That upfront cost is the reason most teams never start (LangChain). Agent-assisted migration is how teams adopt the pattern across an existing doc set.

CI integration¶

Two triggers cover the update cadence:

Push trigger on source or doc change: run the affected sample at once so broken snippets cannot reach main
Scheduled trigger, daily or weekly: run the full suite to catch breakage from upstream dependency updates, model-API deprecations, or third-party endpoint changes that match no commit in this repo

A failed scheduled run should open an issue tagged for the docs team, not silently fail a CI badge. This keeps the drift cost visible.

Example¶

Concrete LangChain structure from the published skill (SKILL.md):

# src/code-samples/langchain/return-a-string.py
# :snippet-start: tool-return-values-py
from langchain.tools import tool

@tool
def get_weather(city: str) -> str:
    """Get weather for a city."""
    return f"It is currently sunny in {city}."
# :snippet-end:

# :remove-start:
if __name__ == "__main__":
    result = get_weather.invoke({"city": "San Francisco"})
    assert result == "It is currently sunny in San Francisco."
    print("Tool works as expected")
# :remove-end:

The file runs as a real Python script in CI through make test-code-samples. Bluehawk strips the :remove-start: / :remove-end: block when it extracts the snippet, so the published doc shows only the tool definition. The assertion guarantees the snippet's visible behavior matches what the docs claim.

When this backfires¶

The pattern degrades or inverts under several conditions:

Small doc surface, rare API changes: extraction and CI overhead cost more than the drift caught, so a manual smoke test wins.
Non-executable content: style guides and decision records have no code to assert against, so the pipeline adds no signal.
Environment-dependent examples: snippets needing API keys or production data fail in CI without mocks, or use mocks that drift too.
Long-running or streaming flows: multi-minute runs and human-in-the-loop examples cost too much to test on every push.
Prose that paraphrases output: brittle equality assertions against paraphrased text fatigue reviewers into rubber-stamping.

Tested docs do not guarantee freshness at the retrieval layer. A RAG system indexing last week's version still returns last week's example. Runnable documentation fixes the upstream source; cache invalidation and embedding refresh are separate problems (kapa.ai: RAG Gone Wrong).

Key Takeaways¶

Every inline code example is an implicit assertion — runnable documentation promotes it to a CI-enforced one
The pipeline is extract, mark, inject, run; the expensive step is extraction, which is a good fit for agent-assisted migration
Prior art (doctest, sphinx-doctest, pytest doctest) covers single-expression examples; the agent-era extension covers multi-step tool-using examples
Stale docs feed agent RAG pipelines the wrong context; upstream testing closes that failure mode before retrieval indexes the broken snippet
Skip the pattern when the doc surface is small, examples are non-executable, or test runtime outpaces the change cadence

Continuous Documentation — agent-driven drift detection that complements upstream snippet testing
Pre-Completion Checklists — another pattern that promotes implicit assertions to explicit verification gates
Deterministic Guardrails Around Probabilistic Agents — hard CI checks as the enforcement layer for agent output
Test Harness Design for LLM Context Windows — keeping test output useful when agents consume it as context
Agent-Recorded Video Demos as a Verification Artifact — the visual modality of executable proof-of-work, for changes with a rendered UI