Reviewable Agent Changes: Rationale Beside the Diff
Last updated:
Evidence boundary: this page proposes a practice and ships a template whose behavior is covered by its own tests. No study cited here shows that a rationale beside the diff improves review accuracy; one controlled experiment found that explanations of generated code gave no overall benefit. Treat the agentβs rationale as a claim for the reviewer to check, not as evidence.
A reviewer of agent-written code has the diff but not the reasons. The agent had them while it worked: what the change is for, what it chose not to do, what could break. This page describes how to keep those reasons next to the hunk they explain, in the repository, and how to tell when they no longer match the code.
Table of contents
Section titled βTable of contentsβ- Why the diff is not enough
- What an agent note records
- Where the note lives
- Anchoring and stale notes
- Feedback typed as a question or a change
- The agent-rationale template
- What is not established
- See also
Why the diff is not enough
Section titled βWhy the diff is not enoughβGeoffrey Litt (Notion) frames understanding as the bottleneck: βAgents are writing tons of code for us.β Littβs practice is an explainer the agent writes for each change, starting with background and ending with a quiz the author must pass before asking for review. Kunalβs essay Resisting Cognitive Surrender describes a custom pull request view with the diff, the agentβs running commentary on why it made each change, and a column for the reviewer. Both are individual practices, not measured interventions. They are evaluated in three talks on reviewing agent-written changes and Resisting Cognitive Surrender.
The evidence on explanations is mixed and should shape how a note is used:
- Kaufman et al. (arXiv preprint, July 2026, 86 Python programmers, controlled experiment) found that natural-language explanations of generated assertions βprovided no overall benefitβ, and that low-quality explanations lowered accuracy while raising confidence.
- Gao et al. (to appear at ISSTA 2026, 34 participants, within-subjects study of AI-generated code reviews) found that full explanations gave the highest perceived trust but not the highest agreement with the AI, and that explanation level did not significantly affect review time.
Neither study tested an agentβs rationale beside its own diff. What they support is narrower: in these studies, explanations moved trust or confidence, and neither showed better detection of wrong output. A note is useful when it gives the reviewer something to check: a rejected alternative to weigh, a risk to test, a piece of evidence to rerun.
Reviewing machine output is a third mode covers automation bias and the approval check that goes with this page.
What an agent note records
Section titled βWhat an agent note recordsβThis guide proposes five fields per hunk. They come from comparing what existing tools record (local review surfaces, agent provenance traces), not from a study of which fields help reviewers.
| Field | What to write | Why a reviewer needs it |
|---|---|---|
| Intent | What the change is for, in one sentence | The diff shows what changed, not why (Guptaβs βwhy, not whatβ criterion) |
| Rejected alternative | The option the agent considered and did not take, and why | The reviewerβs first question is often βwhy not the obvious way?β |
| Risk | What this could break, for whom | Points the review at the part worth reading slowly |
| Evidence | The test, command or output that supports the change | Something the reviewer can rerun instead of trusting |
| Open question | What the agent could not decide | Turns a silent guess into a question for the reviewer |
The guideβs proposal, not fields validated by a study. Prompts and selection record.
unknown is a valid answer for every field except intent. An agent that writes βunknownβ for evidence tells the reviewer more than one that invents a test name. docket labels hunks with little evidence as βno evidenceβ and orders hunks by a risk score that includes that density.
Write the note while the reasons are still in the agentβs context, right after the commit, not reconstructed at review time. glassbox has the agent record notes with glassbox note add (no hook forces it), diffo has the agent post layers and comments through its CLI, and ai-pr-review writes its walkthrough after the fact from the diff (evaluation).
Where the note lives
Section titled βWhere the note livesβ| Storage | Example | Visible in the merge request | Through amend and rebase | Through a squash merge |
|---|---|---|---|---|
| Git notes | refs/notes/rationale (this template), refs/notes/ai (git-ai) | No, unless a tool posts it | Yes with notes.rewriteRef set (tested) | No for plain notes (tested); git-aiβs daemon moved its note (tested) |
| File committed with the code | SARIF under .pr-notes/ (glassbox), a JSON sidecar if you commit it (hunk) | As a changed file | Moves with the commitβs content | Kept in the squashed content; line ranges may no longer match |
| Forge comment | Merge request discussion (drift posts them) | Yes | drift marks a thread outdated when the head changes | Depends on the forge; not checked |
Green cells were tested in one local repository; grey cells were read from source. Prompts and selection record.
βTestedβ refers to the runs behind git-ai: line-level AI attribution in git notes and to the templateβs own tests: one local repository each, git 2.56.0. The file-based and forge rows come from reading the toolsβ source, not from running them.
Git notes keep the history of the code clean and stay out of the diff, at the cost of extra setup: git config notes.rewriteRef refs/notes/rationale for amend and rebase, and an explicit git push origin refs/notes/rationale, since a plain push does not send them. A squash or rebase done by the forgeβs merge button happens on the server and was not tested; see Provenance in git notes.
Do not put secrets or customer data in a note. Notes are pushed with the repositoryβs refs and anyone who can fetch them can read them.
Anchoring and stale notes
Section titled βAnchoring and stale notesβA note recorded against line 42 is wrong as soon as someone adds a line above it. Anchoring by content instead of position avoids that. The template combines the two approaches read in other tools: a hash of the hunkβs content first (diffo hashes its hunks), then the annotated lineβs text (glassbox re-anchors by snippet) only when that text is unique among the fileβs added lines, then nothing.
The outcomes are covered by the templateβs tests: lines added above the hunk keep the note anchored on its line, an edit elsewhere in the hunk re-anchors it, an edit of the annotated line marks it stale, and a line whose text appears twice is marked stale rather than guessed. A stale note is still worth reading; it records what the agent intended before the code changed.
Feedback typed as a question or a change
Section titled βFeedback typed as a question or a changeβReview feedback that returns to the agent needs a type. Without one, an agent treats every comment as an instruction. Ryan Lopopolo (OpenAI) describes giving the authoring agent βthe flexibility to either defer or push back against review feedbackβ, because βwithout the context that this is permissible, the coding agents are going to bias toward what they do, which is following instructions.β diffo types each reviewer thread as Change or Question, with a different reply contract for each (evaluation).
A question asks the agent to explain and may end with no code change. A requested change asks for a new commit, and the agent writes a new note for it. The template records notes; it does not carry feedback. Carry the type in whatever channel the review uses: a prefix in the forge comment, or a tool that types threads.
The agent-rationale template
Section titled βThe agent-rationale templateβexamples/skills/agent-rationale/ is a Claude Code skill with a standard-library Python script (3.9 or later):
# After committing, from the repository rootpython3 -I .claude/skills/agent-rationale/scripts/rationale.py add --file src/cart.py --line 42 \ --intent "Sum prices, not item objects" \ --rejected "Give Item an __add__ method" \ --risk "Items without a price attribute now raise AttributeError" \ --evidence "tests/test_cart.py::test_total_uses_price" \ --open-question "unknown"
python3 -I .claude/skills/agent-rationale/scripts/rationale.py showIts 15 tests run in throwaway repositories and cover the behaviors this page relies on: the note is written with all five fields, a line outside the commitβs changes is refused, the note follows amend and rebase with notes.rewriteRef and is lost on amend without it, anchoring reports anchored, reanchored or stale as drawn above, and edge cases (a line starting with ++, a form feed inside a line, duplicate lines, a note not written by the script, a path given from a subdirectory) behave as documented.
python3 -I examples/skills/agent-rationale/scripts/test_rationale.pyThe tests were written before the code they cover and failed first; deliberate mutations of the parser and the anchoring code each made the expected test fail. A headless Claude Code session (claude -p, Claude Code 2.1.319) given the skill in a throwaway repository on November 8, 2026 recorded a note on the right line and wrote unknown for the rejected alternative, the evidence and the open question, which nothing in that repository supported. That run used the template as first committed; the current script reads its note as anchored. One session on one commit shows that the skill can be followed, not how well notes help reviewers.
What is not established
Section titled βWhat is not establishedβ- That a rationale beside the diff improves defect detection, review time or understanding. No study found for this page tested it on agent-written diffs.
- That the five fields are the right ones. They are this guideβs proposal.
- How notes behave through a forgeβs merge button, through cherry-pick, or when several agents annotate the same commit.
- That an agentβs rationale is accurate. It is written by the same agent whose code is under review.
See also
Section titled βSee alsoβ- Multi-provider code review: local review surfaces: diffo, glassbox, hunk, drift, ai-pr-review, guidedreview compared
- Provenance in git notes: git-ai, AgentNote and docket, and what survives a rewrite
- Approve only what you can explain: a check before approving agent output
- Three places an agent change gets explained

