Skip to content

Reviewable Agent Changes: Rationale Beside the Diff

Last updated:

Evidence boundary: this page proposes a practice and ships a template whose behavior is covered by its own tests. No study cited here shows that a rationale beside the diff improves review accuracy; one controlled experiment found that explanations of generated code gave no overall benefit. Treat the agent’s rationale as a claim for the reviewer to check, not as evidence.

A reviewer of agent-written code has the diff but not the reasons. The agent had them while it worked: what the change is for, what it chose not to do, what could break. This page describes how to keep those reasons next to the hunk they explain, in the repository, and how to tell when they no longer match the code.


  1. Why the diff is not enough
  2. What an agent note records
  3. Where the note lives
  4. Anchoring and stale notes
  5. Feedback typed as a question or a change
  6. The agent-rationale template
  7. What is not established
  8. See also

Geoffrey Litt (Notion) frames understanding as the bottleneck: β€œAgents are writing tons of code for us.” Litt’s practice is an explainer the agent writes for each change, starting with background and ending with a quiz the author must pass before asking for review. Kunal’s essay Resisting Cognitive Surrender describes a custom pull request view with the diff, the agent’s running commentary on why it made each change, and a column for the reviewer. Both are individual practices, not measured interventions. They are evaluated in three talks on reviewing agent-written changes and Resisting Cognitive Surrender.

The evidence on explanations is mixed and should shape how a note is used:

  • Kaufman et al. (arXiv preprint, July 2026, 86 Python programmers, controlled experiment) found that natural-language explanations of generated assertions β€œprovided no overall benefit”, and that low-quality explanations lowered accuracy while raising confidence.
  • Gao et al. (to appear at ISSTA 2026, 34 participants, within-subjects study of AI-generated code reviews) found that full explanations gave the highest perceived trust but not the highest agreement with the AI, and that explanation level did not significantly affect review time.

Neither study tested an agent’s rationale beside its own diff. What they support is narrower: in these studies, explanations moved trust or confidence, and neither showed better detection of wrong output. A note is useful when it gives the reviewer something to check: a rejected alternative to weigh, a risk to test, a piece of evidence to rerun.

Reviewing machine output is a third mode covers automation bias and the approval check that goes with this page.


This guide proposes five fields per hunk. They come from comparing what existing tools record (local review surfaces, agent provenance traces), not from a study of which fields help reviewers.

FieldWhat to writeWhy a reviewer needs it
IntentWhat the change is for, in one sentenceThe diff shows what changed, not why (Gupta’s β€œwhy, not what” criterion)
Rejected alternativeThe option the agent considered and did not take, and whyThe reviewer’s first question is often β€œwhy not the obvious way?”
RiskWhat this could break, for whomPoints the review at the part worth reading slowly
EvidenceThe test, command or output that supports the changeSomething the reviewer can rerun instead of trusting
Open questionWhat the agent could not decideTurns a silent guess into a question for the reviewer

Five equal cards for the fields of an agent note per hunk: intent, rejected alternative, risk, evidence and open question, each with an example from the template. Unknown is a valid answer except for intent; a note is a claim to check, not evidence.

The guide’s proposal, not fields validated by a study. Prompts and selection record.

unknown is a valid answer for every field except intent. An agent that writes β€œunknown” for evidence tells the reviewer more than one that invents a test name. docket labels hunks with little evidence as β€œno evidence” and orders hunks by a risk score that includes that density.

Write the note while the reasons are still in the agent’s context, right after the commit, not reconstructed at review time. glassbox has the agent record notes with glassbox note add (no hook forces it), diffo has the agent post layers and comments through its CLI, and ai-pr-review writes its walkthrough after the fact from the diff (evaluation).


StorageExampleVisible in the merge requestThrough amend and rebaseThrough a squash merge
Git notesrefs/notes/rationale (this template), refs/notes/ai (git-ai)No, unless a tool posts itYes with notes.rewriteRef set (tested)No for plain notes (tested); git-ai’s daemon moved its note (tested)
File committed with the codeSARIF under .pr-notes/ (glassbox), a JSON sidecar if you commit it (hunk)As a changed fileMoves with the commit’s contentKept in the squashed content; line ranges may no longer match
Forge commentMerge request discussion (drift posts them)Yesdrift marks a thread outdated when the head changesDepends on the forge; not checked

Three places an agent note can live compared on four rows: stored in, visible in the merge request, amend and rebase, squash merge. Git notes are kept through amend and rebase with notes.rewriteRef and lost on a squash merge for plain notes, both tested; the file and forge cells were read from source, not run. A forge merge button rewrites history on the server, not tested.

Green cells were tested in one local repository; grey cells were read from source. Prompts and selection record.

β€œTested” refers to the runs behind git-ai: line-level AI attribution in git notes and to the template’s own tests: one local repository each, git 2.56.0. The file-based and forge rows come from reading the tools’ source, not from running them.

Git notes keep the history of the code clean and stay out of the diff, at the cost of extra setup: git config notes.rewriteRef refs/notes/rationale for amend and rebase, and an explicit git push origin refs/notes/rationale, since a plain push does not send them. A squash or rebase done by the forge’s merge button happens on the server and was not tested; see Provenance in git notes.

Do not put secrets or customer data in a note. Notes are pushed with the repository’s refs and anyone who can fetch them can read them.


A note recorded against line 42 is wrong as soon as someone adds a line above it. Anchoring by content instead of position avoids that. The template combines the two approaches read in other tools: a hash of the hunk’s content first (diffo hashes its hunks), then the annotated line’s text (glassbox re-anchors by snippet) only when that text is unique among the file’s added lines, then nothing.

Yes

No

Yes, exactly once

No, or more than once

Show notes for a commit

A hunk of the commit has

the same added-lines hash?

anchored

shown on the annotated line

The annotated line's text

is in its added lines?

reanchored

the hunk changed around the line

stale

not shown on any line,

re-check before trusting

Yes

No

Yes, exactly once

No, or more than once

Show notes for a commit

A hunk of the commit has

the same added-lines hash?

anchored

shown on the annotated line

The annotated line's text

is in its added lines?

reanchored

the hunk changed around the line

stale

not shown on any line,

re-check before trusting

The outcomes are covered by the template’s tests: lines added above the hunk keep the note anchored on its line, an edit elsewhere in the hunk re-anchors it, an edit of the annotated line marks it stale, and a line whose text appears twice is marked stale rather than guessed. A stale note is still worth reading; it records what the agent intended before the code changed.


Review feedback that returns to the agent needs a type. Without one, an agent treats every comment as an instruction. Ryan Lopopolo (OpenAI) describes giving the authoring agent β€œthe flexibility to either defer or push back against review feedback”, because β€œwithout the context that this is permissible, the coding agents are going to bias toward what they do, which is following instructions.” diffo types each reviewer thread as Change or Question, with a different reply contract for each (evaluation).

ReviewerGit repositoryClaude CodeReviewerGit repositoryClaude Codealt[Question][Change]Commit the changeRecord a note per hunk in refs/notes/rationaleRead the diff and the notes, anchored or staleWhy not the rejected alternative?Answer with evidence, or say unknownRequested changeNew commit and updated note
ReviewerGit repositoryClaude CodeReviewerGit repositoryClaude Codealt[Question][Change]Commit the changeRecord a note per hunk in refs/notes/rationaleRead the diff and the notes, anchored or staleWhy not the rejected alternative?Answer with evidence, or say unknownRequested changeNew commit and updated note

A question asks the agent to explain and may end with no code change. A requested change asks for a new commit, and the agent writes a new note for it. The template records notes; it does not carry feedback. Carry the type in whatever channel the review uses: a prefix in the forge comment, or a tool that types threads.


examples/skills/agent-rationale/ is a Claude Code skill with a standard-library Python script (3.9 or later):

Terminal window
# After committing, from the repository root
python3 -I .claude/skills/agent-rationale/scripts/rationale.py add --file src/cart.py --line 42 \
--intent "Sum prices, not item objects" \
--rejected "Give Item an __add__ method" \
--risk "Items without a price attribute now raise AttributeError" \
--evidence "tests/test_cart.py::test_total_uses_price" \
--open-question "unknown"
python3 -I .claude/skills/agent-rationale/scripts/rationale.py show

Its 15 tests run in throwaway repositories and cover the behaviors this page relies on: the note is written with all five fields, a line outside the commit’s changes is refused, the note follows amend and rebase with notes.rewriteRef and is lost on amend without it, anchoring reports anchored, reanchored or stale as drawn above, and edge cases (a line starting with ++, a form feed inside a line, duplicate lines, a note not written by the script, a path given from a subdirectory) behave as documented.

Terminal window
python3 -I examples/skills/agent-rationale/scripts/test_rationale.py

The tests were written before the code they cover and failed first; deliberate mutations of the parser and the anchoring code each made the expected test fail. A headless Claude Code session (claude -p, Claude Code 2.1.319) given the skill in a throwaway repository on November 8, 2026 recorded a note on the right line and wrote unknown for the rejected alternative, the evidence and the open question, which nothing in that repository supported. That run used the template as first committed; the current script reads its note as anchored. One session on one commit shows that the skill can be followed, not how well notes help reviewers.


  • That a rationale beside the diff improves defect detection, review time or understanding. No study found for this page tested it on agent-written diffs.
  • That the five fields are the right ones. They are this guide’s proposal.
  • How notes behave through a forge’s merge button, through cherry-pick, or when several agents annotate the same commit.
  • That an agent’s rationale is accurate. It is written by the same agent whose code is under review.