Reference, read on 2026-09-30

Token-saving tools for coding agents: claims and third-party studies

What 45 projects claim, and what 8 public benchmarks and 15 arXiv preprints measured, with a link to every source.

tools catalogued: 45
public benchmarks: 8
arXiv papers: 15

Disclosure. Florian Bruniaux, who maintains this site, is a core contributor to RTK, one of the tools below. This page does not recommend a tool. It sets out what each project claims and what third parties measured, with links to every source, and reports RTK's results as measured, including the unfavourable ones. Tools enter the catalogue through a fixed rule stated there, not by preference.

Catalogue of token-saving tools

45 projects, each with its own claim and any third-party result found. Open a row for the full claim, every result and the limits.

How tools are listed

A project is listed when it is public, not archived, aimed at coding agents, and has at least 100 GitHub stars, or was measured by one of the benchmarks below, or is already covered by the guide. Stars, license and last push come from the GitHub API; claims were read in each project's README or product page on 2026-09-30. The list is sorted by name by default. GitHub stars measure attention, not quality. Each result comes from one study: see Claims and studies before reading a number.

Search tools

Mechanism

Sort by

Measured by a third party (12)

Also found, below the inclusion bar (37 projects, claims not reviewed)

How to read the results

Two quantities: Vendor claims measure one layer: shell output, output tokens, or a single content type. Benchmarks measure the whole task, so the two numbers rarely describe the same quantity.
Dated models: The benchmarks used claude-sonnet-4-6, claude-sonnet-5, GPT-5.6 Sol and an unnamed Claude Sonnet; the Codepointer replay applied Opus 4.8 prices. None tested Claude Sonnet 5.5, Claude Opus 5.5 or the GPT-6 family, so read these results as dated evidence about the mechanism, not as current rankings.
Who publishes: Three benchmarks come from a vendor with a stake: Dasein makes Parsec, the THOL maintainer makes Tokenade, and the Tura author builds a competing agent. Each says so, and the benchmark table below shows it.
Sign convention: Negative means cheaper or fewer tokens than the tool-free baseline. Positive means more expensive or more tokens.

Results by tool

Every third-party result from the 8 benchmarks, one row per tool. Rows are sorted by name, not by outcome. THOL publishes the opposite sign convention; its figures are converted here.

Cost change
Token-count change
Left of zero: cheaper or fewer tokens than the tool-free baseline
Right of zero: more expensive or more tokens

Caveman 6 results cost −19% to +9%output tokens −8.5%
- Dasein Labs ↗ Total cost −19% (19% cheaper; cost) 58 resolved.
- Stet.sh ↗ Workload cost, run 1 then run 2 +9% then −12% (9% more expensive, then 12% cheaper; cost)
- JetBrains (Caveman) ↗ Output tokens −8.5% (8.5% fewer output tokens; output tokens) Forced activation, so a ceiling. Quality 8 better, 10 worse, 64 equal.
- THOL ↗ End-to-end cost, 7 long tasks −13.6% (13.6% cheaper; cost) Interval 30.2% cheaper to 3.6% more expensive.
- Tura ↗ Modeled cost, one task, 2 runs −3.9% (3.9% cheaper; cost) Score 86.5% vs 78.9% baseline.
- Codepointer ↗ Share of replayed spend saved −0.4% (0.4% cheaper; cost) Replay, not a live run.
claude-token-efficient 1 result cost −11.6%
- THOL ↗ End-to-end cost, 7 long tasks −11.6% (11.6% cheaper; cost) Interval 24.2% cheaper to 1.4% more expensive.
code-review-graph 1 result cost −8.5%
- THOL ↗ End-to-end cost, 7 long tasks −8.5% (8.5% cheaper; cost) THOL reports that the agent never used the tool, so the gap is not a tool effect.
CodeGraph 1 result cost −7.6%
- THOL ↗ End-to-end cost, 7 long tasks −7.6% (7.6% cheaper; cost) Interval 21.6% cheaper to 8.7% more expensive. Opt-in tool.
Context Mode 1 result cost +33% to +72%
- Stet.sh ↗ Workload cost, run 1 then run 2 +72% then +33% (72% more expensive, then 33% more expensive; cost)
Edgee 1 result cost −14.7%
- THOL ↗ End-to-end cost, 7 long tasks −14.7% (14.7% cheaper; cost) Interval 37.1% cheaper to 11.1% more expensive. THOL flags no token accounting for this tool.
Fermat 1 result cost −22%
- Dasein Labs ↗ Total cost −22% (22% cheaper; cost) Run on 2026-09-21, not paired with the other arms. 55 resolved.
Graphify 2 results cost −3.3% to +1%
- Marmelab ↗ Cost on the large transformation +1% (1% more expensive; cost)
- THOL ↗ End-to-end cost, 7 long tasks −3.3% (3.3% cheaper; cost) THOL reports that the agent never used the tool, so the gap is not a tool effect.
Headroom 4 results cost −4% to +52.8%
- Dasein Labs ↗ Total cost +44% (44% more expensive; cost) 58 resolved.
- Marmelab ↗ Cost per operation, cache mode then token mode −4% then +32% (4% cheaper, then 32% more expensive; cost) One task, three runs. The authors call -4% noise.
- THOL ↗ End-to-end cost, 7 long tasks +52.8% (52.8% more expensive; cost) Interval 10.2% to 120.2% more expensive, entirely above zero.
- Codepointer ↗ Share of replayed spend saved −2.8% (2.8% cheaper; cost) Replay, not a live run. Median 54% cut on the grep and diff output it touches.
lean-ctx 1 result cost +10.3%
- THOL ↗ End-to-end cost, 7 long tasks +10.3% (10.3% more expensive; cost) Interval 12.6% cheaper to 46.4% more expensive.
LSP (code navigation) 1 result cost −13%
- Marmelab ↗ Cost on the large transformation −13% (13% cheaper; cost)
Ponytail 3 results cost −8.9% to +20%
- Stet.sh ↗ Workload cost, run 1 then run 2 +20% then −2% (20% more expensive, then 2% cheaper; cost) Tests 1 win, 4 losses, 15 ties.
- THOL ↗ End-to-end cost, 7 long tasks −3.5% (3.5% cheaper; cost) Interval 18.1% cheaper to 12.4% more expensive.
- Tura ↗ Modeled cost, one task, 2 runs −8.9% (8.9% cheaper; cost) Score 80.8% vs 78.9% baseline. The two runs spread by 51.7% of their mean.
RTK 6 results cost −9% to +13%
- Dasein Labs ↗ Total cost +13% (13% more expensive; cost) 54 resolved. Best cache hit rate (97.3%) but 6,131 agent steps vs 5,325.
- Stet.sh ↗ Workload cost, run 1 then run 2 +13% then −9% (13% more expensive, then 9% cheaper; cost)
- JetBrains (RTK) ↗ Median cost per task, low effort then high effort +7.6% then +0.1% (7.6% more expensive, then 0.1% more expensive; cost) Low effort p=0.004; high effort p=0.99. Quality unchanged.
- THOL ↗ End-to-end cost, 7 long tasks +7.1% (7.1% more expensive; cost) Interval 7.2% cheaper to 26.0% more expensive.
- Tura ↗ Modeled cost, one task, 2 runs +7.2% (7.2% more expensive; cost) Score 76.9% vs 78.9% baseline, 44% more agent rounds. The author says the data do not identify a plugin effect.
- Codepointer ↗ Share of replayed spend saved −0.5% (0.5% cheaper; cost) Replay, not a live run. 33% to 99% cut on the shell output it recognizes.
squeez 1 result cost −6.9%
- THOL ↗ End-to-end cost, 7 long tasks −6.9% (6.9% cheaper; cost) Interval 26.1% cheaper to 18.4% more expensive.
Woz 1 result cost −13%
- Dasein Labs ↗ Total cost −13% (13% cheaper; cost) 55 resolved, wall-clock +23%.

Dashed line: Switch to GPT-5.6 Terra xhigh, Stet.sh, workload cost, run 1 then run 2: −49% then −49% (49% cheaper, then 49% cheaper). A model change, not a tool: the only repeated cost drop. Drawn for scale, not as a tool row.

Data table: every result as text

Third-party results by tool, study and metric
Tool	Study	Metric	Result	Unit
Caveman	Dasein Labs	Total cost	−19% (19% cheaper)	cost
Caveman	Stet.sh	Workload cost, run 1 then run 2	+9% then −12% (9% more expensive, then 12% cheaper)	cost
Caveman	JetBrains (Caveman)	Output tokens	−8.5% (8.5% fewer output tokens)	output tokens
Caveman	THOL	End-to-end cost, 7 long tasks	−13.6% (13.6% cheaper)	cost
Caveman	Tura	Modeled cost, one task, 2 runs	−3.9% (3.9% cheaper)	cost
Caveman	Codepointer	Share of replayed spend saved	−0.4% (0.4% cheaper)	cost
claude-token-efficient	THOL	End-to-end cost, 7 long tasks	−11.6% (11.6% cheaper)	cost
code-review-graph	THOL	End-to-end cost, 7 long tasks	−8.5% (8.5% cheaper)	cost
CodeGraph	THOL	End-to-end cost, 7 long tasks	−7.6% (7.6% cheaper)	cost
Context Mode	Stet.sh	Workload cost, run 1 then run 2	+72% then +33% (72% more expensive, then 33% more expensive)	cost
Edgee	THOL	End-to-end cost, 7 long tasks	−14.7% (14.7% cheaper)	cost
Fermat	Dasein Labs	Total cost	−22% (22% cheaper)	cost
Graphify	Marmelab	Cost on the large transformation	+1% (1% more expensive)	cost
Graphify	THOL	End-to-end cost, 7 long tasks	−3.3% (3.3% cheaper)	cost
Headroom	Dasein Labs	Total cost	+44% (44% more expensive)	cost
Headroom	Marmelab	Cost per operation, cache mode then token mode	−4% then +32% (4% cheaper, then 32% more expensive)	cost
Headroom	THOL	End-to-end cost, 7 long tasks	+52.8% (52.8% more expensive)	cost
Headroom	Codepointer	Share of replayed spend saved	−2.8% (2.8% cheaper)	cost
lean-ctx	THOL	End-to-end cost, 7 long tasks	+10.3% (10.3% more expensive)	cost
LSP (code navigation)	Marmelab	Cost on the large transformation	−13% (13% cheaper)	cost
Ponytail	Stet.sh	Workload cost, run 1 then run 2	+20% then −2% (20% more expensive, then 2% cheaper)	cost
Ponytail	THOL	End-to-end cost, 7 long tasks	−3.5% (3.5% cheaper)	cost
Ponytail	Tura	Modeled cost, one task, 2 runs	−8.9% (8.9% cheaper)	cost
RTK	Dasein Labs	Total cost	+13% (13% more expensive)	cost
RTK	Stet.sh	Workload cost, run 1 then run 2	+13% then −9% (13% more expensive, then 9% cheaper)	cost
RTK	JetBrains (RTK)	Median cost per task, low effort then high effort	+7.6% then +0.1% (7.6% more expensive, then 0.1% more expensive)	cost
RTK	THOL	End-to-end cost, 7 long tasks	+7.1% (7.1% more expensive)	cost
RTK	Tura	Modeled cost, one task, 2 runs	+7.2% (7.2% more expensive)	cost
RTK	Codepointer	Share of replayed spend saved	−0.5% (0.5% cheaper)	cost
squeez	THOL	End-to-end cost, 7 long tasks	−6.9% (6.9% cheaper)	cost
Switch to GPT-5.6 Terra xhigh	Stet.sh	Workload cost, run 1 then run 2	−49% then −49% (49% cheaper, then 49% cheaper)	cost
Woz	Dasein Labs	Total cost	−13% (13% cheaper)	cost

Where tool tokens come from

Why compressing one tool's output moves the total by less than the output figure suggests.

How each benchmark was run

Eight checks per benchmark, plus who published it and what they stand to gain.

Read on 2026-09-30. "Not stated" means the publisher's page did not say, not that the answer is no. The tally counts the checks answered yes; it says nothing about the size of any result.

Quality checks for each benchmark
Benchmark	Paired against a baseline	Task success measured	Cache reported separately	Dollar cost	Versions named	More than 10 tasks	Repeated runs	Code published
Code-Compression Bench ↗ Dasein Labs 2026-07-04 (Fermat arm 2026-09-21, not paired) Yes on 6 of 8 checks	Yes	Yes	Yes	Yes	No	Yes	No	Yes
Publisher's interest: Sponsored and operated by Dasein Labs, which makes Parsec, one of the arms. The Parsec arm also injects a turn-0 brief and a stop decision, beyond compression. Method: 100 SWE-bench Verified tasks, headless Claude Code via the Python Agent SDK, claude-sonnet-4-6, official Docker grader, one run per arm, cache-aware cost.
I Tested Six Ways to Save $$$ on 5.6 Sol ↗ Stet.sh (author not named) 2026-07-20, updated 2026-07-22 Yes on 4 of 8 checks	Yes	Yes	Not stated	Yes	No	No	Yes	Not stated
Publisher's interest: The author builds Stet.sh, the evaluation tool used. No compared tool is theirs. Method: 10 merged changes from one production repo, Codex CLI with GPT-5.6 Sol at medium effort as baseline, 7 arms, 2 repetitions, 140 runs.
Cutting the Coding Agent Bill ↗ Marmelab (Erwan Bourlon, Jérôme Piernot) 2026-08-27 Yes on 3 of 8 checks	Yes	No	Not stated	Yes	No	No	Yes	No
Publisher's interest: Marmelab builds Atomic CRM Builder, the workload. No compared tool is theirs. Method: Atomic CRM Builder tasks, 1 to 4 tasks per tool, 3 to 4 runs per configuration. Harness, model and tool versions not stated.
RTK and Claude Code token savings ↗ JetBrains AI blog (Denis Shiryaev) July 2026 Yes on 7 of 8 checks	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Not stated
Publisher's interest: No affiliation with RTK stated. Method: 86 SkillsBench tasks, 425 billed trials, Harbor 0.18, Claude Code 2.1.201 headless, rtk 0.43.0, claude-sonnet-5, low and high effort, per-task medians with a Wilcoxon test. RTK AI Labs published a response on 2026-08-07 that questions the one-run-per-task design and reports -4.8% on 13 dev tasks (p = 0.305).
Speaking to AI agents like cavemen saves 65% of tokens. We test. ↗ JetBrains AI blog (Denis Shiryaev) July 2026 Yes on 5 of 8 checks	Yes	Yes	Not stated	Yes	No	Yes	Yes	Not stated
Publisher's interest: No affiliation with Caveman stated. Method: 86 SkillsBench tasks, 82 clean pairs, about 240 billed trials, Harbor 0.17, claude-sonnet-5 at low effort, Caveman forcibly activated (its best case).
Token-Harness Optimizer Leaderboard ↗ Paul Irolla Claude Code 2.1.206 campaign, results committed 2026-07-18 Yes on 7 of 8 checks	Yes	Yes	Not stated	Yes	Yes	Yes	Yes	Yes
Publisher's interest: Maintained by the author of Tokenade, which tops the table. The page says so. Method: Headless Claude Code 2.1.206 with claude-sonnet-4-6, 17 tasks, 10 runs per task and tool. The headline keeps the 7 tasks where the tool-free control used at least 200,000 tokens (70 runs per tool) and reports the ratio of end-to-end USD with bootstrap 95% intervals.
Token-saving plugins: measure cost per completed task ↗ Yohji Sakamoto, maintainer of the Tura coding agent 2026-07-19, data updated 2026-08-15 Yes on 7 of 8 checks	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes
Publisher's interest: Published by the maintainer of Tura, a coding agent that competes with the harnesses these plugins extend. The author says so and calls the post not an independent review. Method: One task (rewrite the Rust eza tool in Python, 52 assertions), Codex CLI 0.144.1 with gpt-5.6-sol at high reasoning, 2 runs per arm against matched baseline runs. Cost is modeled from token counts at list prices, not billed.
Cutting LLM token costs with rtk, headroom, and caveman ↗ Codepointer (Yongkyun) 2026-06-18 Yes on 3 of 8 checks	No	No	Yes	Yes	Not stated	Yes	No	Not stated
Publisher's interest: No relationship with the three projects stated. Method: Counterfactual replay of 500 sessions sampled from the author's own Claude Code history (614M tokens, $926.31 at list prices). Headroom was applied to the recorded payloads; RTK and Caveman at their published per-stream rates. No live runs, task success not measured.

Code-Compression Bench 6 of 8 checks
Dasein Labs, 2026-07-04 (Fermat arm 2026-09-21, not paired)
- Paired against a baseline: Yes
- Task success measured: Yes
- Cache reported separately: Yes
- Dollar cost: Yes
- Versions named: No
- More than 10 tasks: Yes
- Repeated runs: No
- Code published: Yes
Publisher's interest: Sponsored and operated by Dasein Labs, which makes Parsec, one of the arms. The Parsec arm also injects a turn-0 brief and a stop decision, beyond compression.

Method: 100 SWE-bench Verified tasks, headless Claude Code via the Python Agent SDK, claude-sonnet-4-6, official Docker grader, one run per arm, cache-aware cost.
Read the benchmark ↗
I Tested Six Ways to Save $$$ on 5.6 Sol 4 of 8 checks
Stet.sh (author not named), 2026-07-20, updated 2026-07-22
- Paired against a baseline: Yes
- Task success measured: Yes
- Cache reported separately: Not stated
- Dollar cost: Yes
- Versions named: No
- More than 10 tasks: No
- Repeated runs: Yes
- Code published: Not stated
Publisher's interest: The author builds Stet.sh, the evaluation tool used. No compared tool is theirs.

Method: 10 merged changes from one production repo, Codex CLI with GPT-5.6 Sol at medium effort as baseline, 7 arms, 2 repetitions, 140 runs.
Read the benchmark ↗
Cutting the Coding Agent Bill 3 of 8 checks
Marmelab (Erwan Bourlon, Jérôme Piernot), 2026-08-27
- Paired against a baseline: Yes
- Task success measured: No
- Cache reported separately: Not stated
- Dollar cost: Yes
- Versions named: No
- More than 10 tasks: No
- Repeated runs: Yes
- Code published: No
Publisher's interest: Marmelab builds Atomic CRM Builder, the workload. No compared tool is theirs.

Method: Atomic CRM Builder tasks, 1 to 4 tasks per tool, 3 to 4 runs per configuration. Harness, model and tool versions not stated.
Read the benchmark ↗
RTK and Claude Code token savings 7 of 8 checks
JetBrains AI blog (Denis Shiryaev), July 2026
- Paired against a baseline: Yes
- Task success measured: Yes
- Cache reported separately: Yes
- Dollar cost: Yes
- Versions named: Yes
- More than 10 tasks: Yes
- Repeated runs: Yes
- Code published: Not stated
Publisher's interest: No affiliation with RTK stated.

Method: 86 SkillsBench tasks, 425 billed trials, Harbor 0.18, Claude Code 2.1.201 headless, rtk 0.43.0, claude-sonnet-5, low and high effort, per-task medians with a Wilcoxon test. RTK AI Labs published a response on 2026-08-07 that questions the one-run-per-task design and reports -4.8% on 13 dev tasks (p = 0.305).
Read the benchmark ↗
Speaking to AI agents like cavemen saves 65% of tokens. We test. 5 of 8 checks
JetBrains AI blog (Denis Shiryaev), July 2026
- Paired against a baseline: Yes
- Task success measured: Yes
- Cache reported separately: Not stated
- Dollar cost: Yes
- Versions named: No
- More than 10 tasks: Yes
- Repeated runs: Yes
- Code published: Not stated
Publisher's interest: No affiliation with Caveman stated.

Method: 86 SkillsBench tasks, 82 clean pairs, about 240 billed trials, Harbor 0.17, claude-sonnet-5 at low effort, Caveman forcibly activated (its best case).
Read the benchmark ↗
Token-Harness Optimizer Leaderboard 7 of 8 checks
Paul Irolla, Claude Code 2.1.206 campaign, results committed 2026-07-18
- Paired against a baseline: Yes
- Task success measured: Yes
- Cache reported separately: Not stated
- Dollar cost: Yes
- Versions named: Yes
- More than 10 tasks: Yes
- Repeated runs: Yes
- Code published: Yes
Publisher's interest: Maintained by the author of Tokenade, which tops the table. The page says so.

Method: Headless Claude Code 2.1.206 with claude-sonnet-4-6, 17 tasks, 10 runs per task and tool. The headline keeps the 7 tasks where the tool-free control used at least 200,000 tokens (70 runs per tool) and reports the ratio of end-to-end USD with bootstrap 95% intervals.
Read the benchmark ↗
Token-saving plugins: measure cost per completed task 7 of 8 checks
Yohji Sakamoto, maintainer of the Tura coding agent, 2026-07-19, data updated 2026-08-15
- Paired against a baseline: Yes
- Task success measured: Yes
- Cache reported separately: Yes
- Dollar cost: Yes
- Versions named: Yes
- More than 10 tasks: No
- Repeated runs: Yes
- Code published: Yes
Publisher's interest: Published by the maintainer of Tura, a coding agent that competes with the harnesses these plugins extend. The author says so and calls the post not an independent review.

Method: One task (rewrite the Rust eza tool in Python, 52 assertions), Codex CLI 0.144.1 with gpt-5.6-sol at high reasoning, 2 runs per arm against matched baseline runs. Cost is modeled from token counts at list prices, not billed.
Read the benchmark ↗
Cutting LLM token costs with rtk, headroom, and caveman 3 of 8 checks
Codepointer (Yongkyun), 2026-06-18
- Paired against a baseline: No
- Task success measured: No
- Cache reported separately: Yes
- Dollar cost: Yes
- Versions named: Not stated
- More than 10 tasks: Yes
- Repeated runs: No
- Code published: Not stated
Publisher's interest: No relationship with the three projects stated.

Method: Counterfactual replay of 500 sessions sampled from the author's own Claude Code history (614M tokens, $926.31 at list prices). Headroom was applied to the recorded payloads; RTK and Caveman at their published per-stream rates. No live runs, task success not measured.
Read the benchmark ↗

What research papers found

Studies that measure token reduction against cost, success or time, rather than tokens alone.

arXiv preprints, read at their abstract pages on 2026-09-30. Preprints are not peer-reviewed. The first group studies cost without selling a method. In the second group the authors propose the method they measure, which is normal in research but is a conflict to keep in mind.

Studies of cost and behavior (9)

Token Reduction Is Not Cost Reduction ↗ Cost billed or logged

The largest compression setup cut delivered tool-output tokens by 38.4% and raised billed cost by 6.8%. Token reduction and cost reduction correlated weakly (Pearson r = 0.15).
Weinberger, Hozez · arXiv 2607.12161 · 2026-07-13, v5 2026-08-12
Beyond Token Savings: A Systematic Study of Context Compression in LLM Agents ↗ Cost estimated

Nearly 35,000 agent runs on SWE-bench Verified and Terminal-Bench. On Terminal-Bench with Qwen, policies using about one third of the tokens took 20% to 80% longer than the uncompressed agent.
Satish, Sinha, Kawada, Yadwadkar · arXiv 2609.32961 · 2026-09-26
CAVEWOMAN: How LLMs Behave Under Linguistic Input and Output Compression ↗ Cost billed or logged

Output compression cut realized cost 1.4x to 2.4x per model on most API models. Input compression raised net cost (about 1.15x on the five-benchmark mean). Single generations, not an agent loop.
Adeyemi, Rossi, Dernoncourt · arXiv 2606.24083 · 2026-06-23
Don't Break the Cache: Prompt Caching for Long-Horizon Agentic Tasks ↗ Cost billed or logged

Prompt caching reduced API costs by 41% to 80% and time to first token by 13% to 31% across providers, over 500 agent sessions.
Lumer, Nizar, Jangiti et al. · arXiv 2601.06007 · 2026-01-09, v2 2026-01-31
Agentic Coding in the Wild: GitHub Copilot Traces at Production Scale ↗ Workload study

June 2026 traces (3.2M users, 13M sessions, 761M LLM calls): cache hit rates average 90% within a turn and fall to 55% across turn boundaries.
Liu, Qiu, Goiri, Fonseca, Bianchini, Choukse · arXiv 2608.00101 · 2026-07-30
Can your AI agent be cheaper? Task specifications and token spend ↗ Tokens only

Across 2,700 runs with Kimi K3, reducing a full task specification to a bare user story raised token spend by 29.7%.
Smekal · arXiv 2608.25399 · 2026-08-26
When Does Restricting a Coding Agent to execute_code Help? ↗ Cost billed or logged

A code-only tool set was cheaper than or tied with the cheapest tool-rich arm in three of four cells; on SWE-bench with Claude it was 14.4% costlier, not significant. Pass rates tied.
Yang, Yu, Desell · arXiv 2607.10569 · 2026-07-12
What Does Context Compression Cost an Agent? ↗ Workload study

Completion stayed flat (80% to 85%, p = 1.0) while retrieval calls rose from 21.0 to 63.9 (p = .002): compression moved work into extra tool calls.
Liu · arXiv 2608.16370 · 2026-08-17
Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering ↗ Tokens only

On 30 ChatDev tasks with GPT-5, the code review stage used 59.4% of tokens; input tokens were 53.9% of the total.
Salim, Latendresse, Khatoonabadi, Shihab · arXiv 2601.14470 · 2026-01-20

Methods measured by their own authors (6)

Reducing Cost of LLM Agents with Trajectory Reduction (AgentDiet) ↗ Cost estimated

Input tokens fell 39.9% to 59.7% and total computational cost 21.1% to 35.9% at the same agent performance.
Xiao, Gao, Peng, Xiong · arXiv 2509.23586 · 2025-09-28, v2 2026-03-15
CoACT: Action-Preserving Observation Compression for Coding Agents ↗ Tokens only

Average total token consumption fell 33.0% on SWE-bench Verified with success close to the uncompressed agent. No dollar figure.
Chen, Zhu, Zhang, Li · arXiv 2607.02911 · 2026-07-03
SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents ↗ Tokens only

23% to 54% token reduction on agent tasks such as SWE-bench Verified, with success rates the authors report as equal or better.
Wang, Shi, Yang et al. · arXiv 2601.16746 · 2026-01-23, v4 2026-05-07
ContextSniper: AntTrail's Token-Efficient Code Memory ↗ Cost estimated

For OpenClaw on SWE-bench Lite, total tokens fell 51.5% and logged cost 36.4%; resolution rates differ by one task out of 50.
Luk, Najafi, Jia, Yang et al. · arXiv 2607.01916 · 2026-07-02, v6 2026-09-14
An Empirical Cost Attribution of Context-Compression Gateways in Multi-Turn Coding Agents ↗ Cost estimated

Content compression saved about 2% of the cache-priced prefix per turn; the authors say their single-shot benchmark must not be cited as a cost-saving argument.
Chen, Shi · arXiv 2609.22114 · v1 listed as 2026-08-19
Paritok-4B: Intent-Conditioned Context Compression for Coding Agents ↗ Cost estimated

Compressed agent context to 25.7% of its size while retaining 86.5% of uncompressed single-shot solve quality on 300 SWE-bench Lite instances.
Shi, Chen · arXiv 2608.24188 · 2026-08-25

Claims and third-party studies

8 tools where a project claim and third-party results can be read side by side, with every result found for each tool.

Before reading these comparisons. These are not benchmarks run by this site. Each result comes from one published study, and each study is one among several. Results depend on the harness, the model, the effort setting, the task set and the date of the run. A project's claim and a study's result often measure different quantities, such as shell output against total task cost. A single result does not show that a tool works or fails on your workload. Open each source, check what it measured, and test on your own tasks.

Caveman

Project claim

50% fewer (median)

of output tokens on 10 dev questions against an 'Answer concisely.' control, skill only. The earlier 65% (v1.9.1 release notes) is no longer in the README
Project source ↗
Third-party results (6)
- Dasein Labs ↗ Total cost (cost) −19% (19% cheaper)
- Stet.sh ↗ Workload cost, run 1 then run 2 (cost) +9% then −12% (9% more expensive, then 12% cheaper)
- JetBrains (Caveman) ↗ Output tokens (output tokens) −8.5% (8.5% fewer output tokens)
- THOL ↗ End-to-end cost, 7 long tasks (cost) −13.6% (13.6% cheaper)
- Tura ↗ Modeled cost, one task, 2 runs (cost) −3.9% (3.9% cheaper)
- Codepointer ↗ Share of replayed spend saved (cost) −0.4% (0.4% cheaper)
RTK
Different denominator

Project claim

up to 90% fewer

of bash output the agent reads, per the README, which adds that this is not the same as cutting the bill by 90%. The repository description says 60-90% of LLM token consumption on common dev commands
Project source ↗
Third-party results (6)
- Dasein Labs ↗ Total cost (cost) +13% (13% more expensive)
- Stet.sh ↗ Workload cost, run 1 then run 2 (cost) +13% then −9% (13% more expensive, then 9% cheaper)
- JetBrains (RTK) ↗ Median cost per task, low effort then high effort (cost) +7.6% then +0.1% (7.6% more expensive, then 0.1% more expensive)
- THOL ↗ End-to-end cost, 7 long tasks (cost) +7.1% (7.1% more expensive)
- Tura ↗ Modeled cost, one task, 2 runs (cost) +7.2% (7.2% more expensive)
- Codepointer ↗ Share of replayed spend saved (cost) −0.5% (0.5% cheaper)
The claim and the results use different denominators. Each number answers a different question.
Maintainer response
- RTK AI Labs, "RTK on SkillsBench: What the Benchmark Measures" (vendor analysis of the JetBrains study) ↗ 2026-08-07
- RTK maintainers on the JetBrains study (aeppling, issue #3157) ↗ 2026-07-23
Ponytail

Project claim

~20% cheaper

of cost, 12 tasks, Haiku 4.5, n=4
Project source ↗
Third-party results (3)
- Stet.sh ↗ Workload cost, run 1 then run 2 (cost) +20% then −2% (20% more expensive, then 2% cheaper)
- THOL ↗ End-to-end cost, 7 long tasks (cost) −3.5% (3.5% cheaper)
- Tura ↗ Modeled cost, one task, 2 runs (cost) −8.9% (8.9% cheaper)
Headroom
Different denominator

Project claim

20% fewer

of tokens for coding agents, per the repository description (its README's four seeded scenarios show 21% to 57%)
Project source ↗
Third-party results (4)
- Dasein Labs ↗ Total cost (cost) +44% (44% more expensive)
- Marmelab ↗ Cost per operation, cache mode then token mode (cost) −4% then +32% (4% cheaper, then 32% more expensive)
- THOL ↗ End-to-end cost, 7 long tasks (cost) +52.8% (52.8% more expensive)
- Codepointer ↗ Share of replayed spend saved (cost) −2.8% (2.8% cheaper)
The claim and the results use different denominators. Each number answers a different question.
Context Mode
Different denominator

Project claim

98% reduction

of bytes of raw tool output kept out of the context window, on the vendor's scenarios
Project source ↗
Third-party results (1)
- Stet.sh ↗ Workload cost, run 1 then run 2 (cost) +72% then +33% (72% more expensive, then 33% more expensive)
The claim and the results use different denominators. Each number answers a different question.
CodeGraph

Project claim

44% cheaper

of cost on 7 repos, one architecture question each, Opus 4.8, median of 4 runs
Project source ↗
Third-party results (1)
- THOL ↗ End-to-end cost, 7 long tasks (cost) −7.6% (7.6% cheaper)
lean-ctx
Different denominator

Project claim

99.4% saving

of input-side cost on a replayed 72-turn session, not a live task
Project source ↗
Third-party results (1)
- THOL ↗ End-to-end cost, 7 long tasks (cost) +10.3% (10.3% more expensive)
The claim and the results use different denominators. Each number answers a different question.
Edgee

Project claim

up to 70% off

of the token bill, combining routing, compression and visibility; no measurement stated
Project source ↗
Third-party results (1)
- THOL ↗ End-to-end cost, 7 long tasks (cost) −14.7% (14.7% cheaper)

Sources and further reading

Examined but not used

tokbench: a pilot with one bug-fix task, one run per arm, on GLM models through the author's own harness. It measures input tokens, not cost. Its author discloses owning the harness and using lean-ctx daily.
tokenme: its own repository marks the 92.7% result as withdrawn because the outputs were written by hand.
Project-run benchmarks: kept in the catalogue under each project, labelled as the project's own benchmark, not counted as third-party measurements.
Blog posts and aggregator lists that repeat a figure without a primary source: used as leads only.

Keep reading

Independent benchmarks, in the guide → The written analysis behind this page, with sources.
FinOps levers map → Where the "tool output compression" lever sits among the other cost levers.
AI FinOps: the lever map → The guide section that maps each cost lever to its regime and its limits.
LLM market snapshot → Current model prices, which set the size of every percentage above.
How to read a token-compression benchmark → Blog article: denominators, paired tasks, cache costs and success criteria, with Tokenade versus RTK as a worked example.

Catalogue of token-saving tools

How to read the results

Results by tool

Where tool tokens come from

How each benchmark was run

What research papers found

Studies of cost and behavior (9)

Claims and third-party studies

Caveman

RTK

Ponytail

Headroom

Context Mode

CodeGraph

lean-ctx

Edgee

Sources and further reading

Examined but not used

Keep reading