Benchmark

The numbers on the landing page come from a scripted-agent benchmark. This page documents the protocol, the results, and what the data does — and does not — support.

Protocol

• Driver: the pi coding agent, running each scripted task end-to-end with tokens billed via OpenRouter.
• Tasks: four real tasks against real repositories — three source files ranging from 51 KB to 96 KB, plus one 1,914-line build log.
• Arms:
  • base — the agent’s built-in read + bash tools.
  • cliskill — ctxctl driven through the agent skill (CLI calls).
  • mcp — ctxctl exposed as five native MCP tools (ctxctl_outline, ctxctl_symbol, ctxctl_read, ctxctl_deps, ctxctl_exec).
• Model: deepseek-v4-flash via OpenRouter, single run per arm.

Results

Measurement	Result
Native ctxctl mcp tools vs built-in read/bash — session cost	−33%
Same benchmark — uncached (fully billed) input tokens	−31%
Log-analysis task via ctxctl exec	−84% cost
Exploring a 96 KB file through outline/symbol slices	−86% uncached input
File outlines vs whole-file reads	91–95% smaller

The baseline matters: savings were measured on top of a provider prefix cache that was already aggressive, so the −33% session-cost figure understates how much new (billed) input ctxctl removed.

Tool-call reliability

Cost savings are only half the story. Errored MCP calls were once invisible (the original bridge dropped server-side error flags), so we instrumented the driver to count and classify them, then re-ran the same three source-comprehension scenarios as ctxctl evolved:

Run	Binary	Calls	Errored calls	Error rate
Aug-23 baseline	0.3.1	111	9	8.11% (undercount)
Instrumented re-run	0.3.2	134	25	18.66%
Self-healing re-run	0.3.3	130	21	16.15%

The dominant failure class is symbol-not-found — an agent asking a symbol lookup for a name that does not exist in the file. That class fell from ×20 (v0.3.2) to ×9 (v0.3.3) after self-healing symbol errors: misses now return a ranked list of similar symbols plus an embedded mini-outline of the file, so the agent can re-aim inside one round trip. In the v0.3.3 run every miss carried the mini-outline, and both attempts to follow a suggestion recovered with a successful call — against zero of two under v0.3.2, where suggestions could point at the very name just tried.

Measurement honesty: the Aug-23 figures came from a bridge that hid errors — six symbol-not-found misses in one scenario were returned as plain text and invisible to any metric, so 8.11% understates the true baseline. And each arm is a single run with unseeded agent trajectories, so per-scenario swings (e.g. graphology: 21 calls at v0.3.2 vs 10 at v0.3.3) are variance, not regression or improvement. What is deterministic: the error payloads themselves and whether a suggested retry resolves.

Per-scenario tables, methodology, and threats to validity: the full reports for the v0.3.2 instrumentation run and the v0.3.3 self-healing run.

Findings

• Native tools fix adoption. Exposing ctxctl as native MCP tools removed the friction of skill-driven CLI usage and delivered the largest session-cost reduction.
• Exec compression is the least conditional win (~80%+ everywhere). The build-log task landed at −84% cost.
• File exploration scales with file size. On the 96 KB fixture, outline/symbol slices cut uncached input by up to −86%.
• Savings scale inversely with provider prefix-caching quality. Models whose providers cache aggressively still cut uncached input; a weak-caching model (solar-pro4) saved 47% session cost.
• No shell inside exec. exec splits commands shell-word style without an interactive shell, so pipes and redirection need explicit sh -c wrapping — v0.3.1 turns unquoted shell metacharacters into a fail-fast error with that hint.

Caveats

Read the percentages with care:

• Arms are single runs. Treat the numbers as directional evidence, not precise figures.
• Savings depend on the provider’s prefix-caching quality; arms with strong caching shrink relative cost gains even though uncached input still drops.
• Token counts use the cl100k_base BPE tokenizer as a cross-provider approximation.
• The harness scripts are not yet published; we plan to publish them for reproducibility.