Skip to content
CtxCtl

Benchmark

The numbers on the landing page come from a scripted-agent benchmark. This page documents the protocol, the full result tables, and what the data does — and does not — support.

Protocol

Section titled “Protocol”

  • Driver: the pi coding agent, running each scripted task end-to-end with tokens billed via OpenRouter.

  • Tasks: four scripted scenarios against real repositories:

    Scenario Fixture Size
    fabric Canvas.ts 51 KB
    konva Node.ts 96 KB
    graphology graph.js 77 KB
    execlog deterministic build-log fixture 1,914 lines

  • Arms:

    • base — the agent’s built-in read + bash tools.
    • cliskill — ctxctl driven through the agent skill (CLI calls).
    • mcp — ctxctl exposed as five native MCP tools.

  • Model: deepseek-v4-flash via OpenRouter. Base arm averaged over n=3 runs; cliskill and mcp arms are single-run.

Totals

Section titled “Totals”
Arm Session cost Cost vs base Uncached input vs base
base (built-in read + bash) $0.0047 ± 0.0026 — —
cliskill (ctxctl via skill) $0.0039 −17.2% −35.1%
mcp (native ctxctl tools) $0.0032 −32.7% −31.4%

The base arm averaged 26,284 uncached (fully billed) input tokens per session at an 81.7% provider cache hit rate — i.e. savings measured on top of an already-aggressive prefix cache.

Per-scenario results

Section titled “Per-scenario results”

Session cost per scenario (deepseek-v4-flash; base n=3 reps, cliskill/mcp n=1):

Scenario base cliskill mcp
fabric $0.0042±0.0011 $0.0040 $0.0060
konva $0.0059±0.0047 $0.0045 $0.0032
graphology $0.0038±0.0014 $0.0027 $0.0028
execlog $0.0050±0.0030 $0.0045 $0.0008
Scenario Headline result
fabric Behavioral outlier: defied the cost pattern while producing the best answer of the benchmark
konva Exploring the 96 KB file via outline/symbol slices cut uncached input −46% (cliskill) and −86% (mcp)
graphology Consistent modest win in every ctxctl arm
execlog Log-analysis task via ctxctl exec: −84% task cost

Findings

Section titled “Findings”
  • Native tools fix adoption. Exposing ctxctl as native MCP tools removed the friction of skill-driven CLI usage and delivered the largest session-cost reduction (−33% vs read/bash).
  • Exec compression is the least conditional win (~80%+ everywhere). The build-log task landed at −84% cost.
  • File exploration scales with file size. On the 96 KB konva fixture, outline/symbol slices cut uncached input by up to −86%.
  • fabric was a behavioral outlier — it broke the cost pattern yet produced the benchmark’s best answer, a reminder that cheaper context can also mean sharper context.
  • Latency trade-off: tool-heavy ctxctl arms took roughly 2× wall-clock time. You pay in seconds what you save in dollars and cache pressure.
  • Savings scale inversely with provider prefix-caching quality. Models whose providers cache aggressively still cut uncached input; a weak-caching model (solar-pro4) saved 47% session cost.
  • No-shell lesson. ctxctl exec splits commands shell-word style without an interactive shell, so pipes and redirection need explicit sh -c wrapping — worth encoding in agent docs so models stop assuming a full shell.

Caveats

Section titled “Caveats”

Read the precise percentages with care:

  • The deepseek-v4-flash base arm is an n=3 average with visible variance ($0.0047 ± 0.0026); the cliskill and mcp arms are single runs. Rerun before quoting exact figures.
  • Savings depend on the provider’s prefix-caching quality; arms with strong caching shrink relative cost gains even though uncached input still drops.
  • Token counts use the cl100k_base BPE tokenizer as a cross-provider approximation.
  • The harness scripts are not yet published; we plan to publish them for reproducibility.