torusv0.24.0docs

Efficiency: torus vs stock pi overhead

torus layers a harness (child extension set, tool registrations, system-prompt additions) on top of the stock pi engine. This page documents how that harness-layer context overhead is measured. The comparison the bench makes: stock pi (bare engine, no extensions) vs torus (engine + the same child-extension set delegations get), on identical prompts; the delta is torus’s overhead.

Summary

Headline results first: totals per config across the full task set, the overhead delta (torus − stock), and how that delta moved since the prior release’s run. The table is pasted from the torus 0.14.0 bench output; the HTML report carries the same table, regenerated on every run.

modelfieldstocktorusΔ overheadΔ change
zai/glm-5.3-flashtokens in8,19511,899+3,704+2,797 vs torus 0.6.0
zai/glm-5.3-flashcache read64,768180,480+115,712-5,696 vs torus 0.6.0
zai/glm-5.3-flashcost$0.0039$0.0084+$0.0045+$0.0007 vs torus 0.6.0
zai/glm-5.3-flashms72,276 ms96,867 ms+24,591-31,036 vs torus 0.6.0
zai/glm-5.3tokens in25,05314,730-10,323-664 vs torus 0.6.0
zai/glm-5.3cache read56,768146,048+89,280-1,408 vs torus 0.6.0
zai/glm-5.3cost$0.0569$0.0646+$0.0077-$0.0028 vs torus 0.6.0
zai/glm-5.3ms61,132 ms89,135 ms+28,003-10,581 vs torus 0.6.0

What the 0.14.0 numbers say: the overhead is structural, not behavioral. Cache read dominates because the extension declaration surface is cached input, billed at a fraction of fresh-input price. A full sweep costs under half a cent more on flash and $0.0077 more on glm-5.3. Both models came in below 0.6.0 on cache read, cost, and wall-clock; the flash tokens-in delta grew by 2,797 tokens, and the per-task verdicts below place most of that movement within run-to-run noise.

Method

  • Task set. A fixed, read-only, repo-local prompt set in three difficulty tiers (trivial lookup / aggregate sweep / multi-file reasoning), defined in scripts/bench/tasks.mjs. Deterministic inputs, no file writes, no network beyond the model API, so any token delta comes from the harness layer, not the task.
  • Configs. Per task, the pinned engine (resolveEngineBin) is spawned with --mode json: stock gets the bare argv, torus gets the same argv plus the canonical childExtensionArgs list and TORUS_ENGINE_CHILD=1; byte-for-byte the spawn a delegation receives.
  • Warmup discarded. One warmup run per task × config executes first and is excluded from statistics, so measured runs start from a steady-state prompt cache and warm extension load path.
  • Interleaved rotation. Measured passes alternate config order (stock, torus / torus, stock / …), so provider-side prompt-cache drift and slow-warming trends cancel across configs instead of confounding the delta.
  • Medians with spread. Report cells are medians; ± is half the interquartile range over measured runs. A delta row is labeled signal only when the median delta exceeds the larger config’s IQR (run-to-run noise); anything smaller is labeled within noise rather than passed off as a finding. With fewer than two measured runs there is no dispersion basis and verdicts render as —.
  • Aggregation. Each run’s JSONL event stream is folded by the shared reduceEngineEvent pipeline (the same reducer roster/team/fleet consume): turns, tokens in/out, cacheRead/cacheWrite, and the engine-computed dollar cost come from message_end usage. Malformed event lines are skipped and counted, never fatal.
  • Cache accounting. zai/GLM speaks the OpenAI-compatible schema, which reports cache reads (cached_tokens) but has no cache-write token accounting, so the cache-write column reads 0 on this provider by construction (writes happen implicitly, unbilled and unreported; they surface later as cache reads). On providers with explicit cache-write usage (e.g. Anthropic-style cache_creation tokens) the column fills.
  • Metadata pinned. Every report header carries the date, torus version, git commit, engine pin, and model list, so published numbers are reproducible claims.
  • Run history. The JSON sidecar accumulates one entry per run, so re-benchmarking at each release appends rather than overwrites; older single-run sidecar files migrate to the envelope on first append. Both reports open with a summary table (newest totals plus movement since the prior release), and the HTML report adds a run-history table of per-release overhead deltas.

Running it

The bench spawns the real engine and calls the model API; run it with valid credentials, from the repo root:

npm run bench:overhead -- --model zai/glm-5.3-flash,zai/glm-5.3
OptionMeaning
--model <ids>comma-separated model ids, one full sweep each (default zai/glm-5.3-flash)
--runs Nmeasured runs per task × config (default 3)
--warmup Nwarmup runs per task × config, discarded from stats (default 1)
--tasks <ids>comma-separated task subset (default: all tasks)
--stock-only / --torus-onlysingle-config sweeps (no delta rows)
--timeout <sec>per-run wall-clock cap (default 600)
--json <path>append the run’s raw records (warmups included) to the sidecar history; delete the file to start fresh
--html <path>write the self-contained HTML report (charts + method)

Exit status is 0 when every run (warmups included) completed, 1 otherwise; failures are reported and the bench keeps going.

Artifacts in this repo: docs/efficiency.html (generated report with distribution charts and a run-history table) and docs/bench-records.json (raw per-run records; the sidecar carries one entry per run across releases, and single-run files migrate on first append).

Numbers

Each release re-runs the bench and appends the run to the sidecar history; the newest numbers lead and earlier releases stay below, unchanged. The HTML report’s run-history table carries the same per-release deltas.

torus 0.14.0 · 2026-10-18 · engine pin 1.3.0 · commit 8f4da67

model(s): zai/glm-5.3-flash, zai/glm-5.3 · engine: node_modules/.bin/pi (pin 1.3.0) · torus 0.14.0 · commit 8f4da67 generated 2026-10-18T02:37:44.313Z · 1 warmup run(s) per config discarded · config order rotated per pass (interleaved) 3 measured runs per task per config.

Cells are medians; ± is half the interquartile range across measured runs. Verdicts: signal means the delta exceeds run-to-run noise (max IQR of the two configs).

zai/glm-5.3-flash · 0.14.0

taskconfigturnstokens intokens outcache readcache writecostms
readme-namestock2685 ±1,08892 ±58,704 ±1,0880$0.0004 ±$0.000111,426 ms ±1,298 ms
readme-nametorus21,156 ±2,88985 ±324,320 ±2,8480$0.0009 ±$0.000312,841 ms ±191 ms
agents-doc-headingsstock22,406 ±1,104400 ±58,640 ±1,1040$0.0008 ±$0.000112,280 ms ±812 ms
agents-doc-headingstorus23,038 ±3,043467 ±824,320 ±3,0400$0.0014 ±$0.000417,303 ms ±133 ms
register-tool-countstock281 ±2077 ±288,7040$0.0003 ±$0.00008,035 ms ±529 ms
register-tool-counttorus2143 ±992 ±2024,3200$0.0008 ±$0.000012,539 ms ±708 ms
skills-inventorystock2134 ±1114 ±68,7040$0.0003 ±$0.000011,177 ms ±3,618 ms
skills-inventorytorus2195 ±2117 ±824,3200$0.0008 ±$0.000013,136 ms ±892 ms
extension-registrationstock3 ±13,857 ±1,409683 ±2316,448 ±3,3440$0.0014 ±$0.000318,151 ms ±2,182 ms
extension-registrationtorus3 ±15,847 ±2,467660 ±11545,824 ±8,9760$0.0029 ±$0.000424,919 ms ±3,558 ms
engine-pin-reportstock3 ±01,032 ±72141 ±513,568 ±1,2000$0.0006 ±$0.000011,207 ms ±5,445 ms
engine-pin-reporttorus3 ±01,520 ±289136 ±2537,376 ±3,4080$0.0015 ±$0.000116,129 ms ±1,538 ms
totalsstock14 ±18,195 ±3,6951,507 ±7164,768 ±6,7360$0.0039 ±$0.000672,276 ms ±13,882 ms
totalstorus14 ±111,899 ±8,6991,557 ±179180,480 ±18,2720$0.0084 ±$0.001396,867 ms ±7,019 ms

Δ torus − stock (medians) vs run-to-run noise:

taskfieldΔ mednoise (IQR)verdict
readme-nameturns00flat
readme-nametokens in+4715,777within noise
readme-nametokens out-710within noise
readme-namecache read+15,6165,696signal
readme-namecache write00flat
readme-namecost+$0.0005$0.0007within noise
readme-namems+1,4152,595 mswithin noise
agents-doc-headingsturns00flat
agents-doc-headingstokens in+6326,087within noise
agents-doc-headingstokens out+6716signal
agents-doc-headingscache read+15,6806,080signal
agents-doc-headingscache write00flat
agents-doc-headingscost+$0.0006$0.0007within noise
agents-doc-headingsms+5,0231,623 mssignal
register-tool-countturns00flat
register-tool-counttokens in+6241signal
register-tool-counttokens out+1556within noise
register-tool-countcache read+15,6160signal
register-tool-countcache write00flat
register-tool-countcost+$0.0005$0.0000signal
register-tool-countms+4,5041,416 mssignal
skills-inventoryturns00flat
skills-inventorytokens in+614signal
skills-inventorytokens out+316within noise
skills-inventorycache read+15,6160signal
skills-inventorycache write00flat
skills-inventorycost+$0.0005$0.0000signal
skills-inventoryms+1,9597,236 mswithin noise
extension-registrationturns01within noise
extension-registrationtokens in+1,9904,934within noise
extension-registrationtokens out-23231within noise
extension-registrationcache read+29,37617,952signal
extension-registrationcache write00flat
extension-registrationcost+$0.0015$0.0009signal
extension-registrationms+6,7687,115 mswithin noise
engine-pin-reportturns01within noise
engine-pin-reporttokens in+488579within noise
engine-pin-reporttokens out-550within noise
engine-pin-reportcache read+23,8086,816signal
engine-pin-reportcache write00flat
engine-pin-reportcost+$0.0009$0.0002signal
engine-pin-reportms+4,92210,889 mswithin noise

zai/glm-5.3 · 0.14.0

taskconfigturnstokens intokens outcache readcache writecostms
readme-namestock2666 ±3568 ±18,6400$0.0035 ±$0.00015,616 ms ±859 ms
readme-nametorus2960 ±3474 ±724,3200$0.0079 ±$0.000012,400 ms ±589 ms
agents-doc-headingsstock3 ±02,586 ±675459 ±3913,312 ±1,0880$0.0091 ±$0.001111,020 ms ±243 ms
agents-doc-headingstorus23,008 ±1268 ±6424,3200$0.0117 ±$0.000314,598 ms ±575 ms
register-tool-countstock2104 ±9117 ±108,7040$0.0029 ±$0.00016,014 ms ±696 ms
register-tool-counttorus2168 ±9104 ±224,3200$0.0070 ±$0.000012,227 ms ±563 ms
skills-inventorystock2146 ±4155 ±98,7040$0.0031 ±$0.00008,704 ms ±1,383 ms
skills-inventorytorus2192 ±33138 ±224,320 ±320$0.0072 ±$0.000017,675 ms ±1,887 ms
extension-registrationstock220,705 ±5,169953 ±678,704 ±5,1680$0.0344 ±$0.005916,342 ms ±2,280 ms
extension-registrationtorus28,958 ±17716 ±1324,384 ±160$0.0220 ±$0.000019,717 ms ±1,603 ms
engine-pin-reportstock2846 ±35105 ±28,704 ±160$0.0039 ±$0.000013,436 ms ±2,515 ms
engine-pin-reporttorus21,444 ±4104 ±524,3840$0.0088 ±$0.000012,518 ms ±1,838 ms
totalsstock13 ±025,053 ±5,9271,857 ±12956,768 ±6,2720$0.0569 ±$0.007261,132 ms ±7,975 ms
totalstorus1214,730 ±981,404 ±92146,048 ±480$0.0646 ±$0.000489,135 ms ±7,054 ms

Δ torus − stock (medians) vs run-to-run noise:

taskfieldΔ mednoise (IQR)verdict
readme-nameturns00flat
readme-nametokens in+29470signal
readme-nametokens out+614within noise
readme-namecache read+15,6800signal
readme-namecache write00flat
readme-namecost+$0.0044$0.0001signal
readme-namems+6,7841,719 mssignal
agents-doc-headingsturns-11signal
agents-doc-headingstokens in+4221,351within noise
agents-doc-headingstokens out-191127signal
agents-doc-headingscache read+11,0082,176signal
agents-doc-headingscache write00flat
agents-doc-headingscost+$0.0026$0.0021signal
agents-doc-headingsms+3,5781,149 mssignal
register-tool-countturns00flat
register-tool-counttokens in+6419signal
register-tool-counttokens out-1321within noise
register-tool-countcache read+15,6160signal
register-tool-countcache write00flat
register-tool-countcost+$0.0041$0.0001signal
register-tool-countms+6,2131,391 mssignal
skills-inventoryturns00flat
skills-inventorytokens in+4666within noise
skills-inventorytokens out-1718within noise
skills-inventorycache read+15,61664signal
skills-inventorycache write00flat
skills-inventorycost+$0.0040$0.0001signal
skills-inventoryms+8,9713,774 mssignal
extension-registrationturns00flat
extension-registrationtokens in-11,74710,339signal
extension-registrationtokens out-237135signal
extension-registrationcache read+15,68010,336signal
extension-registrationcache write00flat
extension-registrationcost-$0.0123$0.0118signal
extension-registrationms+3,3754,560 mswithin noise
engine-pin-reportturns00flat
engine-pin-reporttokens in+59869signal
engine-pin-reporttokens out-110within noise
engine-pin-reportcache read+15,68032signal
engine-pin-reportcache write00flat
engine-pin-reportcost+$0.0049$0.0001signal
engine-pin-reportms-9185,029 mswithin noise

torus 0.6.0 · 2026-10-07 · engine pin 1.0.4 · commit 1c7256e (historical)

Measured 2026-10-07, torus 0.6.0 + engine pin 1.0.4, commit 1c7256e · 3 measured runs per task per config (1 warmup discarded, config order rotated per pass) · npm run bench:overhead -- --model zai/glm-5.3-flash,zai/glm-5.3. Cells are medians; ± is half the interquartile range across measured runs. Verdicts: signal means the delta exceeds run-to-run noise (max IQR of the two configs).

zai/glm-5.3-flash · 0.6.0

taskconfigturnstokens intokens outcache readcache writecostms
readme-namestock23,644 ±1,72281 ±95,888 ±1,6960$0.0008 ±$0.00025,402 ms ±265 ms
readme-nametorus23,949 ±3,509110 ±821,824 ±3,4880$0.0013 ±$0.000412,805 ms ±579 ms
agents-doc-headingsstock22,411 ±1,440370 ±378,448 ±1,4400$0.0008 ±$0.00028,257 ms ±242 ms
agents-doc-headingstorus22,908 ±2,880405 ±3424,384 ±2,8800$0.0014 ±$0.000417,335 ms ±835 ms
register-tool-countstock2121 ±865 ±68,5120$0.0003 ±$0.00004,558 ms ±840 ms
register-tool-counttorus219 ±1793 ±924,512 ±160$0.0008 ±$0.000012,227 ms ±232 ms
skills-inventorystock2158 ±3299 ±68,512 ±320$0.0003 ±$0.00005,375 ms ±891 ms
skills-inventorytorus2126 ±1799 ±224,448 ±160$0.0008 ±$0.000011,939 ms ±98 ms
extension-registrationstock3 ±07,698 ±827709 ±7620,992 ±1,8400$0.0023 ±$0.000216,718 ms ±1,990 ms
extension-registrationtorus4 ±07,073 ±513772 ±11054,592 ±4,1920$0.0032 ±$0.000233,399 ms ±3,786 ms
engine-pin-reportstock3539 ±11684 ±113,504 ±1120$0.0005 ±$0.00005,417 ms ±319 ms
engine-pin-reporttorus31,403 ±232118 ±1537,504 ±2560$0.0014 ±$0.000013,649 ms ±879 ms
totalsstock14 ±014,571 ±4,1441,408 ±13465,856 ±5,1200$0.0050 ±$0.000645,727 ms ±4,547 ms
totalstorus15 ±015,478 ±7,1681,597 ±177187,264 ±10,8480$0.0088 ±$0.0011101,354 ms ±6,409 ms

Δ torus − stock (medians) vs run-to-run noise:

taskfieldΔ mednoise (IQR)verdict
readme-nameturns00flat
readme-nametokens in+3057,018within noise
readme-nametokens out+2918signal
readme-namecache read+15,9366,976signal
readme-namecache write00flat
readme-namecost+$0.0005$0.0008within noise
readme-namems+7,4031,158 mssignal
agents-doc-headingsturns00flat
agents-doc-headingstokens in+4975,760within noise
agents-doc-headingstokens out+3574within noise
agents-doc-headingscache read+15,9365,760signal
agents-doc-headingscache write00flat
agents-doc-headingscost+$0.0006$0.0007within noise
agents-doc-headingsms+9,0781,669 mssignal
register-tool-countturns00flat
register-tool-counttokens in-10234signal
register-tool-counttokens out+2817signal
register-tool-countcache read+16,00032signal
register-tool-countcache write00flat
register-tool-countcost+$0.0005$0.0000signal
register-tool-countms+7,6691,681 mssignal
skills-inventoryturns00flat
skills-inventorytokens in-3264within noise
skills-inventorytokens out012within noise
skills-inventorycache read+15,93664signal
skills-inventorycache write00flat
skills-inventorycost+$0.0005$0.0000signal
skills-inventoryms+6,5641,781 mssignal
extension-registrationturns+11signal
extension-registrationtokens in-6251,654within noise
extension-registrationtokens out+63220within noise
extension-registrationcache read+33,6008,384signal
extension-registrationcache write00flat
extension-registrationcost+$0.0009$0.0005signal
extension-registrationms+16,6817,572 mssignal
engine-pin-reportturns00flat
engine-pin-reporttokens in+864463signal
engine-pin-reporttokens out+3429signal
engine-pin-reportcache read+24,000512signal
engine-pin-reportcache write00flat
engine-pin-reportcost+$0.0009$0.0001signal
engine-pin-reportms+8,2321,758 mssignal

zai/glm-5.3 · 0.6.0

taskconfigturnstokens intokens outcache readcache writecostms
readme-namestock2742 ±8263 ±58,5120$0.0035 ±$0.00014,110 ms ±199 ms
readme-nametorus2694 ±27176 ±524,448 ±2720$0.0076 ±$0.000310,978 ms ±82 ms
agents-doc-headingsstock2 ±02,374 ±77365 ±628,512 ±1,6320$0.0072 ±$0.00087,970 ms ±1,075 ms
agents-doc-headingstorus2 ±02,843 ±67386 ±7324,448 ±3,0880$0.0120 ±$0.001214,236 ms ±3,195 ms
register-tool-countstock2177 ±23150 ±98,512 ±160$0.0031 ±$0.00015,453 ms ±569 ms
register-tool-counttorus2113 ±21128 ±1124,4480$0.0071 ±$0.000110,587 ms ±785 ms
skills-inventorystock2168 ±17136 ±68,512 ±160$0.0031 ±$0.00005,051 ms ±484 ms
skills-inventorytorus2135 ±3123 ±1024,4480$0.0071 ±$0.000012,175 ms ±358 ms
extension-registrationstock219,273 ±1679 ±678,5760$0.0322 ±$0.000311,288 ms ±936 ms
extension-registrationtorus2 ±18,756 ±180644 ±3524,448 ±10,5440$0.0218 ±$0.002919,495 ms ±2,202 ms
engine-pin-reportstock3 ±0779 ±129140 ±1613,376 ±1,3280$0.0047 ±$0.00046,717 ms ±1,110 ms
engine-pin-reporttorus21,313 ±320123 ±624,448 ±3200$0.0086 ±$0.000411,702 ms ±1,479 ms
totalsstock13 ±123,513 ±3291,533 ±16456,000 ±2,9920$0.0538 ±$0.001740,589 ms ±4,373 ms
totalstorus12 ±113,854 ±8611,480 ±140146,688 ±14,2240$0.0643 ±$0.004979,173 ms ±8,100 ms

Δ torus − stock (medians) vs run-to-run noise:

taskfieldΔ mednoise (IQR)verdict
readme-nameturns00flat
readme-nametokens in-48542within noise
readme-nametokens out+1310signal
readme-namecache read+15,936544signal
readme-namecache write00flat
readme-namecost+$0.0041$0.0006signal
readme-namems+6,868398 mssignal
agents-doc-headingsturns01within noise
agents-doc-headingstokens in+469154signal
agents-doc-headingstokens out+21147within noise
agents-doc-headingscache read+15,9366,176signal
agents-doc-headingscache write00flat
agents-doc-headingscost+$0.0049$0.0024signal
agents-doc-headingsms+6,2666,390 mswithin noise
register-tool-countturns00flat
register-tool-counttokens in-6446signal
register-tool-counttokens out-2221signal
register-tool-countcache read+15,93632signal
register-tool-countcache write00flat
register-tool-countcost+$0.0040$0.0002signal
register-tool-countms+5,1341,570 mssignal
skills-inventoryturns00flat
skills-inventorytokens in-3335within noise
skills-inventorytokens out-1320within noise
skills-inventorycache read+15,93632signal
skills-inventorycache write00flat
skills-inventorycost+$0.0040$0.0001signal
skills-inventoryms+7,124969 mssignal
extension-registrationturns01within noise
extension-registrationtokens in-10,517360signal
extension-registrationtokens out-35135within noise
extension-registrationcache read+15,87221,088within noise
extension-registrationcache write00flat
extension-registrationcost-$0.0104$0.0059signal
extension-registrationms+8,2074,403 mssignal
engine-pin-reportturns-11signal
engine-pin-reporttokens in+534640within noise
engine-pin-reporttokens out-1732within noise
engine-pin-reportcache read+11,0722,656signal
engine-pin-reportcache write00flat
engine-pin-reportcost+$0.0039$0.0008signal
engine-pin-reportms+4,9852,957 mssignal

Reading them

  • Cache read is the dominant, consistent overhead signal: the extension declaration surface, cached by the provider at a fraction of input price.
  • Non-cached tokens-in deltas are mostly within run-to-run noise, and can be negative: on the multi-file reasoning task the stock engine flails (more turns, more fresh input) while the torus context reaches the answer directly.
  • Cost deltas are small but above noise on glm-5.3; on flash pricing they sit closer to the noise floor.
  • Wall-clock overhead per run is dominated by extension loading at spawn; amortized in real sessions that run many turns.
  • Ambient roles (dream/reflect, session-title) resolve via the fast tier, so a fast-tier override in the config’s chains section pointing at a local model moves all ambient work to it.
  • Consumed ACP agents bypass the chain machinery entirely (they run their own models), so their usage/cost figures are best-effort zeros; see the acp section in docs/extensions.md.
  • Compare models only within their own section; pricing and token behavior differ.
  • For release gating, cache read and cost deltas are the dependable signals. Each run compares torus with stock on the same engine pin, so the overhead delta is engine-normalized. Tokens in, turns, and wall-clock move with task behavior and network jitter; treat them as context, not gate conditions.