TerminalBench

Terminal-emulator performance protocol · 日本語

Reporting specification v1 · Release workflow

Make terminal performance a claim-safe release artifact.

Run versioned JSON benchmarks, validate their claims, and check pairwise eligibility before you compare.

Try TerminalBench

Records process and PTY observation boundaries—not pixels on screen or physical keypress-to-photon latency.

First command

bin/tbench run --output results/report-a.json

Registered measurement · profile v1.0.0

Claim-bound excerpt 
{
  "classification": "proxy",
  "scope": "pty-output-pipeline",
  "observed_boundary":
    "fixture-write-start-to-producer-exit",
  "limitations": [
    "Measures producer-side PTY write completion; terminal parsing, final-frame presentation, and frame skipping are not observed."
  ],
  "excluded_claims": [
    "On-screen rendering throughput"
  ]
}

A registered measurement excerpt carries its observation boundary and excluded claim.

A benchmark becomes useful when its claims survive the gate.

TerminalBench records each run as versioned JSON. The validator checks what every metric may claim. The eligibility gate then decides whether two reports can proceed to comparison.

The safe sequence

Run. Validate. Check eligibility. Then compare.

  1. 01 / Run

    Record the observation

    Run TerminalBench inside each terminal emulator. Each report carries its environment, method, observation boundary, limitations, and excluded claims.

    bin/tbench run --output results/report-a.json

  2. 02 / Validate

    Reject unsafe reports

    Validate both JSON files. Structure alone is not enough: semantic checks enforce registered profiles, numeric invariants, and claim boundaries.

    bin/tbench-validate validate results/report-a.json
bin/tbench-validate validate results/report-b.json

  3. 03 / Eligibility

    Decide whether the pair may proceed

    The pairwise gate checks measurement identity, workload identity, and required known controls. It returns one of three decisions. Unknown is never a wildcard.

    bin/tbench-validate compare results/report-a.json results/report-b.json

  4. 04 / Compare

    Present values only after COMPARABLE

    The comparator presents the values as a Markdown table or, with --format html, a self-contained chart report: a baseline-relative overview (--baseline picks the reference), a Results at a glance table of medians and direction-aware competition ranks (exact ties share a rank), per-metric dot-and-interval charts (bootstrap 95% CI with samples; median-to-p95 whisker otherwise), and a win tally. It does not replace validation or the eligibility decision.

    bin/tbench compare results/report-a.json results/report-b.json --format html

Stop before the final command on NOT_COMPARABLE or UNDETERMINED.

Quickstart

Create the first report.

First, clone or open the TerminalBench repository and enter its repository root.

From the repository root, run:

bin/tbench run --output results/report-a.json

The runner is a single self-contained Rust binary; bin/tbench builds it on first use, so a Rust toolchain (cargo) is needed only to build. Validation, eligibility, and value comparison are built into the same binary. A controlling terminal is required for terminal-facing measurements; unsupported contexts warn, skip, or degrade as documented.

Show the complete safe comparison sequence 
set -e

# Run once inside each terminal emulator.
bin/tbench run --output results/report-a.json &&
bin/tbench run --output results/report-b.json &&

# Validate both reports, then require COMPARABLE.
bin/tbench-validate validate results/report-a.json &&
bin/tbench-validate validate results/report-b.json &&
bin/tbench-validate compare results/report-a.json results/report-b.json &&

# Present values only after the gate exits 0.
bin/tbench compare results/report-a.json results/report-b.json

Stop on NOT_COMPARABLE (exit 7) or UNDETERMINED (exit 8).

Pairwise eligibility

Three decisions. No guessed equivalence.

The gate decides permission to compare. It does not compute a winner.

COMPARABLE

Exit 0

Both reports are conformant, and their required known controls and measurement contracts satisfy the v1 pairwise rules.

NOT_COMPARABLE

Exit 7

A known mismatch blocks comparison. Keep the reason with the reports and do not present a winner.

UNDETERMINED

Exit 8

A decisive fact or workload identity is unknown. Collect the missing context; unknown never means “same.”

Claim boundary

The report carries the boundary. The validator enforces it.

What the report may say

For render throughput, the registered proxy observes fixture write start through producer exit. The report carries that scope, its limitations, and the claim it excludes.

{
  "classification": "proxy",
  "scope": "pty-output-pipeline",
  "observed_boundary":
    "fixture-write-start-to-producer-exit",
  "excluded_claims": [
    "On-screen rendering throughput"
  ]
}

What the validator refuses

Relabeling the same proxy as direct screen presentation fails closed with a stable rule ID, a JSON Pointer, expected and actual values, and a remediation.

{
  "rule_id": "TBV-CLAIM-BOUNDARY",
  "pointer": "/results/0/measurement/observed_boundary",
  "expected": "fixture-write-start-to-producer-exit",
  "actual": "final-frame-presented-to-display",
  "remediation": "Copy the immutable profile field exactly;
    choose a different profile for different semantics."
}

The schema checks structure. The semantic validator decides conformance and claim safety.

Stable process outcomes

Every stop has an explicit exit.

Use the exit status to stop automation before an unsupported comparison reaches presentation.

ExitVisible meaning
0Conformant or COMPARABLE
2Usage error
3I/O, UTF-8, strict JSON, or duplicate-key error
4Conformance or claim-safety failure
5Unsupported wire version
6Unsupported profile ID or version
7NOT_COMPARABLE
8UNDETERMINED

What v1 does—and does not—establish

Conformance is specific. Trust stays bounded.

Established by a conformant v1 report

  • The report follows the required wire structure.
  • Its metric profiles carry registered methods, scopes, limitations, and exclusions.
  • The semantic validator accepts its invariants and claim boundaries.

Not established by that report

  • Absolute measurement accuracy
  • On-screen rendering or frame presentation
  • Physical keypress-to-photon latency
  • Reproducibility across every machine or environment
  • Independent neutrality, external adoption, or market leadership
  • Pairwise comparability before the eligibility gate returns COMPARABLE

TerminalBench is a reporting and comparability protocol, not a fastest-terminal leaderboard.

FAQ

Questions to settle before the first run.

What does TerminalBench measure?

It records shell process timings and terminal-emulator proxy or sampled observations for PTY output, synthetic PTY echo, and process resources. Every v1 metric names its method, scope, observed boundary, limitations, and excluded claims.

Does it measure what appears on screen?

No. Producer-side PTY completion does not observe terminal parsing, final-frame presentation, pixels, or frame skipping. The render and scroll profiles exclude on-screen claims.

Does input.latency measure a physical keypress?

No. It measures a synthetic PTY echo round trip. It excludes the keyboard, GUI event routing, rendering, display presentation, and physical keypress-to-photon latency.

Can I compare any two reports?

No. Validate both reports, then run the pairwise eligibility gate. Compare values only after it returns COMPARABLE. A known mismatch returns NOT_COMPARABLE; missing decisive context returns UNDETERMINED.

Is this a fastest-terminal ranking?

No. The protocol records bounded observations and decides pairwise eligibility. It does not operate a leaderboard or certify a universal winner.

What happens when the environment is incomplete?

Reports record required facts as known or unknown. If a decisive fact remains unknown and no known mismatch already blocks the pair, eligibility is UNDETERMINED. CI can provide documented stable controls when platform discovery cannot.

What does the first run require?

A Rust toolchain (cargo), used once to build the single self-contained binary — after that there are no runtime dependencies. Terminal-facing scenarios need a controlling terminal, and input latency needs a usable PTY.

Is this the AI-agent benchmark named Terminal-Bench?

No. This project is a terminal-emulator performance reporting and comparability protocol.