# Testing signals — garrytan/gstack

> Canonical page: https://codevetter.com/unpack/gstack/testing-signals

# Testing signals — garrytan/gstack

Scanned commit 35dd014c580037d1c009776d8b0584587c1bd42c (2026-09-21). Generated by codevetter codevetter 1.12.0 on 2026-09-22.

Testing is heavyweight and multi-tier. The free suite (~750 files in test/ plus browse/test, make-pdf/test, design/test — ~8,700 tests) runs under bun test orchestrated by scripts/test-free-shards.ts with strict-output classification; paid LLM-judge and E2E evals spawn real claude/codex/gemini processes through a hermetic env and are diff-selected + tier-classified (gate vs periodic). CI enforces a secretless required free lane, a sliced paid lane on PRs, a curated Windows subset, weekly periodic evals, plus dedicated CSO platform gates. Numerous tripwire tests pin security contracts, generated-file freshness, token budgets, and wiring invariants. Aside-dependent tests self-skip off-macOS.
- Test framework is bun:test with a global preload test-setup.ts that snapshots and restores PATH/Path after every test to kill env-pollution flakes between files in a shared Bun process. — [test-setup.ts#L1-53](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/test-setup.ts#L1-L53)
- Free suite runs via scripts/test-free-shards.ts — N concurrent shard processes serial-within, packed by recorded durations (scripts/free-test-durations.json), strict classifier (missing terminal summary = FAIL), wall-clock SIGKILL timeout, exit codes 0/1/124; `bun test` alone is explicitly forbidden because it would load paid eval files. — [scripts/test-free-shards.ts#L1-75](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/scripts/test-free-shards.ts#L1-L75) [CLAUDE.md](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/CLAUDE.md)
- Windows runs a curated subset: test-free-shards.ts --windows-only filters out files whose content matches POSIX-only patterns (/bin/bash, sh -c, raw /tmp, chmod, xargs); driven by .github/workflows/windows-free-tests.yml. — [scripts/test-free-shards.ts#L10-13](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/scripts/test-free-shards.ts#L10-L13) [.github/workflows/windows-free-tests.yml](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/.github/workflows/windows-free-tests.yml)
- Paid tests are two kinds: LLM-judge evals (test/skill-llm-eval*.test.ts via test/helpers/llm-judge.ts, needs ANTHROPIC_API_KEY) and E2E (test/skill-e2e-*.test.ts spawning real `claude -p` / PTY / Agent SDK / codex / gemini sessions via test/helpers/session-runner.ts, claude-pty-runner.ts, agent-sdk-runner.ts, codex-session-runner.ts, gemini-session-runner.ts). — [package.json#L29-40](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/package.json#L29-L40) [test/helpers/session-runner.ts](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/test/helpers/session-runner.ts) [test/helpers/claude-pty-runner.ts](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/test/helpers/claude-pty-runner.ts) [test/helpers/llm-judge.ts](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/test/helpers/llm-judge.ts)
- E2E children run hermetic: test/helpers/hermetic-env.ts builds an allowlist-scrubbed env (drops CONDUCTOR_*, CLAUDE_*, GSTACK_*, MCP_*, GBRAIN_*, GH_TOKEN; keeps only named auth vars), seeds a fresh CLAUDE_CONFIG_DIR and temp GSTACK_HOME, adds --strict-mcp-config; EVALS_HERMETIC=0 is the debug escape hatch evaluated at call time. — [test/helpers/hermetic-env.ts#L1-100](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/test/helpers/hermetic-env.ts#L1-L100) [docs/TESTING_INTERNALS.md#L19-37](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/docs/TESTING_INTERNALS.md#L19-L37)
- Diff-based selection: E2E_TOUCHFILES/E2E_TIERS in test/helpers/touchfiles-data.ts map tests to the files they depend on; test-selection.ts selects on `git diff` vs base; EVALS_ALL=1 or :all script variants force everything; test/e2e-tier-alignment.test.ts enforces tier declarations agree with each file's EVALS_TIER self-gate. — [test/helpers/touchfiles.ts#L1-43](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/test/helpers/touchfiles.ts#L1-L43) [test/e2e-tier-alignment.test.ts](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/test/e2e-tier-alignment.test.ts) [CLAUDE.md](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/CLAUDE.md)
- CI gates: free-tests.yml is a REQUIRED secretless check (contents:read, persist-credentials:false, fork PRs get real signal; test/free-tests-workflow-wiring.test.ts fails if a secret sneaks in), aggregating free-suite plus cso-macos/windows/docker jobs into the `free-tests` context. — [.github/workflows/free-tests.yml#L10-45](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/.github/workflows/free-tests.yml#L10-L45) [.github/workflows/free-tests.yml#L235-255](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/.github/workflows/free-tests.yml#L235-L255)
- Paid lane is a sliced pipeline (planner → 6 executors → fail-closed reconcile) run inside a purpose-built image (.github/docker/Dockerfile.ci, keyed on Dockerfile+bun.lock+patches only); weekly evals-periodic.yml runs EVALS_ALL minus enumerated exclusions in test/helpers/periodic-exclude-data.ts. — [.github/workflows/evals.yml#L19-98](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/.github/workflows/evals.yml#L19-L98) [.github/workflows/evals-periodic.yml](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/.github/workflows/evals-periodic.yml) [test/helpers/periodic-exclude-data.ts](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/test/helpers/periodic-exclude-data.ts)
- Contract-pinning tests are pervasive: test/aside-driver.test.ts pins the {{ASIDE_SETUP}} sentences, test/design-checklist-sync.test.ts pins generated checklist/dom-dump drift, test/hermetic-wiring.test.ts pins the env scrub, test/egress-receipt-wiring.test.ts scans for unreceipted off-machine sinks, test/setup-windows-fallback.test.ts pins the no-raw-ln invariant in setup. — [test/aside-driver.test.ts](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/test/aside-driver.test.ts) [test/design-checklist-sync.test.ts](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/test/design-checklist-sync.test.ts) [test/hermetic-wiring.test.ts](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/test/hermetic-wiring.test.ts) [test/egress-receipt-wiring.test.ts](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/test/egress-receipt-wiring.test.ts) [scripts/resolvers/aside.ts#L39-42](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/scripts/resolvers/aside.ts#L39-L42)
- Token/context budgets are enforced by tests: test/catalog-budget.test.ts caps aggregate skill frontmatter name+description at 1,150 token-equivalents via test/helpers/skill-census.ts; test/context-budget-ratchet.test.ts pins absolute ceilings against test/fixtures/context-budget.json; bin/gstack-context-bill reports the bill-of-materials. — [test/catalog-budget.test.ts](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/test/catalog-budget.test.ts) [test/context-budget-ratchet.test.ts](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/test/context-budget-ratchet.test.ts) [test/helpers/skill-census.ts](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/test/helpers/skill-census.ts) [CLAUDE.md](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/CLAUDE.md)
- Flake management is systematic: GSTACK_FREE_RETRY_FLAKY enables an attribution-gated retry pass whose results append to a flake ledger uploaded as a CI artifact even on green; scripts/eval-flake-rank.ts ranks tests by retried-pass signal; eval-store.ts keeps every retry attempt as its own record. — [.github/workflows/free-tests.yml#L196-220](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/.github/workflows/free-tests.yml#L196-L220) [scripts/eval-flake-rank.ts](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/scripts/eval-flake-rank.ts) [test/helpers/eval-store.ts](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/test/helpers/eval-store.ts)
- Aside-dependent tests (test/skill-e2e-aside.test.ts, Aside qa/design cases, live render in test/aside-render.test.ts) self-skip unless run on a Mac with the Aside app open via asideAvailable(); make-pdf and diagram gates run on whichever engine resolves via browserAvailable(). — [test/helpers/aside-available.ts](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/test/helpers/aside-available.ts) [docs/TESTING_INTERNALS.md#L67-80](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/docs/TESTING_INTERNALS.md#L67-L80) [CLAUDE.md](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/CLAUDE.md)
- Eval observability tooling ships in the repo: scripts/eval-list.ts, eval-compare.ts, eval-summary.ts, eval-watch.ts, eval-select.ts, plus bin/gstack-detach (a Python wrapper that daemonizes long eval runs with setsid + caffeinate + machine-wide lock + run-scoped logs + EXIT sentinel). — [scripts/eval-select.ts](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/scripts/eval-select.ts) [bin/gstack-detach#L1-30](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/bin/gstack-detach#L1-L30) [package.json#L44-53](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/package.json#L44-L53)
- Coverage gap: anything requiring the real Aside app (cookie/session realism) only exercises on developer Macs — CI runners have no Aside — so the primary-engine path has weaker continuous coverage than the fallback engine it wraps. (inferred) — [docs/TESTING_INTERNALS.md#L67-80](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/docs/TESTING_INTERNALS.md#L67-L80) [.github/workflows/evals.yml](https://github.com/garrytan/gstack/blob/35dd014c580037d1c009776d8b0584587c1bd42c/.github/workflows/evals.yml)

Every claim links to its cited files at the pinned commit. Citations identify source evidence; they do not establish runtime correctness. Independent CodeVetter scan — not reviewed by upstream maintainers.

## Public product links

- [CodeVetter](https://codevetter.com/)
- [Download](https://codevetter.com/download)
- [Documentation](https://codevetter.com/docs/)
- [Source](https://github.com/Codevetter/codevetter)