CodeVetter · Repo Unpack · Testing signals
garrytan/gstack
Testing is heavyweight and multi-tier. The free suite (~750 files in test/ plus browse/test, make-pdf/test, design/test — ~8,700 tests) runs under bun test orchestrated by scripts/test-free-shards.ts with strict-output classification; paid LLM-judge and E2E evals spawn real claude/codex/gemini processes through a hermetic env and are diff-selected + tier-classified (gate vs periodic). CI enforces a secretless required free lane, a sliced paid lane on PRs, a curated Windows subset, weekly periodic evals, plus dedicated CSO platform gates. Numerous tripwire tests pin security contracts, generated-file freshness, token budgets, and wiring invariants. Aside-dependent tests self-skip off-macOS.
- Test framework is bun:test with a global preload test-setup.ts that snapshots and restores PATH/Path after every test to kill env-pollution flakes between files in a shared Bun process.
- Free suite runs via scripts/test-free-shards.ts — N concurrent shard processes serial-within, packed by recorded durations (scripts/free-test-durations.json), strict classifier (missing terminal summary = FAIL), wall-clock SIGKILL timeout, exit codes 0/1/124; `bun test` alone is explicitly forbidden because it would load paid eval files.
- Windows runs a curated subset: test-free-shards.ts --windows-only filters out files whose content matches POSIX-only patterns (/bin/bash, sh -c, raw /tmp, chmod, xargs); driven by .github/workflows/windows-free-tests.yml.
- Paid tests are two kinds: LLM-judge evals (test/skill-llm-eval*.test.ts via test/helpers/llm-judge.ts, needs ANTHROPIC_API_KEY) and E2E (test/skill-e2e-*.test.ts spawning real `claude -p` / PTY / Agent SDK / codex / gemini sessions via test/helpers/session-runner.ts, claude-pty-runner.ts, agent-sdk-runner.ts, codex-session-runner.ts, gemini-session-runner.ts).
- E2E children run hermetic: test/helpers/hermetic-env.ts builds an allowlist-scrubbed env (drops CONDUCTOR_*, CLAUDE_*, GSTACK_*, MCP_*, GBRAIN_*, GH_TOKEN; keeps only named auth vars), seeds a fresh CLAUDE_CONFIG_DIR and temp GSTACK_HOME, adds --strict-mcp-config; EVALS_HERMETIC=0 is the debug escape hatch evaluated at call time.
- Diff-based selection: E2E_TOUCHFILES/E2E_TIERS in test/helpers/touchfiles-data.ts map tests to the files they depend on; test-selection.ts selects on `git diff` vs base; EVALS_ALL=1 or :all script variants force everything; test/e2e-tier-alignment.test.ts enforces tier declarations agree with each file's EVALS_TIER self-gate.
- CI gates: free-tests.yml is a REQUIRED secretless check (contents:read, persist-credentials:false, fork PRs get real signal; test/free-tests-workflow-wiring.test.ts fails if a secret sneaks in), aggregating free-suite plus cso-macos/windows/docker jobs into the `free-tests` context.
- Paid lane is a sliced pipeline (planner → 6 executors → fail-closed reconcile) run inside a purpose-built image (.github/docker/Dockerfile.ci, keyed on Dockerfile+bun.lock+patches only); weekly evals-periodic.yml runs EVALS_ALL minus enumerated exclusions in test/helpers/periodic-exclude-data.ts.
- Contract-pinning tests are pervasive: test/aside-driver.test.ts pins the {{ASIDE_SETUP}} sentences, test/design-checklist-sync.test.ts pins generated checklist/dom-dump drift, test/hermetic-wiring.test.ts pins the env scrub, test/egress-receipt-wiring.test.ts scans for unreceipted off-machine sinks, test/setup-windows-fallback.test.ts pins the no-raw-ln invariant in setup.
- Token/context budgets are enforced by tests: test/catalog-budget.test.ts caps aggregate skill frontmatter name+description at 1,150 token-equivalents via test/helpers/skill-census.ts; test/context-budget-ratchet.test.ts pins absolute ceilings against test/fixtures/context-budget.json; bin/gstack-context-bill reports the bill-of-materials.
- Flake management is systematic: GSTACK_FREE_RETRY_FLAKY enables an attribution-gated retry pass whose results append to a flake ledger uploaded as a CI artifact even on green; scripts/eval-flake-rank.ts ranks tests by retried-pass signal; eval-store.ts keeps every retry attempt as its own record.
- Aside-dependent tests (test/skill-e2e-aside.test.ts, Aside qa/design cases, live render in test/aside-render.test.ts) self-skip unless run on a Mac with the Aside app open via asideAvailable(); make-pdf and diagram gates run on whichever engine resolves via browserAvailable().
- Eval observability tooling ships in the repo: scripts/eval-list.ts, eval-compare.ts, eval-summary.ts, eval-watch.ts, eval-select.ts, plus bin/gstack-detach (a Python wrapper that daemonizes long eval runs with setsid + caffeinate + machine-wide lock + run-scoped logs + EXIT sentinel).
- Coverage gap: anything requiring the real Aside app (cookie/session realism) only exercises on developer Macs — CI runners have no Aside — so the primary-engine path has weaker continuous coverage than the fallback engine it wraps. (inferred)