CodeVetter · Repo Unpack · Testing signals
DietrichGebert/ponytail
Tests are plain node:test/assert suites — no framework, matching the project's own dogma. tests/*.test.js covers hook behavior end-to-end via spawnSync (every host env permutation), manifest/command/packaging drift guards, and grader self-checks; pi-extension/test and ponytail-mcp/test cover the adapters' pure functions. CI runs the suite plus two consistency scripts on every push/PR. Coverage is strong on the hooks protocol and drift guards; weaker on the Hermes Python plugin logic depth and on actual model behavior (benchmarks are manual, API-spend).
- Test framework is node:test + assert with no dependencies; `npm test` = `node --test tests/*.test.js && npm test --prefix pi-extension && npm test --prefix ponytail-mcp`.
- tests/hooks.test.js is an integration suite that spawnSyncs the real hook scripts under fabricated HOME/env dirs, asserting flag-file placement, per-host stdout JSON shapes, statusline nudge once-only behavior, CLAUDE_CONFIG_DIR override, and Copilot-vs-Codex state isolation.
- Subagent matcher behavior is fully covered: case-insensitive unanchored match, anchored rejection of superset names, missing agent_type fail-open, invalid-regex fail-open, and the no-matcher path's independence from stdin.
- Drift guards: tests/commands.test.js greps pi-extension/index.js for registerCommand calls and fails if any registered command lacks commands/<name>.toml or .opencode/command/<name>.md; tests/package.test.js guards scripts/uninstall.js staying in the npm files list.
- Per-host manifest tests exist for every adapter: gemini-extension.test.js, qoder-plugin.test.js, grok-plugin.test.js, hermes-plugin.test.js, copilot-plugin.test.js, opencode-plugin.test.js, openclaw-skills.test.js, plus hooks-windows.test.js and cursor-hooks.test.js for platform edges.
- Two consistency scripts run in CI before tests: scripts/check-rule-copies.js byte-compares seven rule-file copies against AGENTS.md and asserts 9 rule invariants survive in SKILL.md+AGENTS.md; scripts/check-versions.js requires all 8 version-bearing manifests to share one pinned semver equal to the release tag.
- Benchmark self-verification: `python benchmarks/agentic/run.py --selftest` validates every task scorer (good ref passes, bad ref caught on its declared axis), the plugin-dir resolver, and the tree-kill — before any API spend; tests/correctness.test.js and benchmarks/correctness.test.js cover scorer units.
- tests/behavior.test.js unit-tests the promptfoo behavior grader (benchmarks/behavior.js) by feeding known behavior-present/absent outputs through each probe checker, making the behavior.yaml eval trustworthy without an API key.
- CI (.github/workflows/test.yml) runs on push to main, tags, and PRs: checkout → Node 22 + Python 3.12 + pandas → npm install for ponytail-mcp → check-rule-copies → check-versions → npm test. There is no coverage tool, lint step, or Windows/macOS CI matrix.
- The Hermes Python plugin (__init__.py) has a dedicated test (tests/hermes-plugin.test.js) but the deep logic — _filter_skill_body_for_mode parity with the JS version, gateway access denial — is exercised through JS-side assertions rather than a Python suite; there is no pytest configuration in the repo. (inferred)
- Uncovered-by-default areas: live model behavior (benchmarks require ANTHROPIC_API_KEY/manual runs and are not in CI), the .ps1 statusline path on Windows (guarded only by hooks-windows.test.js shape checks), and whether hosts actually deliver injected context (docs/cursor-hooks.md notes that's a live-session check). (inferred)