CodeVetter · Repo Unpack · Testing signals
msitarzewski/agency-agents
Tests are shell scripts and one unittest module, not a pytest or npm suite. They are designed for bash 3.2 and no jq. CI runs installer behavior on Linux and macOS, catalog consistency on every PR, and agent lint only when agent or linter paths change. The generated-output eval needs PyYAML and Python 3.11 tomllib and treats manifest drift as advisory on pull requests. Large gaps are the converter installers other than the two identity tools, and the agent prompt bodies themselves, which no test executes.
- test-install.sh is a custom bash harness (pass/fail/xfail_eq, not a framework). It sets HOME to a mktemp sandbox, refuses to touch the real home, and only installs claude-code and copilot so it does not depend on fresh convert.sh output. It checks --help, --list teams against divisions.json, --dry-run writes zero files, default ~/.claude/agents count equals the frontmatter roster, and --path overrides the default.
- test-install.yml also runs bash -n on every scripts/*.sh before the harness, on ubuntu-latest and macos-latest, fail-fast false, for every pull_request and every push to main.
- test-agent-selection.sh asserts --agent definitely-not-an-agent and a bad --agents-file exit non-zero with 'Unknown agent' / 'in agents-file', and that --agent 'Developer Tooling Engineer' dry-runs as Agents: 1. check-tools.yml runs it on every PR and main push.
- test-convert-frontmatter.sh runs convert.sh for gemini-cli, opencode, and qwen into a temp dir and asserts selected frontmatter fields are single-quoted YAML scalars, including description on developer-tooling-engineer and tools on programmatic-display-buyer.
- test-convert-outputs.sh is the generated-product eval: round-trip of description, strict YAML/TOML parse, one output per roster agent per tool, and sha256 drift against scripts/convert-outputs.sha256. It requires python3 with yaml and tomllib. CI passes --drift=advisory on pull_request and strict mode on push to main.
- check-hermes-plugin.py loads build-hermes-plugin.py, builds into a temp dir, calls register on a RecordingContext, asserts the four tool names and JSON-schema shape, and runs a live search for 'backend architecture' then inspect by the returned slug.
- test-hermes-plugin.py is unittest. It imports the committed integrations/hermes/agency-agents-router/__init__.py if present and drives FakeLifecycle through State.SUCCEEDED, FAILED, CANCELLED, timeouts, and launch errors. That generated __init__.py is gitignored, so a clean checkout does not contain the module the test imports until convert.sh --tool hermes has been run.
- check-hermes-config-rewrite.sh is a regression for the plugins.enabled indent bug. The workflow runs it on every PR and main push and documents the two symptoms: wrong indent collapsing enabled into a scalar, and non-idempotent duplicate inserts.
- check-divisions.sh and check-runbooks.sh are CI gates, not unit tests. Divisions must match disk, both AGENT_DIRS arrays, lint-agents.yml path filters, and each division must contain at least one frontmatter file. Runbooks must be valid JSON, have unique slugs, existing doc paths, and roster stems that git ls-files can resolve.
- lint-agents.yml runs only on pull_request, path-filtered to division trees plus scripts/lint-agents.sh, convert.sh, and lib.sh. The lint job lints git-diff changed agent files and then check-agent-originality.sh on those files. A second job lints the entire roster only when the linter, converter, or lib.sh changed.
- check-agent-originality.sh embeds Python that builds 8-word shingles after stripping a fixed ENTITY proper-noun list, then Jaccard-compares candidates to the roster. ORIGINALITY_FAIL defaults to 40 and ORIGINALITY_WARN to 20. Exit 1 at or above the fail threshold. The header says the existing library's worst pair is about 1.5 percent.
- What is not covered: no test executes an agent prompt; test-install.sh never calls install_opencode, install_cursor, install_hermes, or install_openclaw; lint does not run on push to main; and there is no coverage tool, fixture app, or browser suite. Agent markdown under testing/ is a division of QA personas, not this repo's test suite.