CodeVetter · Repo Unpack · Testing signals
nextlevelbuilder/ui-ux-pro-max-skill
Testing is unusually heavy for a dataset repo: 13 stdlib unittest modules under src/ui-ux-pro-max/scripts/tests/ (mirrored into .claude/skills), a fixture-driven relevance evaluator with fingerprint-pinned thresholds, 5 Playwright e2e specs for the CLI (run as a Node test harness, not browser tests), node --test unit tests for the gallery, two bash smoke suites over all domains/stacks, and four validation scripts (CSV structure, semantic data contract, agent-guide claims, catalog summary). The aggregate `npm run verify:data` gate runs all of it and is wired into tests.yml, smoke-stacks.yml, release.yml, and prepublishOnly.
- Python tests live in src/ui-ux-pro-max/scripts/tests/ as 13 unittest modules (test_core, test_design_system_mode, test_data_contracts, test_style_taxonomy, test_catalog_refresh, test_relevance_evaluator, test_web_stack_freshness, etc.) deliberately stdlib-only so they add no dependencies; CI runs them via `python3 -m pytest .claude/skills -v`.
- test_core.py covers tokenizer synonym/stopword behavior, BM25 edge cases, and cache invalidation by file signature (re-writes a temp CSV and asserts the index rebuilds).
- The relevance harness uses fixtures (relevance-cases.json, relevance-thresholds.json, relevance-baseline.json) and graded judgments to compute routingAccuracy, precision@1/3, MRR@3, nDCG@3, negativeAbstention, typoRecoveryAt3, and designSystemCoherence, gated by check_thresholds with calibration vs held_out splits.
- Hard-negative cases tagged 'hard-negative' are run against every domain and every stack to measure false-positive abstention (globalNegativeApplicability), so retrieval regressions that over-match are caught by the gate.
- validate_data.py is a 1095-line semantic contract: it enforces exactly-192-row parity across products/colors/ui-reasoning, WCAG contrast floors per palette pair, style status/parent/redirect invariants, decision-rule validity against the closed grammar, icon import regexes, and per-stack official-doc host allowlists.
- validate-csv.py checks structural CSV hygiene (blank/duplicate headers, field-count mismatches, blank rows) across every data CSV; validate-agent-guide.py recomputes the advertised counts (styles, palettes, fonts, stacks) and asserts the SKILL.md/template descriptions and CLI examples stay accurate.
- smoke-domains.sh dynamically enumerates CSV_CONFIG domains (expected count 12 via EXPECTED_DOMAIN_COUNT) and asserts a focused probe returns >=1 result per domain, printing diagnostics on failure; smoke-stacks.sh does the same for stacks.
- CLI e2e specs (cli/tests/e2e/) use @playwright/test as a plain Node harness: dry-run.spec.ts asserts planPlatformInstallActions lists writes but touches nothing, template-script-paths.spec.ts and agent-query-contract.spec.ts pin the rendered skill content contract; playwright.config.ts targets chromium with CI retries.
- Gallery tests run via `tsc -p tsconfig.test.json && node --test` over compiled JS (lib/parseStyles.test.js, components/metadata-badges.test.js) — an unusual compile-then-test pipeline rather than jest/vitest.
- check-asset-sync.yml embeds two grep-based contract suites: skill-relative path enforcement (no ~/.claude/skills/, project-rooted, or CLAUDE_PLUGIN_ROOT paths outside the one allowed SKILL.md) and a bundled-skill allowlist scan over every shipped markdown file.
- Coverage gaps: no tests exercise the CLI's actual write path (generatePlatformFiles is only tested via the read-only planners), uninstall.ts has no spec, and format_ascii_box/markdown output is only implicitly covered through persistence tests. (inferred)
- release.yml runs the full gate (verify:data, bun build, Playwright, gallery build) before semantic-release publishes to npm with NPM_TOKEN and creates the GitHub release — so untested code cannot ship.