CodeVetter · Repo Unpack · Data flow
Graphify-Labs/graphify
Input boundary is detect(), which walks the corpus, classifies files into code/document/paper/image/video, and filters sensitive paths. Code files flow through extract(): suffix/filename/shebang dispatch to per-language extractors emitting {nodes, edges} dicts, then cross-file resolution passes (LanguageResolver registry) turn imports/member calls into INFERRED edges. Docs/papers/images flow through the semantic pass — either host-agent subagents writing chunk JSONs (skill.md path) or llm.extract_corpus_parallel packing token-budgeted chunks against a configured backend. Merged extractions are validated, deduplicated, built into a NetworkX graph, clustered, and exported to graphify-out/graph.json plus reports; incremental updates re-extract only changed files via manifest.json hashing.
- detect() classifies files via CODE_EXTENSIONS (~90 suffixes), DOC_EXTENSIONS, PAPER/IMAGE/VIDEO/OFFICE extension sets, a shebang interpreter check for extensionless executables, and filters _SENSITIVE_PATTERNS (.env, .pem, id_rsa, .npmrc, ...) and credential-store dirs (.ssh/.aws/.gnupg/.gcloud) before anything is read for extraction.
- Office/PDF inputs are screened as potential zip/XML bombs before parsing: _zip_within_caps enforces a 50 MiB raw cap, 512 MiB decompressed cap, and 200:1 compression-ratio limit with chunked streaming so the check itself never materializes the bomb.
- _get_extractor() dispatches a file to its extractor by exact filename first (.blade.php, MCP config, package manifests), then content sniffing for ambiguous suffixes (.h ObjC/C++ markers, .m ObjC-vs-MATLAB), then shebang for extensionless files (_SHEBANG_DISPATCH), then the _DISPATCH suffix table covering ~40 languages.
- Each extractor returns {nodes:[{id,label,source_file,source_location}], edges:[{source,target,relation,confidence}]}; validate.py::validate_extraction enforces this schema before build() consumes it, and confidence is EXTRACTED/INFERRED/AMBIGUOUS.
- extract() is two-phase: phase 1 checks the per-file AST cache (load_cached, keyed by content hash under cache/ast/v{pkg_version}-s{schema}/); phase 2 runs uncached files through ProcessPoolExecutor or sequentially, then runs registered LanguageResolver passes gated on which suffixes are present.
- The semantic pass wraps every file in a hash-stamped `<untrusted_source path=... sha256=...>` block, neutralizes known jailbreak tokens via _neutralise_injection_sentinels, packs chunks by tiktoken/char-estimate token budgets (_pack_chunks_by_tokens), and parses LLM output with _parse_llm_json's balanced-object salvage.
- The skill path's merge order is AST nodes first, then semantic nodes deduplicated by id — so a semantic node silently overwrites an AST node with the same id (richer labels win); build_from_json then mints `external` stub nodes for import-family edges whose targets are outside the graph.
- build.py collapses parallel edges to a single edge per (source,target) pair with a 'specific beats generic' precedence — references/uses/mentions lose to calls/imports/inherits — and drops cross-language-family phantom edges via _EDGE_LANG_FAMILY.
- cluster() partitions the graph (Leiden via graspologic or native backend, hub exclusion optional), analyze.py computes god_nodes/surprising_connections/suggest_questions/find_import_cycles, and report.generate() renders GRAPH_REPORT.md; export.to_json writes graph.json but refuses to overwrite a larger existing graph (shrink guard, bypassed by --force/GRAPHIFY_FORCE).
- Incremental updates are driven by graphify-out/manifest.json: detect_incremental diffs the corpus against stamped per-file hashes; _stamped_manifest_files only re-stamps semantic files that actually produced output so failed/partial extractions are re-queued next run; _stale_graph_sources prunes nodes whose source_file left the corpus.
- Read path: `graphify query` loads graph.json (size-capped by check_graph_file_size_cap, default 512 MiB, overridable via GRAPHIFY_MAX_GRAPH_BYTES), rebuilds an undirected NetworkX graph preserving per-edge _src/_tgt direction markers, runs _query_graph_text BFS/DFS, logs to querylog, and touches a query stamp.
- Post-commit hook flow: git hook shell block resolves a Python interpreter, detached-launches a python -c payload that sets a timeout watchdog, queues changed paths via .pending_changes, takes a non-blocking flock, and calls _rebuild_code; a concurrent commit's changes are merged in by the lock holder rather than dropped.