CodeVetter · Repo Unpack · Data flow
Chalarangelo/30-seconds-of-code
Data flows one direction: content/ sources → extraction (FileHandler globs + front-matter/YAML parsing) → per-model workers that render HTML, tokenize, rank, and build join records → .content/content.json → in-memory model graph via Loader reviver → page adapters + serializers → .content/pages/*.json plus public/ artifacts (search-data.json, feed.xml, sitemap.xml, _redirects) → Astro getStaticPaths → static HTML. The browser is a terminal consumer: it fetches search-data.json and rebuilds the same DocumentIndex used at build time. There is no runtime server state; 'state owners' are the static Model registries hydrated once per build/console process.
- Input boundaries are globbed files: content/snippets/**/s/*.md (front-matter + markdown), content/collections/**/*.yaml, content/languages/*.yaml, plus singleton files grammars.yaml, rankingEngine.yaml, hub.yaml — all read through FileHandler.read which dispatches on extension to JSON/YAML/front-matter readers.
- extractData is the single orchestration point: it loads languages, grammars, hub, and keyword data; selects Shiki or Prism as codeHighlighter; calls MarkdownParser.setupProcessors and sets Ranker.keywordScores; then extracts snippets, collections, and collectionSnippets in that order.
- Snippet extraction derives id from filePath by stripping the 'content/snippets/' prefix and '.md' suffix, renders excerpt and body to HTML through a per-language MarkdownParser processor, computes recTokens and docTokens via tokenize(), computes a 0-1 ranking via Ranker.rankIndexableContent, and builds tableOfContentsHtml via TocReader when tocEnabled.
- MarkdownParser maintains one unified() processor per language short-code plus a 'default'; each processor embeds that language's reference map (merged from allLanguageReferences) for linkInlineCode, so inline-code autolinking is per-language.
- exportXData functions flatten extracted objects into plain records tagged with a 'model' field ('Snippet', 'Collection', 'Language', 'CollectionSnippet'); ContentUtils.prepareContent writes the aggregate to .content/content.json via FileHandler.write.
- CollectionSnippet join records are produced by extractCollectionSnippetData: explicit snippetIds are looked up with a console.warn on missing, matcher-based collections filter ranked snippets by languageKey/tags, the 'snippets' main collection gets position = rank index (-1 for unlisted), and allowUnlisted controls inclusion.
- On load, every model constructor pushes itself into Model.instances[name] and registers each declared index into Model.indexedInstances[name][index] maps; all Snippet and Collection instances are indexed on 'id'.
- Page serialization path: AstroContent.generate calls Collection.pages / snippet.page / Page.home, each Page subclass exposes params and props getters, PageSerializer.serialize emits {params, props:{...props, structuredData: schemaData}}, and results are keyed by page.key into JSON files under .content/pages/.
- Astro routes are pure consumers of generated JSON: [snippet].astro and [...listing].astro implement getStaticPaths by reading the corresponding .content JSON file and mapping entries to {params, props}; index.astro reads index.json synchronously at module scope.
- Search data path: snippet/collection docTokens strings ('token:count' space-joined) are serialized by SearchResultSerializer into public/search-data.json; the browser fetches it, deserializeTokens rebuilds Map<term,freq>, and DocumentIndex re-derives trigram sets for fuzzy matching.
- Ranker scores content against the sorted keyword table from content/rankingEngine.yaml, capping at 20 keywords or a score of 100, and returns max(0.0001, score/100) so no snippet ranks exactly 0.
- Redirects flow: content/redirects.yaml is loaded once in a static block, Redirects.generate writes 'from to status' lines to public/_redirects, and Redirects.for(slug) does a transitive lookup used by ContentModel.allSlugs for canonical/alias handling.
- TimestampDump.generate writes a hex timestamp to src/astro/timestamp.js, which Layout.astro injects via define:vars into an empty script solely to force HMR when content regenerates during watch mode.