CodeVetter · Repo Unpack · Data flow
sqlite/sqlite
Data enters as UTF-8 (or UTF-16, transcoded at the API edge) SQL text or as bound parameter values. `sqlite3RunParser` drives `sqlite3GetToken` token-by-token into the Lemon parser, which builds `Select`/`Expr`/`SrcList` trees inside a `Parse` object; the code generators turn those trees into a VDBE program of `Op` records. At run time the VDBE owns register memory (`Mem *aMem`) and cursors; reads and writes go through B-Tree cursors into the pager's page cache, and out to disk either as rollback-journal writes or WAL frame appends, always via `sqlite3_vfs` methods. Output leaves either as `sqlite3_column_*` values pulled from `p->pResultRow`, or — in the CLI — through QRF's `xWrite` callback into a `FILE*`. Schema itself round-trips through this same machinery: `sqlite3_schema` rows are re-parsed on load, which is why a schema cookie mismatch forces `sqlite3Reprepare`.
- Text input is bounded at the tokenizer: `sqlite3RunParser` decrements `mxSqlLen` (initialized from `db->aLimit[SQLITE_LIMIT_SQL_LENGTH]`) by each token length and aborts with `SQLITE_TOOBIG` when it goes negative.
- The tokenizer→parser handoff is a push loop: `sqlite3GetToken()` returns a token type and length, context-sensitive keywords are disambiguated by `analyzeWindowKeyword`/`analyzeOverKeyword`/`analyzeFilterKeyword`, and each token is pushed into the Lemon engine with `sqlite3Parser(pEngine, tokenType, pParse->sLastToken)`.
- At end of input the parser is deliberately fed two synthetic tokens — `TK_SEMI` then `0` — so the grammar can reduce cleanly without a trailing semicolon in the source text.
- SQL comments are conditionally dropped in the token stream rather than the grammar: `TK_COMMENT` is skipped only when `db->init.busy` (schema reparse) or `SQLITE_DBCONFIG_ENABLE_COMMENTS` is on, otherwise it falls through to an 'unrecognized token' error.
- UTF-16 inputs are normalized at the boundary: `sqlite3_open16` converts the filename via `sqlite3ValueText(pVal, SQLITE_UTF8)` before calling `openDatabase`, and sets `SCHEMA_ENC`/`ENC` to `SQLITE_UTF16NATIVE` if the schema has not yet loaded.
- Query plans are produced by `sqlite3WhereBegin()`, which chooses the nesting order of FROM-clause tables based on available indexes rather than written order, and emits the cursor-opening code that `sqlite3WhereEnd()` later closes.
- `sqlite3Select()` is the funnel for all query code generation; it takes a `SelectDest` describing where rows go (`SRT_Output`, `SRT_Exists`, `SRT_Fifo`, `SRT_DistQueue`, …), so the same tree compiler serves top-level queries, subqueries, and recursive CTE queues.
- Dirty-page flow at commit is branch-per-journal-mode inside `sqlite3PagerCommitPhaseOne`: in WAL mode it takes `sqlite3PcacheDirtyList()` and calls `pagerWalFrames()` (forcing page 1 into the list if empty, per ticket 2d1a5c67dfc2363e), otherwise it goes down the rollback-journal or batch-atomic-write path.
- The pager state machine is explicit: `sqlite3PagerCommitPhaseOne` asserts the pager is in `PAGER_WRITER_LOCKED`, `WRITER_CACHEMOD`, `WRITER_DBMOD` or `PAGER_ERROR`, and returns early if it has not reached `PAGER_WRITER_CACHEMOD` (i.e. nothing was modified).
- WAL readers snapshot through `sqlite3WalBeginReadTransaction` → `walTryBeginRead` → `walIndexReadHdr`, which reads the shared-memory wal-index header; the shm region itself is created by `unixOpenSharedMemory`/`unixLockSharedMemory` in the unix VFS.
- In the CLI the output boundary is `shell_exec()`: it loops `sqlite3_prepare_v2` over the input buffer using `zLeftover` to advance through multi-statement text, and renders each result set through the QRF spec whose `xWrite` is `shellWriteQR`.
- Interactive input is accumulated line-by-line in `process_input()`, which grows `zSql` by half-again increments, treats a leading `.` or `#` on a fresh buffer as a dot-command, and only executes when a semicolon is present *and* `sqlite3_complete(zSql)` agrees the statement is finished.
- Result rows are owned by the VDBE and invalidated on the next step: `sqlite3Step` returns `SQLITE_ROW` with `p->pResultRow` set, and clears `p->pResultRow = 0` on any non-ROW outcome before running WAL callbacks or transferring errors.
- Error state flows through two channels that must be kept consistent — the per-statement `p->rc` and the per-connection `db->errCode` — reconciled by `sqlite3VdbeTransferError()` only for statements prepared with `SQLITE_PREPARE_SAVESQL`.