Runtime performance lab publication record

Published product experiments, external contributions, evidence strength, and local-cost boundaries from the first CodeVetter performance campaign.

Runtime performance lab publication record

This page records what left the local performance laboratory on 2026-08-10. It is a publication ledger, not a second copy of the benchmark evidence. The canonical runtime contracts and commands remain in performance.md, while the active OpenSpec qualification records preserve the complete observations and limitations.

Evidence labels

• Confirmed means CodeVetter compared the same workload and metric under a compatible policy and the candidate cleared the materiality threshold.
• Directional means repeated measurements support the candidate, but a retained paired baseline is missing.
• Guardrail means the flow was already fast or the artifact primarily protects against future regression.
• Unverified means the source change is plausible and correctness-tested, but CodeVetter has not measured a defensible before/after effect.
• Rejected means a measured candidate regressed, stayed below policy, or failed to explain the captured workload. Rejected changes were reverted.

These labels separate observed runtime evidence from agent inference. An open pull request is not itself evidence that an optimization works.

Fleet product pull requests

Nine PRs below are merged. Three remain open because merge would cross a recorded safety or verification boundary. The campaign used dedicated branches without a deployment, migration, production database, hosted-product load test, or paid model call.

The strongest product results are Significant Hobbies, Anime List, and Web Playables. App Health is mechanically real but too small to represent the product’s end game. Merge status does not upgrade evidence strength: the directional, unverified, guardrail, and synthetic-stress entries are not independent claims of proven customer impact. The full inactive-project disposition is retained in artifacts/performance/old-local-projects-results-2026-08-09.md.

External open-source work

Marked

markedjs/marked#4048 was the first external upstream PR created by this campaign. It optimizes reference-link membership by checking the existing link table directly instead of rebuilding and linearly searching its keys for every queued inline source. The revised patch contains no duplicate lexer state.

Thirty alternating fresh-process pairs with per-process warmup measured:

A 2,000-paragraph no-definition control moved +1.3% with a +0.3% to +2.3% interval. Reintroducing Object.keys or using for...in restored that fast path but re-enumerated the full link table and erased most of the target improvement, so neither experiment was retained. Marked’s complete npm test passed: 190 unit tests, 1,779 specification tests, ESM/UMD/CJS and type builds, lint, and generated-output checks. The PR is open and no longer a draft. Two maintainer threads on the earlier implementation are outdated. Snyk passes; Vercel’s fork preview requires upstream authorization and is not a code-test failure.

qs

ljharb/qs#592 adds a guarded fast path for entirely flat query strings. A fresh ten-pair interleaved publication run measured 41.706 to 23.183 ms/op at 40,000 parameters, or 44.413%, and returned shipping_recommended: true with no limitations. The 1,000- and 10,000-parameter inputs improved 32.902% and 47.334%.

All 1,045 upstream tests passed with 100% statement and line coverage. The PR also documents the two unsafe broader candidates that the correctness suite rejected before the final fallback boundary was retained. It is ready for review; upstream maintainers still own the merge decision.

No upstream PR submitted

• go-chi/chi: the profiler found allocation pressure in the router path, but inspection tied it to required http.Request.WithContext and context semantics. No unsafe patch was attempted.
• Picomatch: the candidate stayed below the materiality policy.
• Pixelmatch: the candidate regressed and was reverted.
• GJSON: the selected benchmark already reported zero allocations and no material bottleneck.

Negative results matter: they show that CodeVetter can withhold a patch when the evidence does not justify one.

What CodeVetter gained

The implementation and all qualification artifacts merged through CodeVetter PR #108. Follow-up PR #109 retains paired Vitest domain metrics, source-anchors campaign hotspot lines while preserving raw coordinates, and records the Calorie evidence boundary. Both post-merge CI runs passed the complete desktop, CLI, MCP, and browser pipeline.

The campaign expanded CodeVetter from a one-shot profiler into a bounded local optimization laboratory:

1. Runtime failure capsules capture failing Node/Vitest executions with redaction, source maps, Git-diff relevance, and deterministic evidence versus inference.
2. Performance capsules support Node tests, Node scripts, Vitest, and Go benchmarks with repeated metrics, dual-profile source attribution, scale curves, allocation evidence, and absolute-cost refusal.
3. Local flow tools expose closed capture, inspect, explain, and verify operations over MCP without accepting arbitrary commands after startup.
4. Optimization verification returns confirmed, rejected, inconclusive, or no-confidence and supports alternating paired checkouts.
5. Qualification planning discovers safe repository-owned workloads, excludes cloud and remote operations, and emits one bounded next action.
6. Durable supervision records resumable runs, budgets, exact revisions, evidence digests, cleanup state, and terminal reasons.
7. Autonomous campaigns keep correctness gates and the evaluator immutable while an agent proposes one source experiment at a time.
8. Verification receipts ingest project-runner evidence, compare regression state, and explain affected-test blast radius.
9. Scaled challenge artifacts provide Node and Go temperature aggregation workloads inspired by the One Billion Row Challenge without claiming an official submission.
10. Self-profiling found and fixed repeated source-offset scans in V8 function coverage. The representative 91-document replay improved 18.11%; the 98.808% number belongs only to the adversarial regression fixture.
11. Contribution closeout now challenges retained patch complexity, binds local and optional T-Rex evidence to the pull-request head, reads current and outdated review threads, distinguishes failed checks from fork approval, and stops at upstream ownership without posting or assigning maintainer work.

The most important tooling improvements came from false or incomplete early results: startup-dominated tests now fail closed, Vitest names must identify one exact assertion, console metrics use repeated unprofiled medians, TypeScript locations anchor to original source, unsupported scale semantics stay unverified, and an inference must explain the captured workload before it is allowed to guide a patch.

One Billion Row Challenge boundary

The checked-in artifact is a local learning workload, not an official 1BRC result. The Node parser improved from 118.928 to 35.744 ms/op at 800,000 rows, or 69.945%, after CodeVetter selected the parser before source inspection. A later 8.863% micro-optimization was rejected and reverted.

The Go lane adds streaming and parallel parsing experiments plus a recorded 100-million-row result. It does not claim parity with the best public 1BRC implementations, one-billion-row completion, or hardware-independent speed. That distinction is preserved in the artifact README and result record.

Resource and safety boundary

• Product profiling was local and bounded; it did not hit production databases or hosted product endpoints.
• No Fleet product was deployed, migrated, released, or load-tested in the cloud.
• The Email Manager golden-fixture check fetched three small public npm package artifacts through its existing pnpm dlx command.
• Existing package stores and locally generated build/profile artifacts are not evidence of equivalent network transfer or cloud spend.
• Raw owned profiles were deleted after normalization. Durable campaign data is bounded under .codevetter/performance-runs/ or the campaign’s declared artifact directory.

Publication rule

Future campaign output should enter this ledger only when it has a durable branch, issue, or PR URL and an explicit evidence label. Merge status belongs to GitHub; benchmark truth belongs to the reproducible CodeVetter artifact.