CodeVetter · Repo Unpack · Data flow
yt-dlp/yt-dlp
Data flows: CLI args + config files -> `ydl_opts` params dict -> URL list -> extractor selection by `_VALID_URL` regex -> `ie.extract()` performs HTTP via the shared RequestDirector and returns an info dict -> `process_ie_result` normalizes and recursively resolves playlists/URL results -> format selection computes `requested_formats` -> `process_info` writes sidecars and calls `dl()` -> a protocol FileDownloader streams bytes to `.part`/final files -> post-processors mutate files -> archive/cache/sidecar JSON are the persistence outputs. The info dict is the single mutable data structure threaded through the whole pipeline.
- Input boundary: `get_urls` merges argv URLs with batch-file/stdin lines; `parse_options` then produces the flat `ydl_opts` params dict consumed by `YoutubeDL`.
- Extractor dispatch: `YoutubeDL.extract_info` iterates `self._ies` and picks the first extractor whose `suitable(url)` — a `_match_valid_url` regex match on `_VALID_URL` — returns true; no match yields 'No suitable extractor'.
- `InfoExtractor.extract` runs `initialize()` (pre-login, `_perform_login`, geo-bypass setup, `_real_initialize`) then `_real_extract(url)`, and can retry once with a fake IP on GeoRestrictedError; errors are normalized into `ExtractorError`.
- All extractor HTTP flows through `self._downloader.urlopen` -> `_request_director.send`, which applies basic-auth extraction, `sanitize_url`, proxy/header cleanup, then per-handler `validate`/`send` in preference order.
- After extraction, `add_default_extra_info` stamps `webpage_url`, `original_url`, `extractor`, `extractor_key` onto the info dict; `process_ie_result` then dispatches on `_type` (video/playlist/url/compat_list) and `__process_playlist`/`__process_iterable_entry` recurse into entries via `extract_info`.
- Format selection: `build_format_selector` parses the `-f` spec into selector functions applied in `process_video_result`, producing `requested_formats` (split A/V) or a single format on the info dict.
- `process_info` computes filenames via `prepare_filename`, writes description/subtitles/thumbnails sidecars, then calls `self.dl(temp_filename, info_dict)`; multi-format downloads iterate `requested_formats` into separate files later merged.
- `dl()` picks a FileDownloader via `get_suitable_downloader`, attaches progress hooks, and calls `fd.download(name, new_info)`; `FileDownloader.download` handles skip-if-present, sleep intervals, then `real_download` writes to `filename + '.part'` with resume metadata in `filename + '.ytdl'`.
- Post-processing is staged: `run_all_pps` iterates `self._pps[key]` for each `POSTPROCESS_WHEN` stage ('pre_process','after_filter','video','before_dl','post_process','after_move','after_video','playlist'), each `pp.run(infodict)` returning files to delete/move.
- Output boundaries: media files plus optional sidecars (`.info.json` via `_write_info_json`, `.description`, subtitle files, thumbnails), the download archive line append, and stdout JSON from `sanitize_info` (strips `__private` keys and non-serializable objects to repr).
- Cookies enter via `load_cookies` (browser extraction or Netscape file) into a `YoutubeDLCookieJar`, are attached to every request handler in `build_request_director`, and persisted on `YoutubeDL.__exit__`/`save_cookies`.
- Error flow: `_handle_extraction_exceptions` wraps extraction — `ReExtractInfo` loops, `GeoRestrictedError` reports with VPN hint, `ExtractorError` reports with optional traceback, `ignoreerrors` downgrades to report-only; the CLI exit code is `ydl._download_retcode` (e.g. 101 for DownloadCancelled).