CodeVetter · Repo Unpack · Data flow
ytdl-org/youtube-dl
Data moves in one direction through a well-defined pipeline. Inputs: CLI args + config files + batch files produce a URL list; the params dict is the single source of truth shared by all layers. Extractors turn a URL into an 'info dict' (a documented schema with _type discriminators video/url/url_transparent/playlist); process_ie_result resolves indirection recursively; process_video_result normalizes fields and selects formats; process_info emits sidecar files and invokes a protocol-matched FileDownloader that writes .part files and renames on completion; the postprocessor chain mutates/moves the final file; bookkeeping lands in the download archive, cookie jar, and JSON cache.
- URL input boundary: positional args plus lines from --batch-file (or stdin via '-') are merged into all_urls; bare URLs are decoded with the preferred encoding before reaching the extractor loop.
- Configuration input boundary: parseOpts merges /etc/youtube-dl.conf, user config (XDG_CONFIG_HOME/youtube-dl/config, ~/.config/youtube-dl.conf, %APPDATA% variants, ~/youtube-dl.conf), --config-location file, then argv — later entries override earlier ones; --ignore-config skips them.
- The info dict is the canonical intermediate representation: extractor/common.py's class docstring specifies mandatory fields (id, title, and url-or-formats) plus ~50 optional fields and the _type discriminator values (video, playlist, multi_video, url, url_transparent, compat_list).
- extract_info iterates self._ies calling ie.suitable(url) (compiled _VALID_URL regexes); the first match's extract() runs initialize() (geo-bypass, login) then _real_extract(url); results are stamped with extractor/webpage_url/extractor_key by add_default_extra_info.
- process_ie_result recursively resolves indirection: 'url'/'url_transparent' results re-enter extract_info (possibly pinned to an ie_key); 'playlist'/'multi_video'/'compat_list' results go to __process_playlist which slices entries, injects playlist_* extra_info, and reprocesses each entry via __process_iterable_entry.
- process_video_result normalizes a video info dict: forces string/numeric field types, sorts thumbnails, fills common fields, computes requested_subtitles, filters malformed formats, sorts formats with _sort_formats, and runs the parsed format selector (from build_format_selector) to produce formats_to_download or requested_formats for A+V merge.
- process_info is the output boundary: it computes the filename via prepare_filename, writes .description/.annotations.xml/subtitle/.info.json/thumbnail files first, then calls dl() which instantiates the downloader chosen by get_suitable_downloader and copies http_headers via _calc_headers.
- HttpFD.real_download streams to `filename + '.part'` opened just-in-time via sanitize_open, honoring resume (Range header + Content-Range validation), retry on 5xx/ECONNRESET/timeout, min/max filesize aborts, then try_rename to the final name and try_utime from Last-Modified.
- Fragmented downloads (HlsFD, DashSegmentsFD, F4mFD, IsmFD) share FragmentFD, which keeps a `.ytdl` JSON bookkeeping file per incomplete download (extractor/downloader state, current_fragment, fragment_count) enabling resume and consistency checks.
- Postprocessing output boundary: post_process chains info_dict['__postprocessors'] (e.g. FFmpegMergerPP for A+V merge, fixup PPs) before self._pps; each pp.run returns (files_to_delete, info); originals are deleted unless keepvideo.
- State owners: the cookie jar (YoutubeDLCookieJar, Netscape format, optionally file-backed via --cookies) is owned by YoutubeDL._setup_opener; extractor/player caches (e.g. YouTube 'sigfuncs') live under the Cache section/key namespace; the download archive is a plain text append log.
- Header/cookie hygiene is enforced at boundaries: FileDownloader.download documents stripping Cookie headers; _remove_cookie_header filters them from http_headers; header-supplied cookies are rescoped per-URL domain by _apply_header_cookies to prevent cross-domain leaks.