CodeVetter · Repo Unpack · System map
ytdl-org/youtube-dl
The system is a monolithic Python package `youtube_dl` with a layered pipeline: CLI parsing (options.py) -> orchestrator (YoutubeDL.py) -> site extractors (extractor/*) -> protocol downloaders (downloader/*) -> postprocessors (postprocessor/*). Shared helpers live in an enormous utils.py and a Python 2/3 compat.py shim. Distribution is via setuptools console script, a Makefile-built single-file zipapp `youtube-dl`, and py2exe; self-update is built in via update.py against yt-dl.org. There is no persistent daemon or service: state is limited to a JSON filesystem cache, a Netscape-format cookie jar, and a download-archive text file.
- The CLI has three equivalent entrypoints: `python -m youtube_dl` (youtube_dl/__main__.py calls youtube_dl.main()), the `bin/youtube-dl` wrapper script, and the setuptools console_scripts entry `youtube-dl = youtube_dl:main` in setup.py.
- All CLI startup logic is `_real_main` in youtube_dl/__init__.py: it calls parseOpts, mutates the global `std_headers` dict for --user-agent/--referer/--add-header, validates option combinations, builds the ~120-key `ydl_opts` dict, then runs `ydl.download(all_urls)` inside a `with YoutubeDL(ydl_opts)` context.
- `YoutubeDL` (YoutubeDL.py) is the central orchestrator: it holds `params`, a list `_ies` of InfoExtractors, `_pps` postprocessors, `cookiejar`, `cache`, and `_opener` (a urllib OpenerDirector); its class docstring enumerates the full options contract used by CLI and library consumers.
- The extractor subsystem is a flat package of ~810 modules under youtube_dl/extractor/, each defining InfoExtractor subclasses; they are aggregated by youtube_dl/extractor/extractors.py into `_ALL_CLASSES` (GenericIE appended last) or, for release builds, by a generated `lazy_extractors.py` module.
- The downloader layer maps an info dict's `protocol` field to a FileDownloader subclass via PROTOCOL_MAP (rtmp, m3u8_native->HlsFD, m3u8->FFmpegFD, mms/rtsp->RtspFD, f4m, http_dash_segments, ism, niconico_dmc), defaulting to HttpFD.
- The postprocessor layer is a small registry: `get_postprocessor(key)` resolves `key + 'PP'` in the module globals; built-ins include FFmpegExtractAudioPP, FFmpegVideoConvertorPP, FFmpegMetadataPP, FFmpegEmbedSubtitlePP, FFmpegMergerPP, EmbedThumbnailPP, XAttrMetadataPP, MetadataFromTitlePP and ExecAfterDownloadPP.
- youtube_dl/utils.py (6863 lines) is the shared toolbox: it defines `std_headers`, `sanitized_Request`, the custom urllib handlers (YoutubeDLHandler, YoutubeDLRedirectHandler, YoutubeDLCookieProcessor, PerRequestProxyHandler), `YoutubeDLCookieJar`, `sanitize_filename`, `GeoUtils`, `match_filter_func`, exception classes, and `DEFAULT_OUTTMPL`.
- youtube_dl/compat.py (3901 lines) is a hand-rolled Python 2.6-3.x compatibility layer aliasing stdlib names as `compat_*` symbols; setup.py classifiers advertise Python 2.6 through 3.8 plus Jython/PyPy/IronPython.
- All HTTP(S) traffic goes through `YoutubeDL._setup_opener`, which builds a urllib opener from PerRequestProxyHandler, a params-aware HTTPS handler, YoutubeDLCookieProcessor, YoutubeDLHandler (compression + URL fixups), YoutubeDLRedirectHandler, a DataHandler, and a FileHandler whose `file_open` is stubbed to disable the file:// scheme for security.
- The release build produces a self-contained executable: Makefile target `youtube-dl` copies the package into a zip with a `__main__.py` at the zip root, prepends a python shebang and chmod +x; Windows builds use py2exe options embedded in setup.py.
- Filesystem cache is implemented by `Cache` in youtube_dl/cache.py storing JSON under `$XDG_CACHE_HOME/youtube-dl` (or --cache-dir), keyed by section/key, versioned per release with a minimum-version invalidation check.
- External executables are runtime dependencies probed lazily: ffmpeg/avconv/ffprobe located by FFmpegPostProcessor._determine_executables, rtmpdump via downloader/rtmp.py's rtmpdump_version, and phantomjs via extractor/openload.py's PhantomJSwrapper._version; all three are reported in the verbose debug header.
- Tests and lint are driven by `nosetests` (test/ directory, ~34 test modules) via Makefile `test`/`offlinetest` targets and devscripts/run_tests.sh, plus a `flake8 .` codetest target configured in setup.cfg.