CodeVetter · Repo Unpack evidence
ytdl-org/youtube-dl
github.com/ytdl-org/youtube-dl ↗ · 980 files · 6.6 MB
youtube-dl - download videos from youtube.com or other video platforms - INSTALLATION - DESCRIPTION - OPTIONS - CONFIGURATION - OUTPUT TEMPLATE - FORMAT SELECTION - VIDEO SELECTION - FAQ - DEVELOPER INSTRUCTIONS - EMBEDDING YOUTUBE-DL - BUGS - COPYRIGHT INSTAL
Overview
youtube-dl is a dependency-free Python command-line program (and embeddable library) that downloads video and audio from YouTube and ~1200 other sites. A single YoutubeDL orchestrator class routes each user-supplied URL to the first matching InfoExtractor (regex-matched from a registry of ~810 extractor modules), normalizes the returned 'info dict', selects a format via a small DSL, downloads via a protocol-mapped FileDownloader (native HTTP/HLS/DASH or external tools like ffmpeg/aria2c), and runs a postprocessor chain (ffmpeg muxing, metadata, thumbnails). It still maintains Python 2.6 compatibility through a large compat shim and ships as a pip package, a self-updating single-file zip executable, and a py2exe Windows binary.
System map
The system is a monolithic Python package `youtube_dl` with a layered pipeline: CLI parsing (options.py) -> orchestrator (YoutubeDL.py) -> site extractors (extractor/*) -> protocol downloaders (downloader/*) -> postprocessors (postprocessor/*). Shared helpers live in an enormous utils.py and a Python 2/3 compat.py shim. Distribution is via setuptools console script, a Makefile-built single-file zipapp `youtube-dl`, and py2exe; self-update is built in via update.py against yt-dl.org. There is no persistent daemon or service: state is limited to a JSON filesystem cache, a Netscape-format cookie jar, and a download-archive text file.
- The CLI has three equivalent entrypoints: `python -m youtube_dl` (youtube_dl/__main__.py calls youtube_dl.main()), the `bin/youtube-dl` wrapper script, and the setuptools console_scripts entry `youtube-dl = youtube_dl:main` in setup.py.
- All CLI startup logic is `_real_main` in youtube_dl/__init__.py: it calls parseOpts, mutates the global `std_headers` dict for --user-agent/--referer/--add-header, validates option combinations, builds the ~120-key `ydl_opts` dict, then runs `ydl.download(all_urls)` inside a `with YoutubeDL(ydl_opts)` context.
- `YoutubeDL` (YoutubeDL.py) is the central orchestrator: it holds `params`, a list `_ies` of InfoExtractors, `_pps` postprocessors, `cookiejar`, `cache`, and `_opener` (a urllib OpenerDirector); its class docstring enumerates the full options contract used by CLI and library consumers.
- The extractor subsystem is a flat package of ~810 modules under youtube_dl/extractor/, each defining InfoExtractor subclasses; they are aggregated by youtube_dl/extractor/extractors.py into `_ALL_CLASSES` (GenericIE appended last) or, for release builds, by a generated `lazy_extractors.py` module.
- The downloader layer maps an info dict's `protocol` field to a FileDownloader subclass via PROTOCOL_MAP (rtmp, m3u8_native->HlsFD, m3u8->FFmpegFD, mms/rtsp->RtspFD, f4m, http_dash_segments, ism, niconico_dmc), defaulting to HttpFD.
- The postprocessor layer is a small registry: `get_postprocessor(key)` resolves `key + 'PP'` in the module globals; built-ins include FFmpegExtractAudioPP, FFmpegVideoConvertorPP, FFmpegMetadataPP, FFmpegEmbedSubtitlePP, FFmpegMergerPP, EmbedThumbnailPP, XAttrMetadataPP, MetadataFromTitlePP and ExecAfterDownloadPP.
- youtube_dl/utils.py (6863 lines) is the shared toolbox: it defines `std_headers`, `sanitized_Request`, the custom urllib handlers (YoutubeDLHandler, YoutubeDLRedirectHandler, YoutubeDLCookieProcessor, PerRequestProxyHandler), `YoutubeDLCookieJar`, `sanitize_filename`, `GeoUtils`, `match_filter_func`, exception classes, and `DEFAULT_OUTTMPL`.
- youtube_dl/compat.py (3901 lines) is a hand-rolled Python 2.6-3.x compatibility layer aliasing stdlib names as `compat_*` symbols; setup.py classifiers advertise Python 2.6 through 3.8 plus Jython/PyPy/IronPython.
- All HTTP(S) traffic goes through `YoutubeDL._setup_opener`, which builds a urllib opener from PerRequestProxyHandler, a params-aware HTTPS handler, YoutubeDLCookieProcessor, YoutubeDLHandler (compression + URL fixups), YoutubeDLRedirectHandler, a DataHandler, and a FileHandler whose `file_open` is stubbed to disable the file:// scheme for security.
- The release build produces a self-contained executable: Makefile target `youtube-dl` copies the package into a zip with a `__main__.py` at the zip root, prepends a python shebang and chmod +x; Windows builds use py2exe options embedded in setup.py.
- Filesystem cache is implemented by `Cache` in youtube_dl/cache.py storing JSON under `$XDG_CACHE_HOME/youtube-dl` (or --cache-dir), keyed by section/key, versioned per release with a minimum-version invalidation check.
- External executables are runtime dependencies probed lazily: ffmpeg/avconv/ffprobe located by FFmpegPostProcessor._determine_executables, rtmpdump via downloader/rtmp.py's rtmpdump_version, and phantomjs via extractor/openload.py's PhantomJSwrapper._version; all three are reported in the verbose debug header.
Feature catalog
User-facing surface = one CLI command `youtube-dl` with ~200 options defined in options.py plus the embeddable YoutubeDL API. Features cluster around: site extraction (~1200 supported sites), format/playlist/subtitle selection, sidecar file output (info JSON, description, annotations, thumbnails), ffmpeg-based postprocessing (audio extraction, recoding, embedding), authentication (username/password, netrc, video passwords, Adobe Pass TV providers), geo-restriction bypass, external downloaders, self-update, and simulation/printing modes for scripting.
- Site support: ~810 extractor modules under youtube_dl/extractor/ back roughly 1225 documented sites in docs/supportedsites.md; `--list-extractors` and `--extractor-descriptions` enumerate them at runtime via list_extractors().
- Format selection via `-f/--format` implements a mini-DSL (best/worst, format_id, filters like `[height<=?720]`, merge with `+`, fallback with `/`, groups) parsed by `YoutubeDL.build_format_selector` into selector functions.
- Playlist features include --playlist-start/--playlist-end, --playlist-items (comma list/ranges), --playlist-reverse, --playlist-random, --max-downloads, and --flat-playlist (extract_flat), implemented in __process_playlist.
- Subtitle support: --write-sub/--write-auto-subs/--all-subs/--list-subs download or enumerate subtitle tracks, with --sub-format and --sub-langs filtering in process_subtitles; --convert-subs converts to srt/ass/vtt/lrc via FFmpegSubtitlesConvertorPP.
- Audio extraction `-x/--extract-audio` appends an FFmpegExtractAudio postprocessor honoring --audio-format (best/aac/flac/mp3/m4a/opus/vorbis/wav) and --audio-quality; recoding via --recode-video maps to FFmpegVideoConvertorPP.
- Sidecar outputs: --write-info-json, --write-description, --write-annotations, --write-thumbnail/--write-all-thumbnails, and --xattrs (filesystem extended attributes) are all emitted from process_info/_write_info_json/_write_thumbnails.
- --download-archive records `extractor_key + ' ' + video_id` lines (via _make_archive_id) into a text file; in_download_archive checks it inside _match_entry to skip already-downloaded videos; writes go through locked_file for interprocess safety.
- Authentication options cover --username/--password (+ interactive getpass when password omitted), --twofactor, --netrc, --video-password, and Adobe Pass TV-provider login via --ap-mso/--ap-username/--ap-password with --ap-list-mso listing MSO_INFO providers.
- Self-update: `-U/--update` invokes update_self(), which only applies to the zip/frozen builds, verifies a pinned RSA signature over versions.json, then sha256-checks and replaces the running binary (via a .bat swap on Windows).
- Geo-bypass fakes an X-Forwarded-For header with a random IP drawn from per-extractor _GEO_COUNTRIES/_GEO_IP_BLOCKS or --geo-bypass-country/--geo-bypass-ip-block, with automatic retry on GeoRestrictedError inside InfoExtractor.extract.
- External downloaders: --external-downloader selects among aria2c, aria2p (JSON-RPC library), axel, curl, ffmpeg/avconv, httpie and wget; each ExternalFD subclass builds a CLI via _make_cmd and runs it with subprocess.Popen; --external-downloader-args injects extra args.
- Simulation/inspection modes: --simulate, --skip-download, --dump-json/--dump-single-json, --print-json, --get-url/--get-title/--get-id/--get-filename family, --list-formats and --list-thumbnails, plus --load-info-json to reprocess a saved info dict.
Data flow
Data moves in one direction through a well-defined pipeline. Inputs: CLI args + config files + batch files produce a URL list; the params dict is the single source of truth shared by all layers. Extractors turn a URL into an 'info dict' (a documented schema with _type discriminators video/url/url_transparent/playlist); process_ie_result resolves indirection recursively; process_video_result normalizes fields and selects formats; process_info emits sidecar files and invokes a protocol-matched FileDownloader that writes .part files and renames on completion; the postprocessor chain mutates/moves the final file; bookkeeping lands in the download archive, cookie jar, and JSON cache.
- URL input boundary: positional args plus lines from --batch-file (or stdin via '-') are merged into all_urls; bare URLs are decoded with the preferred encoding before reaching the extractor loop.
- Configuration input boundary: parseOpts merges /etc/youtube-dl.conf, user config (XDG_CONFIG_HOME/youtube-dl/config, ~/.config/youtube-dl.conf, %APPDATA% variants, ~/youtube-dl.conf), --config-location file, then argv — later entries override earlier ones; --ignore-config skips them.
- The info dict is the canonical intermediate representation: extractor/common.py's class docstring specifies mandatory fields (id, title, and url-or-formats) plus ~50 optional fields and the _type discriminator values (video, playlist, multi_video, url, url_transparent, compat_list).
- extract_info iterates self._ies calling ie.suitable(url) (compiled _VALID_URL regexes); the first match's extract() runs initialize() (geo-bypass, login) then _real_extract(url); results are stamped with extractor/webpage_url/extractor_key by add_default_extra_info.
- process_ie_result recursively resolves indirection: 'url'/'url_transparent' results re-enter extract_info (possibly pinned to an ie_key); 'playlist'/'multi_video'/'compat_list' results go to __process_playlist which slices entries, injects playlist_* extra_info, and reprocesses each entry via __process_iterable_entry.
- process_video_result normalizes a video info dict: forces string/numeric field types, sorts thumbnails, fills common fields, computes requested_subtitles, filters malformed formats, sorts formats with _sort_formats, and runs the parsed format selector (from build_format_selector) to produce formats_to_download or requested_formats for A+V merge.
- process_info is the output boundary: it computes the filename via prepare_filename, writes .description/.annotations.xml/subtitle/.info.json/thumbnail files first, then calls dl() which instantiates the downloader chosen by get_suitable_downloader and copies http_headers via _calc_headers.
- HttpFD.real_download streams to `filename + '.part'` opened just-in-time via sanitize_open, honoring resume (Range header + Content-Range validation), retry on 5xx/ECONNRESET/timeout, min/max filesize aborts, then try_rename to the final name and try_utime from Last-Modified.
- Fragmented downloads (HlsFD, DashSegmentsFD, F4mFD, IsmFD) share FragmentFD, which keeps a `.ytdl` JSON bookkeeping file per incomplete download (extractor/downloader state, current_fragment, fragment_count) enabling resume and consistency checks.
- Postprocessing output boundary: post_process chains info_dict['__postprocessors'] (e.g. FFmpegMergerPP for A+V merge, fixup PPs) before self._pps; each pp.run returns (files_to_delete, info); originals are deleted unless keepvideo.
- State owners: the cookie jar (YoutubeDLCookieJar, Netscape format, optionally file-backed via --cookies) is owned by YoutubeDL._setup_opener; extractor/player caches (e.g. YouTube 'sigfuncs') live under the Cache section/key namespace; the download archive is a plain text append log.
- Header/cookie hygiene is enforced at boundaries: FileDownloader.download documents stripping Cookie headers; _remove_cookie_header filters them from http_headers; header-supplied cookies are rescoped per-URL domain by _apply_header_cookies to prevent cross-domain leaks.
Behavior traces
Three core flows were traced end-to-end: (1) CLI startup from argv to download; (2) a single-video download from URL through extraction, format selection, HTTP download, fixup and postprocessing to the archive write; (3) the YouTube signature-deciphering path that downloads player JS and interprets it with the built-in JSInterpreter. A fourth trace covers self-update.
- Startup trace: `main()` -> `_real_main()` -> compat_register_utf8 + setproctitle -> parseOpts (reads /etc/youtube-dl.conf, user config, argv) -> header/batch-file handling -> option validation (netrc/password conflicts, numeric parsing of --limit-rate/--min-filesize, playlist bounds) -> postprocessor list assembly -> ydl_opts dict -> `with YoutubeDL(ydl_opts) as ydl` -> optional update_self / cache.remove -> ydl.download(all_urls) -> sys.exit(retcode).
- YoutubeDL init trace: __init__ stores params, creates Cache, loads header cookies, warns on deprecated params, optionally spawns bidiv/fribidi for --bidi-workaround, then _setup_opener builds cookie jar (loading --cookies file), proxy handler, HTTPS/HTTP handlers, redirect handler and disables file://; auto_init then prints debug header and registers all extractor classes plus configured postprocessors and progress hooks, finishing with register_socks_protocols.
- Download dispatch trace: download(url_list) -> extract_info -> ie.suitable(url) scan -> __extract_info (wrapped by __handle_extraction_exceptions) -> ie.extract(url) -> _real_extract -> process_ie_result -> dispatch on _type; for 'video' -> process_video_result -> format selection -> process_info -> dl() -> get_suitable_downloader(info) -> fd.download(name, info) -> real_download.
- Playlist trace: process_ie_result sees _type playlist -> __process_playlist computes entries (list slice, PagedList.getslice, or islice over iterables, honoring --playlist-items) -> optional reverse/shuffle -> per entry: _match_entry(incomplete) filter -> __process_iterable_entry -> process_ie_result (recursion back into the same dispatch).
- HTTP file download trace: FileDownloader.download checks overwrite/resume preconditions and applies randomized sleep_interval -> HttpFD.real_download -> establish_connection() (sanitized_Request with Range header, Content-Range validation, 416 fallback) -> download() loop: read block -> sanitize_open just-in-time -> stream.write -> slow_down rate limit -> _hook_progress -> loop until data_len -> try_rename .part -> try_utime -> 'finished' hook; errors route through RetryDownload/NextFragment/SucceedDownload control-flow exceptions.
- YouTube signature trace: _decrypt_signature -> _cached(_extract_signature_function) -> filesystem cache 'sigfuncs' (keyed by player_url + signature-shape id) -> on miss _load_player downloads player JS -> _parse_sig_js regex-locates the sig function -> JSInterpreter.extract_function_from_code produces a Python callable -> the result is reduced to a char-permutation cache_spec and cached; _decrypt_nsig does the analogous flow for the throttling 'n' parameter.
- Post-download trace: on dl() success, process_info applies fixup policy (FFmpegFixupStretchedPP for stretched_ratio, FFmpegFixupM4aPP for m4a_dash, FFmpegFixupM3u8PP for native HLS AAC) -> post_process runs __postprocessors then self._pps -> each pp.run returns files_to_delete which are removed unless keepvideo -> record_download_archive appends `extractor id` -> MaxDownloadsReached checked.
- Self-update trace: update_self refuses non-zip/non-frozen installs -> GET https://yt-dl.org/update/LATEST_VERSION -> GET versions.json -> rsa_verify(json, signature, pinned UPDATES_RSA_KEY) -> sha256 the downloaded exe/bin blob -> write alongside the running binary (Windows: write .new + a youtube-dl-updater.bat that moves it after exit).
- Error/abort trace: extraction errors funnel through __handle_extraction_exceptions (GeoRestrictedError gets a VPN hint, ExtractorError is reported, unexpected exceptions re-raise unless --ignore-errors); DownloadError propagates to main() -> sys.exit(1); MaxDownloadsReached exits 101; KeyboardInterrupt exits with an interrupted message.
Testing signals
Testing is unittest-based, executed by nosetests (pynose on Python >=3.9). Coverage splits into offline unit tests (utils, aes, jsinterp, swfinterp, compat, options, YoutubeDL internals, URL matching) and a large live-download suite generated dynamically from each extractor's embedded _TEST/_TESTS dictionaries. Network-dependent suites are explicitly excluded from 'offlinetest'/'core' runs. CI on GitHub Actions spans CPython 2.6-3.12, PyPy and Jython on Ubuntu and Windows, plus a flake8 lint job; the download test-set is continue-on-error.
- Test runner is nosetests: Makefile `test` runs `nosetests --verbose test` then flake8; devscripts/run_tests.sh selects core vs download sets via -I regex on test module names; tox.ini also drives nosetests with the same exclusion list.
- Extractor test cases are embedded in extractor classes as _TEST/_TESTS dicts; InfoExtractor.get_testcases yields them, test/helper.py's gettestcases aggregates across gen_extractors(), and test_download.py dynamically synthesizes one TestDownload.test_<Name> method per case.
- Download tests hit the real sites: they call ydl.extract_info on the case URL, retry up to 3 times on network-flavored errors (URLError, socket.timeout, HTTP 503), assert expected info_dict fields via expect_info_dict, verify the downloaded file exists and meets file_minsize, and optionally check an md5.
- test/parameters.json supplies base params for tests (test=true limiting downloads to _TEST_FILE_SIZE, writeinfojson=true, socket_timeout=20, fixup=never); test/helper.py merges an optional untracked local_parameters.json for credentials.
- Offline-only coverage is defined by exclusion: Makefile offlinetest and tox defaults exclude test_age_restriction, test_download, test_iqiyi_sdk_interpreter, test_socks, test_subtitles, test_write_annotations, test_youtube_lists and test_youtube_signature — i.e. anything needing network or live sites.
- Unit-level suites cover utils (test_utils.py), JS interpreter (test_jsinterp.py), AES (test_aes.py against testdata/), SWF interpreter (test_swfinterp.py against test/swftests/), URL matching across extractors (test_all_urls.py), YoutubeDL internals (test_YoutubeDL.py), options parsing (test_options.py), netrc, cache, traversal and compat.
- Test doubles exist in helper.py: FakeYDL (a YoutubeDL with auto_init=False that records download() calls and raises on trouble), FakeLogger, and expect_warnings/expect_info_dict assertion helpers that also fail when the extractor produces checkable fields missing from the test definition.
- CI (.github/workflows/ci.yml) builds a matrix over CPython 2.6 through 3.12 plus pypy and jython, ubuntu-22.04 primary with windows-2022 and jython extras; it includes custom steps to build Python 2.6/2.7/3.next and install nose/pynose per version; download and jython legs are continue-on-error.
- Lint is flake8 with repo-tuned ignores (E402,E501,E731,E741,W503) and excludes generated/excluded files; it runs as a separate CI job on Python 3.9 and as `make codetest`.
- Notable gaps: there is no coverage tooling enabled by default (commented-out --with-coverage), most of the ~800 extractors rely solely on their embedded live tests (marked skip/_WORKING when broken), and legacy interpreters (Python 2.6, Jython) mean modern test runners can't even execute parts of the matrix. (inferred)
Risk map
The highest-risk areas are shell-outs (postprocessor --exec with shell=True, external downloader and ffmpeg command construction), credential handling (cookies, netrc, passwords), the self-update path that replaces the running binary, and the eval-adjacent JS/SWF interpreters. The codebase also carries heavy legacy weight: Python 2.6 compat shims, dead modules (swfinterp is only referenced by tests), and fragile regex-based scraping across ~800 site extractors that break silently when sites change.
- ExecAfterDownloadPP.run executes the user's --exec command with subprocess.call(shell=True); the {} placeholder is shlex-quoted, but the rest of the command string is raw shell — a deliberate but real command-execution surface.
- External downloaders and ffmpeg are spawned via subprocess.Popen with argv built from info_dict fields (url, http_headers, cookies); cookies are serialized into temp files or -cookies/-headers args, so a malicious extractor result could influence command lines, though lists (not shell strings) are used.
- Self-update replaces the running executable with a network download; it is gated by an RSA signature over versions.json plus a per-file sha256 check, but trusts a hard-coded RSA key and yt-dl.org availability; the Windows path writes and spawns a .bat file that swaps the exe.
- Cookie leak mitigations indicate a history of security issues: _remove_cookie_header strips Cookie from http_headers, _load_cookies scopes header-supplied cookies to the target domain (referencing yt-dlp advisory GHSA-v8mc-9377-rwjj), and YoutubeDLRedirectHandler drops Cookie on redirect; the file:// scheme is explicitly disabled citing issue #8227.
- _UnsafeExtensionError enforces an extension whitelist on output files to prevent path traversal/arbitrary file overwrite (referencing yt-dlp advisory GHSA-79w7-vh3h-8g4j); --no-check-extensions sets the lenient flag to bypass it.
- Aria2pFD connects to an aria2 daemon at http://localhost:6800 with an empty RPC secret and trusts the aria2p library being importable — local RPC without authentication.
- JSInterpreter is a partial JS engine that executes extracted player code semantics; _extract_sig_fn even splices a regex-extracted 'useful values' snippet into the interpreted function — a YouTube player change silently breaks decryption, and malformed code raises JSInterpreter.Exception.
- swfinterp.py (834 lines, SWFInterpreter) is effectively dead production code — the only importer in the tree is test/test_swfinterp.py; PhantomJSwrapper in extractor/openload.py survives only as a dependency of pornhub.py and a debug-header version probe.
- Extractors are regex-fragile by design: _search_regex raises RegexNotFoundError on missing patterns and CONTRIBUTING.md explicitly warns that extractor breakage on site changes is expected; broken extractors are flagged via the _WORKING class attribute rather than removed.
- --call-home/--verbose debug path sends requests to https://yt-dl.org/ip and /latest/version, disclosing the user's public IP to the project's own server.
- Python 2.6 support is still load-bearing: compat.py shims, _workaround_optparse_bug9161, py2exe paths, and CI builds Python 2.6/2.7 from source — code paths that cannot be exercised on modern interpreters and constrain idioms (no f-strings, unicode_literals everywhere). (inferred)
- Passwords entered interactively (compat_getpass) or passed via -p/--password appear in argv and config files; _hide_login_info only scrubs them from verbose debug echoes, not from process listings or config storage. (inferred)
Extension points
The codebase is explicitly designed around plugin points. Primary: site extractors subclass InfoExtractor, define _VALID_URL + _real_extract, embed _TEST dicts, and register via an import line in extractor/extractors.py (plus optional lazy_extractors generation). Secondary: downloaders plug into PROTOCOL_MAP or the external-downloader _BY_NAME registry, postprocessors resolve by the '<key>PP' naming convention, and embedders hook progress_hooks, match_filter, custom logger, and the postprocessors param list.
- New site support follows a documented recipe: create youtube_dl/extractor/<name>.py with an InfoExtractor subclass (class name ending in IE), define _VALID_URL regex and _real_extract, add a _TEST dict, and add one import line to extractor/extractors.py.
- Extractor ordering is semantic: gen_extractor_classes returns _ALL_CLASSES in import order and the first suitable() match wins; GenericIE is appended last as the catch-all, so new extractors must be added before it implicitly by import placement.
- Lazy loading is a compile-time extension artifact: devscripts/make_lazy_extractors.py generates youtube_dl/extractor/lazy_extractors.py from _ALL_CLASSES; this constrains extractor authors — overridden suitable() must import its own dependencies and avoid _VALID_URL references (documented in InfoExtractor's docstring).
- SearchInfoExtractor is the extension base for 'keyword' providers (e.g. ytsearch): subclasses set _SEARCH_KEY and implement _real_extract over a query string, producing a playlist of url_results.
- url_result()/playlist_result() are the delegation mechanism: an extractor can emit _type 'url' entries with an ie_key to hand a URL off to another extractor, enabling composite extractors without subclassing.
- New downloaders subclass FileDownloader and implement real_download(filename, info_dict); protocol-based selection goes through PROTOCOL_MAP in downloader/__init__.py, while external-executable downloaders subclass ExternalFD (_make_cmd) and self-register in _BY_NAME keyed by executable basename.
- New postprocessors subclass PostProcessor with a run(info)->(files_to_delete, info) contract; registration is by naming convention — a param dict {'key': 'X'} resolves to class XPP via get_postprocessor, and the CLI assembles the list in __init__.
- Library/embedding hooks: YoutubeDL accepts progress_hooks (status dicts with downloading/error/finished and byte/eta/speed fields), a logger object, match_filter callable, extract_flat mode, and a postprocessors list — the documented embedding API.
- Per-request behaviors can be injected through info_dict fields read downstream: http_headers, downloader_options (http_chunk_size), fragments, protocol, stretched_ratio, __postprocessors and __files_to_merge — extractors/downstream code cooperate through these keys.
- Geo-bypass is itself an extension surface: extractors declare _GEO_COUNTRIES/_GEO_IP_BLOCKS/_GEO_BYPASS and _initialize_geo_bypass can be invoked mid-extraction with a context dict when restriction info is only known after the first response.
Contributor handoff
For an agent working in this repo: read CONTRIBUTING.md's developer instructions and extractor/common.py's giant docstring first; the info-dict contract and the 'first suitable() wins' ordering are the two invariants that matter most. Python 2.6-compatible style (unicode_literals, compat_* aliases, no f-strings, %-formatting) is mandatory and enforced by CI. Tests are nosetests; extractor changes get a live _TEST; run flake8 with the repo's setup.cfg ignores. Generated files (lazy_extractors.py, README.md, completions, man page) are produced by Makefile/devscripts — edit sources, not artifacts.
- Read first: youtube_dl/extractor/common.py's InfoExtractor docstring (the entire info-dict schema), CONTRIBUTING.md 'DEVELOPER INSTRUCTIONS' + 'youtube-dl coding conventions', youtube_dl/YoutubeDL.py's options docstring, and youtube_dl/__init__.py _real_main for how params map to CLI flags.
- Adding an extractor: new file in youtube_dl/extractor/, class name must end in 'IE', _VALID_URL with a named (?P<id>...) group is conventional, import the class in extractors.py (order = precedence), include a _TEST dict with a real URL; commit style is '[name] Add new extractor'.
- Mandatory info-dict fields are id and title plus either url or formats; everything else must be extracted defensively (helpers: _search_regex with default=, try_get, traverse_obj, int_or_none/float_or_none) because extractors must tolerate missing optional data.
- Run tests with `python test/test_download.py TestDownload.test_<Name>` for a single extractor; `make offlinetest` (or nosetests with the exclusion list) for network-free checks; full `make test` requires live network access and will fail on sites that changed since the pinned commit.
- Lint with `flake8 .` honoring setup.cfg (ignores E402/E501/E731/E741/W503, excludes generated files); style is %-formatting, `from __future__ import unicode_literals` at top of every file, compat_* imports from youtube_dl/compat.py instead of py3-only stdlib paths.
- Lazy-extractor constraint is a known trap: if you override suitable() or reference _VALID_URL oddly, devscripts/make_lazy_extractors.py output can break — keep suitable() self-importing and prefer plain _VALID_URL regexes; regenerate via `make lazy-extractors` or `python setup.py build_lazy_extractors`.
- GenericIE must remain the last resort: _ALL_CLASSES appends it explicitly and many extractors emit url_result(ie=GenericIE.ie_key()) delegates; do not reorder extractors casually — the first regex match wins.
- Generated artifacts to not hand-edit: youtube_dl/extractor/lazy_extractors.py (devscripts/make_lazy_extractors.py), README.md/CONTRIBUTING.md (devscripts/make_readme.py/make_contributing.py), docs/supportedsites.md (make_supportedsites.py), completions and man page (Makefile targets).
- Safe edit boundaries: adding a new extractor file + extractors.py import is low-blast-radius; editing extractor/common.py, utils.py, YoutubeDL.py or downloader/common.py affects every site — keep changes additive, preserve py2.6 compatibility, and remember `_real_main` mutates the global std_headers dict and params dicts are shared references. (inferred)
- Debugging support built in: --verbose prints the debug header (version, git HEAD, Python, exe versions, proxy map), --write-pages/--dump-pages dump raw server responses, --print-traffic logs HTTP, and --youtube-print-sig-code prints the extracted signature function — use these before touching extractor code.
Languages
| Language | Files | Share of source |
|---|---|---|
| Python | 902 | 99% |
| Markdown | 15 | docs/config |
| Shell | 5 | 1% |
| JSON | 2 | docs/config |
| YAML | 2 | docs/config |
Start reading here
Probable source entrypoint: youtube_dl/__init__.py — inferred from the directory tree and manifest, not the upstream docs.
Source roots
youtube_dl/ (844 files) devscripts/ (34 files)
Source layout — youtube_dl/
| Path | Files |
|---|---|
youtube_dl/downloader/ | 12 |
youtube_dl/extractor/ | 291 |
youtube_dl/postprocessor/ | 7 |
youtube_dl/__init__.py | 1 |
youtube_dl/aes.py | 1 |
youtube_dl/cache.py | 1 |
youtube_dl/compat.py | 1 |
youtube_dl/options.py | 1 |
youtube_dl/traversal.py | 1 |
youtube_dl/update.py | 1 |
youtube_dl/version.py | 1 |
Repository structure
| Top-level path | Files |
|---|---|
youtube_dl/ | 844 |
test/ | 70 |
devscripts/ | 34 |
.github/ | 14 |
docs/ | 6 |
Docs found: CONTRIBUTING.mdREADME.mddocs/supportedsites.md
CI & config entrypoints
README.md— Repository readme.github/workflows/ci.yml— GitHub Actions workflow
Structural health signals
Average score 6.3/10 across 96 analyzed source files; 55 hotspots surfaced.
youtube_dl/extractor/adobepass.py— 1572 lines, score 5 (Large file, Deep nesting)youtube_dl/extractor/facebook.py— 709 lines, score 5.2 (Large file, Deep nesting)youtube_dl/postprocessor/ffmpeg.py— 653 lines, score 5.2 (Large file, Deep nesting)
These are deterministic review leads — size, churn, and structural signals — not proof of a bug in this project.
Recent history
956b8c5855912025-11-26 — [YouTube] Bug-fix for `c1f5c3274a`d5f561166b9d2025-11-26 — [core] Re-work format_note display in format list with abbreviated codec named0283f5385ac2025-11-20 — [YouTube] Revert forcing player JS by default6315f4b1dfaa2025-11-19 — [utils] Support additional codecs and dynamic_rangeaeb1254fcf892025-11-04 — [YouTube] Fix playlist thumbnail extraction