CodeVetter · Repo Unpack · Extension points
ytdl-org/youtube-dl
The codebase is explicitly designed around plugin points. Primary: site extractors subclass InfoExtractor, define _VALID_URL + _real_extract, embed _TEST dicts, and register via an import line in extractor/extractors.py (plus optional lazy_extractors generation). Secondary: downloaders plug into PROTOCOL_MAP or the external-downloader _BY_NAME registry, postprocessors resolve by the '<key>PP' naming convention, and embedders hook progress_hooks, match_filter, custom logger, and the postprocessors param list.
- New site support follows a documented recipe: create youtube_dl/extractor/<name>.py with an InfoExtractor subclass (class name ending in IE), define _VALID_URL regex and _real_extract, add a _TEST dict, and add one import line to extractor/extractors.py.
- Extractor ordering is semantic: gen_extractor_classes returns _ALL_CLASSES in import order and the first suitable() match wins; GenericIE is appended last as the catch-all, so new extractors must be added before it implicitly by import placement.
- Lazy loading is a compile-time extension artifact: devscripts/make_lazy_extractors.py generates youtube_dl/extractor/lazy_extractors.py from _ALL_CLASSES; this constrains extractor authors — overridden suitable() must import its own dependencies and avoid _VALID_URL references (documented in InfoExtractor's docstring).
- SearchInfoExtractor is the extension base for 'keyword' providers (e.g. ytsearch): subclasses set _SEARCH_KEY and implement _real_extract over a query string, producing a playlist of url_results.
- url_result()/playlist_result() are the delegation mechanism: an extractor can emit _type 'url' entries with an ie_key to hand a URL off to another extractor, enabling composite extractors without subclassing.
- New downloaders subclass FileDownloader and implement real_download(filename, info_dict); protocol-based selection goes through PROTOCOL_MAP in downloader/__init__.py, while external-executable downloaders subclass ExternalFD (_make_cmd) and self-register in _BY_NAME keyed by executable basename.
- New postprocessors subclass PostProcessor with a run(info)->(files_to_delete, info) contract; registration is by naming convention — a param dict {'key': 'X'} resolves to class XPP via get_postprocessor, and the CLI assembles the list in __init__.
- Library/embedding hooks: YoutubeDL accepts progress_hooks (status dicts with downloading/error/finished and byte/eta/speed fields), a logger object, match_filter callable, extract_flat mode, and a postprocessors list — the documented embedding API.
- Per-request behaviors can be injected through info_dict fields read downstream: http_headers, downloader_options (http_chunk_size), fragments, protocol, stretched_ratio, __postprocessors and __files_to_merge — extractors/downstream code cooperate through these keys.
- Geo-bypass is itself an extension surface: extractors declare _GEO_COUNTRIES/_GEO_IP_BLOCKS/_GEO_BYPASS and _initialize_geo_bypass can be invoked mid-extraction with a context dict when restriction info is only known after the first response.