CodeVetter · Repo Unpack · Data flow
microsoft/markitdown
Data enters through convert()/convert_local()/convert_stream()/convert_uri()/convert_response(), is normalized to a seekable BinaryIO plus a list of StreamInfo guesses, then fanned out to priority-sorted converters. Type detection is layered: caller hints and headers produce a base StreamInfo, mimetypes fills gaps, then magika identifies content from bytes and charset-normalizer guesses text encoding; incompatible guesses produce two candidate StreamInfos tried in order. The winning converter returns a DocumentConverterResult whose markdown is normalized (trailing whitespace stripped, 3+ newlines collapsed) before returning.
- convert() is a type-based dispatcher: str with scheme http/https/file/data routes to convert_uri, other strings and Path to convert_local, requests.Response to convert_response, and readable non-TextIO objects to convert_stream; anything else raises TypeError.
- convert_local() seeds StreamInfo with local_path, os.path.splitext extension, and basename, merges caller stream_info (plus deprecated file_extension/url kwargs), opens the file 'rb', guesses stream info, then calls _convert.
- convert_stream() buffers non-seekable streams fully into io.BytesIO (4 KiB chunks) before guessing — the entire file is resident in memory from that point on.
- convert_uri() branches by scheme: file: URIs are resolved by file_uri_to_path (rejecting non-localhost netlocs and UNC/device paths), data: URIs are decoded by parse_data_uri (base64 or percent), and http(s): is fetched with self._requests_session.get(uri, stream=True) then converted via convert_response.
- convert_response() builds the base StreamInfo from Content-Type (mimetype + charset), Content-Disposition filename (parsed via email.message.Message, RFC 2231-aware), or the URL path extension, then buffers the whole body via response.iter_content(chunk_size=512) into BytesIO.
- _get_stream_info_guesses() enhances the base guess via the mimetypes module, then calls magika.identify_stream; for text predictions it samples up to 64 KiB (completing a split trailing UTF-8 char via _read_charset_sample) and runs charset_normalizer.from_bytes().best() for charset detection.
- When magika's prediction is incompatible with caller-supplied mimetype/extension/charset, both the enhanced base guess and the magika guess are appended as separate StreamInfo candidates, each tried in sequence against all converters in _convert.
- _convert() iterates 'stream_info_guesses + [StreamInfo()]' outer loop and priority-sorted converters inner loop, asserting the stream position is unchanged by accepts(); on convert() success it normalizes text_content (rstrip each line, collapse >=3 newlines to 2) and returns immediately.
- Per-call kwargs are enriched in _convert with global llm_client/llm_model/llm_prompt/style_map/exiftool_path, a private _parent_converters list for nested processing, and legacy file_extension/url keys derived from the current StreamInfo guess.
- Converters signal outcomes via exceptions: convert() failures are collected as FailedConversionAttempt and, if no converter succeeds, raised as FileConversionException (with per-converter trace summary); if no converter even accepted, UnsupportedFormatException is raised.
- Office/HTML converters converge on HTML as an intermediate: DocxConverter (mammoth), EpubConverter (spine files), PptxConverter and RSS/Wikipedia converters produce HTML that is finalized by HtmlConverter -> _CustomMarkdownify, which also strips javascript links and truncates data URIs unless keep_data_uris is set.
- In the MCP server, each convert_to_markdown call constructs a fresh MarkItDown instance (plugins gated by env MARKITDOWN_ENABLE_PLUGINS) and calls convert_uri; every exception type is mapped to a ToolError that deliberately omits internal detail (e.g., FileConversionException becomes 'File conversion failed.', OSError reports only os.strerror).