encode/httpx
github.com/encode/httpx ↗ · 113 files · 968 KB
HTTPX - A next-generation HTTP client for Python.
Overview
HTTPX is a Python HTTP client for application developers, with synchronous and asynchronous APIs plus an optional terminal client. It combines reusable connection pools, authentication, redirects, cookies, uploads, and response decoding behind a transport interface that can also call WSGI and ASGI applications directly. This brief reflects source inspection at commit b5addb64f0161ff6bfe94c124ef76f6a1fba5254; tests and CI were inspected, not executed.
System map
The package separates public APIs, client policy, HTTP models, and transport execution. Client configuration and cookies live in Python objects, while network connection pools belong to httpcore. Packaging produces a Python distribution and a MkDocs documentation site. The inspected workflows define verification and publication, but their outcomes were not observed.
- The distribution requires Python >=3.9, uses Hatchling, and declares certifi, httpcore==1.*, anyio, and idna as runtime dependencies.
- httpx/__init__.py assembles the public API from internal modules; the console entrypoint is httpx:main, with an import fallback that explains how to install missing CLI extras and exits 1.
- _api.request() creates a context-managed Client for one call; BaseClient holds shared configuration, while Client and AsyncClient implement separate dispatch pipelines.
- Request, Response, Headers, and Cookies are HTTP model objects; URL delegates parsing to the repository's RFC3986-oriented normalization implementation rather than relying exclusively on stdlib parsing.
- HTTPTransport and AsyncHTTPTransport select httpcore connection pools or HTTP/SOCKS proxy pools and translate HTTPX requests and responses across that dependency boundary.
- Alternate runtime boundaries include synchronous WSGI calls, asynchronous ASGI calls, and callable-backed MockTransport dispatch; each returns the same Response model.
- Session state is client-owned: BaseClient initializes Cookies, received responses update that jar, and Cookies wraps stdlib CookieJar. CLI downloads provide an explicit filesystem output boundary.
- CI invokes install, check, build, test, and coverage scripts on Ubuntu/Python 3.9–3.13; tag publication builds distributions and docs, then uploads to PyPI and deploys MkDocs after checking tag/version agreement.
Feature catalog
The user-facing surface consists of library APIs and a single HTTP request CLI. Library callers can choose convenience helpers or reusable clients, customize request construction, and consume buffered or streaming responses. Protocol, proxy, authentication, and in-process application integrations share the client model. The following eight groups describe capabilities supported by the inspected implementation.
- The top-level API exports get, options, head, post, put, patch, delete, request, and stream; generic request accepts body content, form data, files, JSON, query parameters, headers, cookies, and authentication.
- Reusable clients combine default headers, query parameters, and cookies with request values, append relative paths to base_url, retain cookies, and offer synchronous or awaited request methods.
- Response offers decoded bytes, text, JSON, links, cookies, HTTP metadata, status predicates, and explicit raise_for_status(); the latter raises for any non-success response and requires an associated request.
- Uploads support raw strings/bytes, sync or async byte iterables, URL-encoded forms, compact UTF-8 JSON, and multipart files with optional filename, content type, and per-part headers.
- Streaming responses expose raw bytes, decompressed bytes, incremental text and lines, plus corresponding async methods; gzip/deflate are built in and Brotli/Zstandard decoders depend on installed extras.
- Authentication supports Basic, Digest, explicit NetRCAuth, callable request transforms, and custom Auth flows. Redirect following is optional, with history and next_request available for automatic or manual navigation.
- Transport configuration exposes HTTP/1 and opt-in HTTP/2, HTTP/HTTPS/SOCKS proxies, TLS contexts, four timeout categories, connection limits, socket options, local addresses, Unix sockets, and connection-error retries; ASGI/WSGI transports call local apps.
- The CLI accepts URL, method, body/query/header/cookie/auth options, proxy, timeout, redirects, TLS verification toggle, HTTP/2, download destination, and verbose tracing; it renders text/JSON or a binary-size summary and exits according to response success.
Data flow
Request construction turns caller inputs into normalized URL, header, cookie, extension, and stream objects. Authentication and redirect loops can produce additional exchanges before the final response reaches the caller. Transports preserve raw protocol data and delegate network execution, while response models decode and optionally buffer bodies. State persists through the client cookie jar and connection pools; CLI downloads explicitly write decoded content.
- build_request() merges URL, headers, cookies, and query parameters, inserts a timeout extension unless already supplied, then constructs Request with the merged values and body inputs.
- URL construction validates component types and invokes urlparse(); normalization lowercases scheme, encodes hosts, normalizes ports and paths, and preserves the distinction between absent and empty query strings.
- Request uppercases the method, copies extensions, applies cookie headers, and invokes encode_request(); body selection prefers explicit content, then files, nonempty form data, and JSON, with a deprecated raw-data compatibility branch.
- Body encoders return headers plus a byte stream: known lengths use Content-Length, unknown iterable lengths use chunked transfer, and multipart length becomes unknown when any file length cannot be determined.
- send() resolves timeout/auth policy; each exchange runs request hooks, dispatches through the URL-selected transport, runs response hooks, and either returns, follows a redirect, or exposes next_request.
- HTTPTransport converts raw scheme, host, port, request target, headers, stream, and extensions into httpcore.Request; pool output becomes Response with wrapped response stream and preserved extensions.
- The client attaches the originating request, binds elapsed-time tracking to stream closure, extracts Set-Cookie into its jar, and supplies default text encoding; subsequent requests obtain Cookie headers through CookieJar adapters.
- iter_raw() counts received bytes; iter_bytes() applies content decoding and chunking; read() joins decoded chunks into cached content. CLI download_response() writes iter_bytes() chunks while reporting raw-byte progress.
Behavior traces
These traces follow implementation call order rather than executing scenarios. They cover client startup, an ordinary network request, redirects, authentication retries, cookie persistence, asynchronous application dispatch, and CLI downloads. Cleanup paths determine when connections are released and elapsed time becomes available. Each trace retains material branches such as disabled redirects, streaming mode, and application exceptions.
- Startup: Client.__init__() calls BaseClient.__init__(), checks the h2 dependency when HTTP/2 is enabled, computes proxy mappings, initializes the transport, constructs and sorts mounts; __enter__() changes state to OPENED and enters configured transports.
- Ordinary GET: _api.get() → request() → temporary Client → Client.request() → build_request() → send() → auth loop → redirect loop → _send_single_request() → HTTPTransport.handle_request() → httpcore pool → Response.read().
- Redirect: after response hooks, _send_handling_redirects() invokes _build_redirect_request(), which computes method, URL, headers, stream, and jar cookies. Following consumes the previous body and loops; otherwise next_request is set, and excessive history raises TooManyRedirects.
- Digest challenge: DigestAuth.auth_flow() yields the initial request, checks a 401 for a Digest challenge, parses and caches it, builds Authorization, applies challenge-response cookies, and yields a retry; the client reads intermediate responses and adds them to auth history.
- Cookie persistence: response dispatch calls extract_cookies(); a redirect copies the client jar and removes the previous Cookie header before Request rebuilds it. The inspected login/logout test follows a Set-Cookie redirect, reuses the session, then expires it.
- Async ASGI request: AsyncClient.send() → async auth/redirect loops → _send_single_request() → ASGITransport.handle_async_request(); scope and receive/send callbacks bridge the request to await app(), then collected body parts become an ASGIResponseStream and aread() buffers the result unless streaming.
- CLI download: Click opens the destination in wb mode; main() infers GET or POST when method is omitted, creates Client and its streaming context, installs a trace callback, calls download_response(), and exits 0 for a successful response or 1 otherwise.
- Cleanup: Client.stream()/AsyncClient.stream() close responses in finally; complete raw iteration also closes them. Bound streams record elapsed time and close the underlying stream, while client close/aclose closes the default and mounted transports and marks the client CLOSED.
Testing signals
Tests and CI were inspected, not executed; test existence and configured thresholds are not passing results. The suite uses pytest, AnyIO-marked async tests, mock transports, and local Uvicorn servers. Inspected cases exercise important policy, streaming, CLI, and timeout behavior. Coverage exclusions and filtered conformance fixtures limit what the advertised coverage goal establishes, and this investigation is bounded rather than exhaustive.
- pytest is pinned in tooling requirements; pyproject configures warnings as errors and a network marker, while async tests use pytest.mark.anyio and test helpers accommodate asyncio and Trio.
- The inspected Test Suite workflow runs on Ubuntu with Python 3.9–3.13 and calls install, check, build, test, and coverage scripts. No execution result was obtained for this commit.
- scripts/test runs pytest under coverage and adds check/coverage steps outside GitHub Actions; coverage includes httpx and tests and enforces fail-under=100. Certificate environment branches and several CLI TLS/HTTP2 trace branches carry coverage exclusions.
- MockTransport redirect tests assert cross-domain Authorization removal, same-domain retention and HTTP-to-HTTPS retention, 308 body preservation, 303 body removal, generator replay failure, and login/logout cookie persistence.
- Hook tests check sync/async request and response events, closure after response-hook exceptions, and repeated hook invocation on redirects; auth tests explicitly expect unsupported Digest auth-int and consumed async-body replay failures.
- CLI tests use CliRunner with a local server to assert text/JSON/binary output, redirects, POST JSON, verbose authorization output, downloads in an isolated filesystem, and unsupported-protocol exit behavior.
- Timeout tests exercise read, write, connection and pool failures; the connection case is network-marked and targets 10.255.255.1, while the pool case holds a stream with max_connections=1 before issuing a second request.
- Decoder tests assert decompression, stacked encodings, invalid-data errors, streaming text and split-line behavior; WHATWG URL tests omit fixture failure cases and return early for two space-encoding cases, so that adapter is not full WHATWG conformance coverage.
Risk map
The following are source observations and one explicitly labeled resource-risk hypothesis, not confirmed vulnerabilities. Sensitive boundaries include outbound routing, certificate trust, credentials, redirects, stream ownership, and filesystem downloads. Several restrictions are intentional compatibility or API limits with corresponding tests. Documentation drift and excluded branches also matter when an agent assesses behavior.
- Verbose CLI tracing prints raw request headers, including reversible Basic Authorization; help text warns about this and test_auth asserts the exposed header. Headers.__repr__ masking therefore does not imply CLI output redaction.
- trust_env defaults true; environment/system proxy discovery can alter routing, and SSL_CERT_FILE/SSL_CERT_DIR can replace default certificate locations. Custom transports disable automatic environment-proxy discovery, while explicit proxy configuration remains supported.
- verify=False explicitly disables hostname checking and certificate verification. NetRCAuth loads a caller-selected or default netrc file at construction, and deprecated cert arguments load certificate chains from external paths.
- Redirect authorization policy depends on scheme, host and effective port; Authorization survives only same-origin redirects or the defined same-host HTTP:80-to-HTTPS:443 upgrade exception. Redirect requests also discard the original Cookie header and rebuild cookies from the jar.
- Manual stream=True callers must arrange response closure, and generator request bodies can raise StreamConsumed when redirected or retried for Digest authentication; inspected tests encode both replay restrictions.
- Resource-risk hypothesis: large or highly compressed responses can demand substantial memory because read()/aread() join decoded bodies, ASGITransport collects response parts, and its stream joins them. Connection Limits do not impose a body-size bound in these inspected paths. (inferred)
- Digest authentication retains challenge and nonce-counter state and deliberately rejects auth-int-only challenges with NotImplementedError; a test expects that failure, so auth-int must not be described as supported.
- The WSGITransport docstring still recommends Client(app=app), but the inspected Client constructor accepts transport and has no app argument. This concrete documentation drift makes constructor source a necessary check before copying examples.
Extension points
Extension is primarily explicit object injection rather than automatic plugin discovery. Transports replace execution, mounts route execution, and Auth generators implement multi-exchange credential behavior. Hooks and extensions add observation and transport-specific metadata without changing the simplified Request/Response model. Internal decoder registries and the CLI option table provide additional edit locations, with public exports checked separately.
- Custom transports subclass BaseTransport.handle_request() or AsyncBaseTransport.handle_async_request(), return Response, and implement close/aclose when they own resources; Client._init_transport() returns an injected transport directly.
- mounts maps URL patterns to transports or None; explicit entries update proxy-derived mounts, sorting uses URLPattern priority, and the first match selects its transport or falls back to the default for None.
- Auth.auth_flow() is a Request/Response generator protocol supporting multiple exchanges; requires_request_body and requires_response_body trigger buffering, and I/O-bearing schemes can override sync_auth_flow()/async_auth_flow().
- Client event_hooks registers lists under request and response, can be changed through the property, and may mutate models. AsyncClient awaits hooks; response hooks needing content must explicitly read or await aread().
- Request/Response extensions are an untyped transport-specific channel; the default transport forwards them, documented request keys include timeout and trace, and response network_stream permits lower-level connection access.
- MockTransport injects a Request-to-Response handler, reads the request body before dispatch, and allows async clients to use either sync or coroutine handlers; sync clients reject an async handler result.
- SUPPORTED_DECODERS maps content-encoding names to ContentDecoder classes and drops unavailable optional codecs; Response selects and combines those decoders. default_encoding separately accepts a callable over buffered bytes for text detection.
- CLI options are defined by Click decorators around main(), while package-level symbols are curated in __all__; test_all_imports_are_exported checks that public imports match the sorted export list.
Contributor handoff
Tests and CI were inspected, not executed, and this brief is a bounded source investigation rather than an exhaustive audit. Start at the public entrypoint and follow the specific policy, model, and transport boundary involved in the task. Preserve synchronous/asynchronous parity, stream ownership, request-merging rules, and explicit disabling semantics. Verification commands below describe the repository workflow for a separately authorized execution task; publication is a distinct external action.
- Read pyproject.toml, __init__.py and _api.py before altering entrypoints: optional extras determine CLI/protocol availability, public exports are explicit, and top-level request helpers own temporary client lifetime.
- Client dispatch fixes need parity review in both send/auth/redirect pipelines; shared URL/header/cookie redirect transformations live on BaseClient, while sync and async hooks and stream operations differ.
- Preserve build_request() versus send() behavior: construction merges client defaults, but manually constructed requests bypass that merge. send() still sets missing timeout metadata and resolves authentication policy.
- USE_CLIENT_DEFAULT distinguishes omission from explicit None for auth and timeout; relative request URLs append to a slash-normalized base path rather than replacing it. These semantics are deliberate and documented in implementation.
- Treat stream cleanup and replay as acceptance criteria: stream contexts close in finally, raw iteration rejects closed/consumed streams, and generator redirects have a regression case expecting StreamConsumed.
- For separately authorized execution, the documented focused command is scripts/test -- tests/test_multipart.py; arguments reach pytest, but local execution also runs global checks and a 100-percent coverage gate. Test servers require ports 8000 and 8001.
- Repository checks enforce changelog/version agreement, Ruff formatting and lint, and strict mypy over httpx and tests; QueryParams is immutable, and per-request client cookies emit a deprecation warning.
- Build and release are separate boundaries: scripts/build creates distributions and docs; scripts/publish uploads distributions and force-deploys docs, and the Publish workflow responds to pushed tags. None of these commands was executed during this investigation.
Languages
| Language | Files | Share of source |
|---|---|---|
| Python | 60 | 98% |
| Markdown | 29 | docs/config |
| YAML | 6 | docs/config |
| CSS | 1 | docs/config |
| HTML | 1 | docs/config |
| JSON | 1 | docs/config |
| TOML | 1 | 2% |
Start reading here
Probable source entrypoint: pyproject.toml — inferred from the directory tree and manifest, not the upstream docs.
Source roots
httpx/ (24 files)
Source layout — httpx/
| Path | Files |
|---|---|
httpx/_transports/ | 6 |
httpx/__init__.py | 1 |
httpx/__version__.py | 1 |
httpx/_api.py | 1 |
httpx/_auth.py | 1 |
httpx/_client.py | 1 |
httpx/_config.py | 1 |
httpx/_content.py | 1 |
httpx/_decoders.py | 1 |
httpx/_exceptions.py | 1 |
httpx/_main.py | 1 |
httpx/_models.py | 1 |
httpx/_multipart.py | 1 |
httpx/_status_codes.py | 1 |
Repository structure
| Top-level path | Files |
|---|---|
tests/ | 40 |
docs/ | 26 |
httpx/ | 24 |
.github/ | 8 |
scripts/ | 8 |
Docs found: LICENSE.mdREADME.mddocs/advanced/authentication.mddocs/advanced/clients.mddocs/advanced/event-hooks.mddocs/advanced/extensions.mddocs/advanced/proxies.mddocs/advanced/resource-limits.mddocs/advanced/ssl.mddocs/advanced/text-encodings.md
Package
pyproject.toml — httpx, 18 declared dependencies.
brotlibrotli;brotlicffi;]cliclick==8.*"pygments==2.*"rich>=10,<15"]http2h2>=3,<5"]
CI & config entrypoints
README.md— Repository readme.github/workflows/publish.yml— GitHub Actions workflow.github/workflows/test-suite.yml— GitHub Actions workflow
Structural health signals
Average score 7.1/10 across 23 analyzed source files; 6 hotspots surfaced.
httpx/_client.py— 1497 lines, score 4.7 (Large file, Deep nesting)httpx/_models.py— 1277 lines, score 4.7 (Large file, Deep nesting)httpx/_urls.py— 641 lines, score 6.2 (Large file, Deep nesting)
These are deterministic review leads — size, churn, and structural signals — not proof of a bug in this project.
Recent history
b5addb64f0162026-02-23 — Adapt test_response_decode_text_using_autodetect for chardet 6.0 (#3773)ae1b9f66238f2025-12-10 — Expose `FunctionAuth` in `__all__` (#3699)ca097c96f97d2025-12-10 — docs/ssl: fix typo (#3703)def4778d622e2025-10-16 — Fixed a syntax error in the file upload example (#3692)435e1dac899a2025-10-04 — Bump actions/setup-python from 5 to 6 (#3677)