Utilities#

Useful methods for working with http.client, completely decoupled from code specific to urllib3.

At the very core, just like its predecessors, urllib3 is built on top of http.client – the lowest level HTTP library included in the Python standard library.

To aid the limited functionality of the http.client module, urllib3 provides various helper methods which are used with the higher level components but can also be used independently.

class urllib3.util.Retry(total=10, connect=None, read=None, redirect=None, status=None, other=None, allowed_methods=frozenset({'DELETE', 'GET', 'HEAD', 'OPTIONS', 'PUT', 'TRACE'}), status_forcelist=None, backoff_factor=0, backoff_max=120, raise_on_redirect=True, raise_on_status=True, history=None, respect_retry_after_header=True, remove_headers_on_redirect=frozenset({'Authorization'}))#

Bases: object

Retry configuration.

Each retry attempt will create a new Retry object with updated values, so they can be safely reused.

Retries can be defined as a default for a pool:

retries = Retry(connect=5, read=2, redirect=5)
http = PoolManager(retries=retries)
response = http.request("GET", "https://example.com/")

Or per-request (which overrides the default for the pool):

response = http.request("GET", "https://example.com/", retries=Retry(10))

Retries can be disabled by passing False:

response = http.request("GET", "https://example.com/", retries=False)

Errors will be wrapped in MaxRetryError unless retries are disabled, in which case the causing exception will be raised.

Parameters
  • total (int) –

    Total number of retries to allow. Takes precedence over other counts.

    Set to None to remove this constraint and fall back on other counts.

    Set to 0 to fail on the first retry.

    Set to False to disable and imply raise_on_redirect=False.

  • connect (int) –

    How many connection-related errors to retry on.

    These are errors raised before the request is sent to the remote server, which we assume has not triggered the server to process the request.

    Set to 0 to fail on the first retry of this type.

  • read (int) –

    How many times to retry on read errors.

    These errors are raised after the request was sent to the server, so the request may have side-effects.

    Set to 0 to fail on the first retry of this type.

  • redirect (int) –

    How many redirects to perform. Limit this to avoid infinite redirect loops.

    A redirect is a HTTP response with a status code 301, 302, 303, 307 or 308.

    Set to 0 to fail on the first retry of this type.

    Set to False to disable and imply raise_on_redirect=False.

  • status (int) –

    How many times to retry on bad status codes.

    These are retries made on responses, where status code matches status_forcelist.

    Set to 0 to fail on the first retry of this type.

  • other (int) –

    How many times to retry on other errors.

    Other errors are errors that are not connect, read, redirect or status errors. These errors might be raised after the request was sent to the server, so the request might have side-effects.

    Set to 0 to fail on the first retry of this type.

    If total is not set, it’s a good idea to set this to 0 to account for unexpected edge cases and avoid infinite retry loops.

  • allowed_methods (Collection) –

    Set of uppercased HTTP method verbs that we should retry on.

    By default, we only retry on methods which are considered to be idempotent (multiple requests with the same parameters end with the same state). See Retry.DEFAULT_ALLOWED_METHODS.

    Set to a None value to retry on any verb.

  • status_forcelist (Collection) –

    A set of integer HTTP status codes that we should force a retry on. A retry is initiated if the request method is in allowed_methods and the response status code is in status_forcelist.

    By default, this is disabled with None.

  • backoff_factor (float) –

    A backoff factor to apply between attempts after the second try (most errors are resolved immediately by a second try without a delay). urllib3 will sleep for:

    {backoff factor} * (2 ** ({number of total retries} - 1))
    

    seconds. If the backoff_factor is 0.1, then Retry.sleep() will sleep for [0.0s, 0.2s, 0.4s, …] between retries. It will never be longer than backoff_max.

    By default, backoff is disabled (set to 0).

  • raise_on_redirect (bool) – Whether, if the number of redirects is exhausted, to raise a MaxRetryError, or to return a response with a response code in the 3xx range.

  • raise_on_status (bool) – Similar meaning to raise_on_redirect: whether we should raise an exception, or return a response, if status falls in status_forcelist range and retries have been exhausted.

  • history (tuple) – The history of the request encountered during each call to increment(). The list is in the order the requests occurred. Each list item is of class RequestHistory.

  • respect_retry_after_header (bool) – Whether to respect Retry-After header on status codes defined as Retry.RETRY_AFTER_STATUS_CODES or not.

  • remove_headers_on_redirect (Collection) – Sequence of headers to remove from the request when a response indicating a redirect is returned before firing off the redirected request.

  • backoff_max (float) –

Return type

None

DEFAULT_ALLOWED_METHODS = frozenset({'DELETE', 'GET', 'HEAD', 'OPTIONS', 'PUT', 'TRACE'})#

Default methods to be used for allowed_methods

DEFAULT_BACKOFF_MAX = 120#

Default maximum backoff time.

DEFAULT_REMOVE_HEADERS_ON_REDIRECT = frozenset({'Authorization'})#

Default headers to be used for remove_headers_on_redirect

RETRY_AFTER_STATUS_CODES = frozenset({413, 429, 503})#

Default status codes to be used for status_forcelist

classmethod from_int(retries, redirect=True, default=None)#

Backwards-compatibility for the old retries format.

Parameters
Return type

urllib3.util.retry.Retry

get_backoff_time()#

Formula for computing the current backoff

Return type

float

get_retry_after(response)#

Get the value of Retry-After in seconds.

Parameters

response (HTTPResponse) –

Return type

Optional[float]

increment(method=None, url=None, response=None, error=None, _pool=None, _stacktrace=None)#

Return a new Retry object with incremented retry counters.

Parameters
Returns

A new Retry object.

Return type

Retry

is_exhausted()#

Are we out of retries?

Return type

bool

is_retry(method, status_code, has_retry_after=False)#

Is this method/status code retryable? (Based on allowlists and control variables such as the number of total retries to allow, whether to respect the Retry-After header, whether this header is present, and whether the returned status code is on the list of status codes to be retried upon on the presence of the aforementioned header)

Parameters
  • method (str) –

  • status_code (int) –

  • has_retry_after (bool) –

Return type

bool

sleep(response=None)#

Sleep between retry attempts.

This method will respect a server’s Retry-After response header and sleep the duration of the time requested. If that is not present, it will use an exponential backoff. By default, the backoff factor is 0 and this method will return immediately.

Parameters

response (Optional[HTTPResponse]) –

Return type

None

class urllib3.util.SSLContext(protocol=_SSLMethod.PROTOCOL_TLS, *args, **kwargs)#

Bases: _ssl._SSLContext

An SSLContext holds various SSL-related configuration options and data, such as certificates and possibly a private key.

sslobject_class#

alias of ssl.SSLObject

sslsocket_class#

alias of ssl.SSLSocket

class urllib3.util.Timeout(total=None, connect=_TYPE_DEFAULT.token, read=_TYPE_DEFAULT.token)#

Bases: object

Timeout configuration.

Timeouts can be defined as a default for a pool:

import urllib3

timeout = urllib3.util.Timeout(connect=2.0, read=7.0)

http = urllib3.PoolManager(timeout=timeout)

resp = http.request("GET", "https://example.com/")

print(resp.status)

Or per-request (which overrides the default for the pool):

response = http.request("GET", "https://example.com/", timeout=Timeout(10))

Timeouts can be disabled by setting all the parameters to None:

no_timeout = Timeout(connect=None, read=None)
response = http.request("GET", "https://example.com/", timeout=no_timeout)
Parameters
  • total (int, float, or None) –

    This combines the connect and read timeouts into one; the read timeout will be set to the time leftover from the connect attempt. In the event that both a connect timeout and a total are specified, or a read timeout and a total are specified, the shorter timeout will be applied.

    Defaults to None.

  • connect (int, float, or None) – The maximum amount of time (in seconds) to wait for a connection attempt to a server to succeed. Omitting the parameter will default the connect timeout to the system default, probably the global default timeout in socket.py. None will set an infinite timeout for connection attempts.

  • read (int, float, or None) –

    The maximum amount of time (in seconds) to wait between consecutive read operations for a response from the server. Omitting the parameter will default the read timeout to the system default, probably the global default timeout in socket.py. None will set an infinite timeout.

Return type

None

Note

Many factors can affect the total amount of time for urllib3 to return an HTTP response.

For example, Python’s DNS resolver does not obey the timeout specified on the socket. Other factors that can affect total request time include high CPU load, high swap, the program running at a low priority level, or other behaviors.

In addition, the read and total timeouts only measure the time between read operations on the socket connecting the client and the server, not the total amount of time for the request to return a complete response. For most requests, the timeout is raised because the server has not sent the first byte in the specified time. This is not always the case; if a server streams one byte every fifteen seconds, a timeout of 20 seconds will not trigger, even though the request will take several minutes to complete.

If your goal is to cut off any request after a set amount of wall clock time, consider having a second “watcher” thread to cut off a slow request.

DEFAULT_TIMEOUT: Optional[Union[float, urllib3.util.timeout._TYPE_DEFAULT]] = 0#

A sentinel object representing the default timeout value

clone()#

Create a copy of the timeout object

Timeout properties are stored per-pool but each request needs a fresh Timeout object to ensure each one has its own start/stop configured.

Returns

a copy of the timeout object

Return type

Timeout

property connect_timeout: Optional[Union[float, urllib3.util.timeout._TYPE_DEFAULT]]#

Get the value to use when setting a connection timeout.

This will be a positive float or integer, the value None (never timeout), or the default system timeout.

Returns

Connect timeout.

Return type

int, float, Timeout.DEFAULT_TIMEOUT or None

classmethod from_float(timeout)#

Create a new Timeout from a legacy timeout value.

The timeout value used by httplib.py sets the same timeout on the connect(), and recv() socket requests. This creates a Timeout object that sets the individual timeouts to the timeout value passed to this function.

Parameters

timeout (integer, float, urllib3.util.Timeout.DEFAULT_TIMEOUT, or None) – The legacy timeout value.

Returns

Timeout object

Return type

Timeout

get_connect_duration()#

Gets the time elapsed since the call to start_connect().

Returns

Elapsed time in seconds.

Return type

float

Raises

urllib3.exceptions.TimeoutStateError – if you attempt to get duration for a timer that hasn’t been started.

property read_timeout: Optional[float]#

Get the value for the read timeout.

This assumes some time has elapsed in the connection timeout and computes the read timeout appropriately.

If self.total is set, the read timeout is dependent on the amount of time taken by the connect timeout. If the connection time has not been established, a TimeoutStateError will be raised.

Returns

Value to use for the read timeout.

Return type

int, float or None

Raises

urllib3.exceptions.TimeoutStateError – If start_connect() has not yet been called on this object.

start_connect()#

Start the timeout clock, used during a connect() attempt

Raises

urllib3.exceptions.TimeoutStateError – if you attempt to start a timer that has been started already.

Return type

float

class urllib3.util.Url(scheme=None, auth=None, host=None, port=None, path=None, query=None, fragment=None)#

Bases: urllib3.util.url.Url

Data structure for representing an HTTP URL. Used as a return value for parse_url(). Both the scheme and host are normalized as they are both case-insensitive according to RFC 3986.

Parameters
property authority: Optional[str]#

Authority component as defined in RFC 3986 3.2. This includes userinfo (auth), host and port.

i.e.

userinfo@host:port

property hostname: Optional[str]#

For backwards-compatibility with urlparse. We’re nice like that.

property netloc: Optional[str]#

Network location including host and port.

If you need the equivalent of urllib.parse’s netloc, use the authority property instead.

property request_uri: str#

Absolute path including the query string.

property url: str#

Convert self into a url

This function should more or less round-trip with parse_url(). The returned url may not be exactly the same as the url inputted to parse_url(), but it should be equivalent by the RFC (e.g., urls with a blank port will have : removed).

Example:

import urllib3

U = urllib3.util.parse_url("https://google.com/mail/")

print(U.url)
# "https://google.com/mail/"

print( urllib3.util.Url("https", "username:password",
                        "host.com", 80, "/path", "query", "fragment"
                        ).url
    )
# "https://username:password@host.com:80/path?query#fragment"
urllib3.util.assert_fingerprint(cert, fingerprint)#

Checks if given fingerprint matches the supplied certificate.

Parameters
  • cert (Optional[bytes]) – Certificate as bytes object.

  • fingerprint (str) – Fingerprint as string of hexdigits, can be interspersed by colons.

Return type

None

urllib3.util.create_urllib3_context(ssl_version=None, cert_reqs=None, options=None, ciphers=None, ssl_minimum_version=None, ssl_maximum_version=None)#

All arguments have the same meaning as ssl_wrap_socket.

By default, this function does a lot of the same work that ssl.create_default_context does on Python 3.4+. It:

  • Disables SSLv2, SSLv3, and compression

  • Sets a restricted set of server ciphers

If you wish to enable SSLv3, you can do:

from urllib3.util import ssl_
context = ssl_.create_urllib3_context()
context.options &= ~ssl_.OP_NO_SSLv3

You can do the same to enable compression (substituting COMPRESSION for SSLv3 in the last line above).

Parameters
  • ssl_version (Optional[int]) –

    The desired protocol version to use. This will default to PROTOCOL_SSLv23 which will negotiate the highest protocol that both the server and your installation of OpenSSL support.

    This parameter is deprecated instead use ‘ssl_minimum_version’.

  • ssl_minimum_version (Optional[int]) – The minimum version of TLS to be used. Use the ‘ssl.TLSVersion’ enum for specifying the value.

  • ssl_maximum_version (Optional[int]) – The maximum version of TLS to be used. Use the ‘ssl.TLSVersion’ enum for specifying the value. Not recommended to set to anything other than ‘ssl.TLSVersion.MAXIMUM_SUPPORTED’ which is the default value.

  • cert_reqs (Optional[int]) – Whether to require the certificate verification. This defaults to ssl.CERT_REQUIRED.

  • options (Optional[int]) – Specific OpenSSL options. These default to ssl.OP_NO_SSLv2, ssl.OP_NO_SSLv3, ssl.OP_NO_COMPRESSION, and ssl.OP_NO_TICKET.

  • ciphers (Optional[str]) – Which cipher suites to allow the server to select. Defaults to either system configured ciphers if OpenSSL 1.1.1+, otherwise uses a secure default set of ciphers.

Returns

Constructed SSLContext object with specified options

Return type

SSLContext

urllib3.util.is_connection_dropped(conn)#

Returns True if the connection is dropped and should be closed.

Parameters

conn (socket.socket) – http.client.HTTPConnection object.

Return type

bool

urllib3.util.is_fp_closed(obj)#

Checks whether a given file-like object is closed.

Parameters

obj (object) – The file-like object to check.

Return type

bool

urllib3.util.make_headers(keep_alive=None, accept_encoding=None, user_agent=None, basic_auth=None, proxy_basic_auth=None, disable_cache=None)#

Shortcuts for generating request headers.

Parameters
  • keep_alive (Optional[bool]) – If True, adds ‘connection: keep-alive’ header.

  • accept_encoding (Optional[Union[bool, List[str], str]]) – Can be a boolean, list, or string. True translates to ‘gzip,deflate’. If either the brotli or brotlicffi package is installed ‘gzip,deflate,br’ is used instead. List will get joined by comma. String will be used as provided.

  • user_agent (Optional[str]) – String representing the user-agent you want, such as “python-urllib3/0.6”

  • basic_auth (Optional[str]) – Colon-separated username:password string for ‘authorization: basic …’ auth header.

  • proxy_basic_auth (Optional[str]) – Colon-separated username:password string for ‘proxy-authorization: basic …’ auth header.

  • disable_cache (Optional[bool]) – If True, adds ‘cache-control: no-cache’ header.

Return type

Dict[str, str]

Example:

import urllib3

print(urllib3.util.make_headers(keep_alive=True, user_agent="Batman/1.0"))
# {'connection': 'keep-alive', 'user-agent': 'Batman/1.0'}
print(urllib3.util.make_headers(accept_encoding=True))
# {'accept-encoding': 'gzip,deflate'}
urllib3.util.parse_url(url)#

Given a url, return a parsed Url namedtuple. Best-effort is performed to parse incomplete urls. Fields not provided will be None. This parser is RFC 3986 compliant.

The parser logic and helper functions are based heavily on work done in the rfc3986 module.

Parameters

url (str) – URL to parse into a Url namedtuple.

Return type

urllib3.util.url.Url

Partly backwards-compatible with urllib.parse.

Example:

import urllib3

print( urllib3.util.parse_url('http://google.com/mail/'))
# Url(scheme='http', host='google.com', port=None, path='/mail/', ...)

print( urllib3.util.parse_url('google.com:80'))
# Url(scheme=None, host='google.com', port=80, path=None, ...)

print( urllib3.util.parse_url('/foo?bar'))
# Url(scheme=None, host=None, port=None, path='/foo', query='bar', ...)
urllib3.util.resolve_cert_reqs(candidate)#

Resolves the argument to a numeric constant, which can be passed to the wrap_socket function/method from the ssl module. Defaults to ssl.CERT_REQUIRED. If given a string it is assumed to be the name of the constant in the ssl module or its abbreviation. (So you can specify REQUIRED instead of CERT_REQUIRED. If it’s neither None nor a string we assume it is already the numeric constant which can directly be passed to wrap_socket.

Parameters

candidate (Union[None, int, str]) –

Return type

VerifyMode

urllib3.util.resolve_ssl_version(candidate)#

like resolve_cert_reqs

Parameters

candidate (Union[None, int, str]) –

Return type

int

urllib3.util.ssl_wrap_socket(sock, keyfile=None, certfile=None, cert_reqs=None, ca_certs=None, server_hostname=None, ssl_version=None, ciphers=None, ssl_context=None, ca_cert_dir=None, key_password=None, ca_cert_data=None, tls_in_tls=False)#

All arguments except for server_hostname, ssl_context, and ca_cert_dir have the same meaning as they do when using ssl.wrap_socket().

Parameters
  • server_hostname (Optional[str]) – When SNI is supported, the expected hostname of the certificate

  • ssl_context (Optional[ssl.SSLContext]) – A pre-made SSLContext object. If none is provided, one will be created using create_urllib3_context().

  • ciphers (Optional[str]) – A string of ciphers we wish the client to support.

  • ca_cert_dir (Optional[str]) – A directory containing CA certificates in multiple separate files, as supported by OpenSSL’s -CApath flag or the capath argument to SSLContext.load_verify_locations().

  • key_password (Optional[str]) – Optional password if the keyfile is encrypted.

  • ca_cert_data (Union[None, str, bytes]) – Optional string containing CA certificates in PEM format suitable for passing as the cadata parameter to SSLContext.load_verify_locations()

  • tls_in_tls (bool) – Use SSLTransport to wrap the existing socket.

  • sock (socket.socket) –

  • keyfile (Optional[str]) –

  • certfile (Optional[str]) –

  • cert_reqs (Optional[int]) –

  • ca_certs (Optional[str]) –

  • ssl_version (Optional[int]) –

Return type

Union[ssl.SSLSocket, SSLTransportType]

urllib3.util.wait_for_read(sock, timeout=None)#

Waits for reading to be available on a given socket. Returns True if the socket is readable, or False if the timeout expired.

Parameters
Return type

bool

urllib3.util.wait_for_write(sock, timeout=None)#

Waits for writing to be available on a given socket. Returns True if the socket is readable, or False if the timeout expired.

Parameters
Return type

bool