urllib3.contrib package

These modules implement various extra features, that may not be ready for prime time or that require optional third-party dependencies.

urllib3.contrib.appengine module

This module provides a pool manager that uses Google App Engine’s URLFetch Service.

Example usage:

from urllib3 import PoolManager
from urllib3.contrib.appengine import AppEngineManager, is_appengine_sandbox

if is_appengine_sandbox():
    # AppEngineManager uses AppEngine's URLFetch API behind the scenes
    http = AppEngineManager()
else:
    # PoolManager uses a socket-level API behind the scenes
    http = PoolManager()

r = http.request('GET', 'https://google.com/')

There are limitations to the URLFetch service and it may not be the best choice for your application. There are three options for using urllib3 on Google App Engine:

  1. You can use AppEngineManager with URLFetch. URLFetch is cost-effective in many circumstances as long as your usage is within the limitations.

  2. You can use a normal PoolManager by enabling sockets. Sockets also have limitations and restrictions and have a lower free quota than URLFetch. To use sockets, be sure to specify the following in your app.yaml:

    env_variables:
        GAE_USE_SOCKETS_HTTPLIB : 'true'
    

3. If you are using App Engine Flexible, you can use the standard PoolManager without any configuration or special environment variables.

class urllib3.contrib.appengine.AppEngineManager(headers=None, retries=None, validate_certificate=True, urlfetch_retries=True)

Bases: urllib3.request.RequestMethods

Connection manager for Google App Engine sandbox applications.

This manager uses the URLFetch service directly instead of using the emulated httplib, and is subject to URLFetch limitations as described in the App Engine documentation here.

Notably it will raise an AppEnginePlatformError if:
  • URLFetch is not available.
  • If you attempt to use this on App Engine Flexible, as full socket support is available.
  • If a request size is more than 10 megabytes.
  • If a response size is more than 32 megabtyes.
  • If you use an unsupported request method such as OPTIONS.

Beyond those cases, it will raise normal urllib3 errors.

urlopen(method, url, body=None, headers=None, retries=None, redirect=True, timeout=<object object>, **response_kw)
exception urllib3.contrib.appengine.AppEnginePlatformError

Bases: urllib3.exceptions.HTTPError

exception urllib3.contrib.appengine.AppEnginePlatformWarning

Bases: urllib3.exceptions.HTTPWarning

urllib3.contrib.appengine.is_appengine()
urllib3.contrib.appengine.is_appengine_sandbox()
urllib3.contrib.appengine.is_local_appengine()
urllib3.contrib.appengine.is_prod_appengine()
urllib3.contrib.appengine.is_prod_appengine_mvms()

urllib3.contrib.ntlmpool module

NTLM authenticating pool, contributed by erikcederstran

Issue #10, see: http://code.google.com/p/urllib3/issues/detail?id=10

class urllib3.contrib.ntlmpool.NTLMConnectionPool(user, pw, authurl, *args, **kwargs)

Bases: urllib3.connectionpool.HTTPSConnectionPool

Implements an NTLM authentication version of an urllib3 connection pool

scheme = 'https'
urlopen(method, url, body=None, headers=None, retries=3, redirect=True, assert_same_host=True)

urllib3.contrib.pyopenssl module

SSL with SNI-support for Python 2. Follow these instructions if you would like to verify SSL certificates in Python 2. Note, the default libraries do not do certificate checking; you need to do additional work to validate certificates yourself.

This needs the following packages installed:

  • pyOpenSSL (tested with 16.0.0)
  • cryptography (minimum 1.3.4, from pyopenssl)
  • idna (minimum 2.0, from cryptography)

However, pyopenssl depends on cryptography, which depends on idna, so while we use all three directly here we end up having relatively few packages required.

You can install them with the following command:

pip install pyopenssl cryptography idna

To activate certificate checking, call inject_into_urllib3() from your Python code before you begin making HTTP requests. This can be done in a sitecustomize module, or at any other time before your application begins using urllib3, like this:

try:
    import urllib3.contrib.pyopenssl
    urllib3.contrib.pyopenssl.inject_into_urllib3()
except ImportError:
    pass

Now you can use urllib3 as you normally would, and it will support SNI when the required modules are installed.

Activating this module also has the positive side effect of disabling SSL/TLS compression in Python 2 (see CRIME attack).

If you want to configure the default list of supported cipher suites, you can set the urllib3.contrib.pyopenssl.DEFAULT_SSL_CIPHER_LIST variable.

urllib3.contrib.pyopenssl.inject_into_urllib3()

Monkey-patch urllib3 with PyOpenSSL-backed SSL-support.

urllib3.contrib.pyopenssl.extract_from_urllib3()

Undo monkey-patching by inject_into_urllib3().

urllib3.contrib.socks module

This module contains provisional support for SOCKS proxies from within urllib3. This module supports SOCKS4 (specifically the SOCKS4A variant) and SOCKS5. To enable its functionality, either install PySocks or install this module with the socks extra.

The SOCKS implementation supports the full range of urllib3 features. It also supports the following SOCKS features:

  • SOCKS4
  • SOCKS4a
  • SOCKS5
  • Usernames and passwords for the SOCKS proxy

Known Limitations:

  • Currently PySocks does not support contacting remote websites via literal IPv6 addresses. Any such connection attempt will fail. You must use a domain name.
  • Currently PySocks does not support IPv6 connections to the SOCKS proxy. Any such connection attempt will fail.
class urllib3.contrib.socks.SOCKSConnection(*args, **kwargs)

Bases: urllib3.connection.HTTPConnection

A plain-text HTTP connection that connects via a SOCKS proxy.

class urllib3.contrib.socks.SOCKSHTTPConnectionPool(host, port=None, strict=False, timeout=<object object>, maxsize=1, block=False, headers=None, retries=None, _proxy=None, _proxy_headers=None, **conn_kw)

Bases: urllib3.connectionpool.HTTPConnectionPool

ConnectionCls

alias of SOCKSConnection

class urllib3.contrib.socks.SOCKSHTTPSConnection(*args, **kwargs)

Bases: urllib3.contrib.socks.SOCKSConnection, urllib3.connection.VerifiedHTTPSConnection

class urllib3.contrib.socks.SOCKSHTTPSConnectionPool(host, port=None, strict=False, timeout=<object object>, maxsize=1, block=False, headers=None, retries=None, _proxy=None, _proxy_headers=None, key_file=None, cert_file=None, cert_reqs=None, ca_certs=None, ssl_version=None, assert_hostname=None, assert_fingerprint=None, ca_cert_dir=None, **conn_kw)

Bases: urllib3.connectionpool.HTTPSConnectionPool

ConnectionCls

alias of SOCKSHTTPSConnection

class urllib3.contrib.socks.SOCKSProxyManager(proxy_url, username=None, password=None, num_pools=10, headers=None, **connection_pool_kw)

Bases: urllib3.poolmanager.PoolManager

A version of the urllib3 ProxyManager that routes connections via the defined SOCKS proxy.

pool_classes_by_scheme = {'http': <class 'urllib3.contrib.socks.SOCKSHTTPConnectionPool'>, 'https': <class 'urllib3.contrib.socks.SOCKSHTTPSConnectionPool'>}