| Server IP : 82.208.35.60 / Your IP : 216.73.217.33 Web Server : Apache/2.4.55 (FreeBSD) OpenSSL/1.1.1q-freebsd PHP/7.3.31 System : FreeBSD server7.d2m.cz 12.4-RELEASE-p9 FreeBSD 12.4-RELEASE-p9 GENERIC amd64 User : studiokobylisy_cz ( 1008) PHP Version : 7.3.31 Disable Function : NONE MySQL : OFF | cURL : ON | WGET : OFF | Perl : OFF | Python : OFF | Sudo : OFF | Pkexec : OFF Directory : /usr/local/share/doc/requests-toolbelt/ |
Upload File : |
<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Uploading Data — requests_toolbelt 0.8.0 documentation</title>
<script type="text/javascript" src="_static/js/modernizr.min.js"></script>
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT:'./',
VERSION:'0.8.0',
LANGUAGE:'None',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
HAS_SOURCE: true,
SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<script type="text/javascript" src="_static/js/theme.js"></script>
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="User-Agent Constructor" href="user-agent.html" />
<link rel="prev" title="Using requests with Threading" href="threading.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="index.html" class="icon icon-home"> requests_toolbelt
</a>
<div class="version">
0.8.0
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
<ul>
<li class="toctree-l1"><a class="reference internal" href="user.html">requests toolbelt</a></li>
<li class="toctree-l1"><a class="reference internal" href="contributing.html">Contributing to this project</a></li>
</ul>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="adapters.html">Transport Adapters</a></li>
<li class="toctree-l1"><a class="reference internal" href="authentication.html">Authentication</a></li>
<li class="toctree-l1"><a class="reference internal" href="deprecated.html">Deprecated Requests Utilities</a></li>
<li class="toctree-l1"><a class="reference internal" href="downloadutils.html">Utilities for Downloading Streaming Responses</a></li>
<li class="toctree-l1"><a class="reference internal" href="dumputils.html">Utilities for Dumping Information About Responses</a></li>
<li class="toctree-l1"><a class="reference internal" href="formdata.html">Utilities for Enhanced Form-Data Serialization</a></li>
<li class="toctree-l1"><a class="reference internal" href="exceptions.html">Custom Toolbelt Exceptions</a></li>
<li class="toctree-l1"><a class="reference internal" href="sessions.html">Specialized Sessions</a></li>
<li class="toctree-l1"><a class="reference internal" href="threading.html">Using requests with Threading</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Uploading Data</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#streaming-multipart-data-encoder">Streaming Multipart Data Encoder</a></li>
<li class="toctree-l2"><a class="reference internal" href="#monitoring-your-streaming-multipart-upload">Monitoring Your Streaming Multipart Upload</a></li>
<li class="toctree-l2"><a class="reference internal" href="#streaming-data-from-a-generator">Streaming Data from a Generator</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="user-agent.html">User-Agent Constructor</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
<nav class="wy-nav-top" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="index.html">requests_toolbelt</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="index.html">Docs</a> »</li>
<li>Uploading Data</li>
<li class="wy-breadcrumbs-aside">
<a href="_sources/uploading-data.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<div class="section" id="uploading-data">
<span id="id1"></span><h1>Uploading Data<a class="headerlink" href="#uploading-data" title="Permalink to this headline">¶</a></h1>
<div class="section" id="streaming-multipart-data-encoder">
<h2>Streaming Multipart Data Encoder<a class="headerlink" href="#streaming-multipart-data-encoder" title="Permalink to this headline">¶</a></h2>
<p>Requests has <a class="reference external" href="http://docs.python-requests.org/en/latest/user/quickstart/#post-a-multipart-encoded-file">support for multipart uploads</a>, but the API means that using
that functionality to build exactly the Multipart upload you want can be
difficult or impossible. Additionally, when using Requests’ Multipart upload
functionality all the data must be read into memory before being sent to the
server. In extreme cases, this can make it impossible to send a file as part of
a <code class="docutils literal"><span class="pre">multipart/form-data</span></code> upload.</p>
<p>The toolbelt contains a class that allows you to build multipart request bodies
in exactly the format you need, and to avoid reading files into memory. An
example of how to use it is like this:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">requests</span>
<span class="kn">from</span> <span class="nn">requests_toolbelt.multipart.encoder</span> <span class="kn">import</span> <span class="n">MultipartEncoder</span>
<span class="n">m</span> <span class="o">=</span> <span class="n">MultipartEncoder</span><span class="p">(</span>
<span class="n">fields</span><span class="o">=</span><span class="p">{</span><span class="s1">'field0'</span><span class="p">:</span> <span class="s1">'value'</span><span class="p">,</span> <span class="s1">'field1'</span><span class="p">:</span> <span class="s1">'value'</span><span class="p">,</span>
<span class="s1">'field2'</span><span class="p">:</span> <span class="p">(</span><span class="s1">'filename'</span><span class="p">,</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'file.py'</span><span class="p">,</span> <span class="s1">'rb'</span><span class="p">),</span> <span class="s1">'text/plain'</span><span class="p">)}</span>
<span class="p">)</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="s1">'http://httpbin.org/post'</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">m</span><span class="p">,</span>
<span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s1">'Content-Type'</span><span class="p">:</span> <span class="n">m</span><span class="o">.</span><span class="n">content_type</span><span class="p">})</span>
</pre></div>
</div>
<p>The <a class="reference internal" href="#requests_toolbelt.multipart.encoder.MultipartEncoder" title="requests_toolbelt.multipart.encoder.MultipartEncoder"><code class="xref py py-class docutils literal"><span class="pre">MultipartEncoder</span></code></a> has the
<code class="docutils literal"><span class="pre">.to_string()</span></code> convenience method, as well. This method renders the
multipart body into a string. This is useful when developing your code,
allowing you to confirm that the multipart body has the form you expect before
you send it on.</p>
<p>The toolbelt also provides a way to monitor your streaming uploads with
the <a class="reference internal" href="#requests_toolbelt.multipart.encoder.MultipartEncoderMonitor" title="requests_toolbelt.multipart.encoder.MultipartEncoderMonitor"><code class="xref py py-class docutils literal"><span class="pre">MultipartEncoderMonitor</span></code></a>.</p>
<dl class="class">
<dt id="requests_toolbelt.multipart.encoder.MultipartEncoder">
<em class="property">class </em><code class="descclassname">requests_toolbelt.multipart.encoder.</code><code class="descname">MultipartEncoder</code><span class="sig-paren">(</span><em>fields</em>, <em>boundary=None</em>, <em>encoding='utf-8'</em><span class="sig-paren">)</span><a class="headerlink" href="#requests_toolbelt.multipart.encoder.MultipartEncoder" title="Permalink to this definition">¶</a></dt>
<dd><p>The <code class="docutils literal"><span class="pre">MultipartEncoder</span></code> oject is a generic interface to the engine that
will create a <code class="docutils literal"><span class="pre">multipart/form-data</span></code> body for you.</p>
<p>The basic usage is:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">requests</span>
<span class="kn">from</span> <span class="nn">requests_toolbelt</span> <span class="kn">import</span> <span class="n">MultipartEncoder</span>
<span class="n">encoder</span> <span class="o">=</span> <span class="n">MultipartEncoder</span><span class="p">({</span><span class="s1">'field'</span><span class="p">:</span> <span class="s1">'value'</span><span class="p">,</span>
<span class="s1">'other_field'</span><span class="p">,</span> <span class="s1">'other_value'</span><span class="p">})</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="s1">'https://httpbin.org/post'</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">encoder</span><span class="p">,</span>
<span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s1">'Content-Type'</span><span class="p">:</span> <span class="n">encoder</span><span class="o">.</span><span class="n">content_type</span><span class="p">})</span>
</pre></div>
</div>
<p>If you do not need to take advantage of streaming the post body, you can
also do:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="s1">'https://httpbin.org/post'</span><span class="p">,</span>
<span class="n">data</span><span class="o">=</span><span class="n">encoder</span><span class="o">.</span><span class="n">to_string</span><span class="p">(),</span>
<span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s1">'Content-Type'</span><span class="p">:</span> <span class="n">encoder</span><span class="o">.</span><span class="n">content_type</span><span class="p">})</span>
</pre></div>
</div>
<p>If you want the encoder to use a specific order, you can use an
OrderedDict or more simply, a list of tuples:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">encoder</span> <span class="o">=</span> <span class="n">MultipartEncoder</span><span class="p">([(</span><span class="s1">'field'</span><span class="p">,</span> <span class="s1">'value'</span><span class="p">),</span>
<span class="p">(</span><span class="s1">'other_field'</span><span class="p">,</span> <span class="s1">'other_value'</span><span class="p">)])</span>
</pre></div>
</div>
<div class="versionchanged">
<p><span class="versionmodified">Changed in version 0.4.0.</span></p>
</div>
<p>You can also provide tuples as part values as you would provide them to
requests’ <code class="docutils literal"><span class="pre">files</span></code> parameter.</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">encoder</span> <span class="o">=</span> <span class="n">MultipartEncoder</span><span class="p">({</span>
<span class="s1">'field'</span><span class="p">:</span> <span class="p">(</span><span class="s1">'file_name'</span><span class="p">,</span> <span class="sa">b</span><span class="s1">'{"a": "b"}'</span><span class="p">,</span> <span class="s1">'application/json'</span><span class="p">,</span>
<span class="p">{</span><span class="s1">'X-My-Header'</span><span class="p">:</span> <span class="s1">'my-value'</span><span class="p">})</span>
<span class="p">])</span>
</pre></div>
</div>
<div class="admonition warning">
<p class="first admonition-title">Warning</p>
<p class="last">This object will end up directly in <code class="xref py py-mod docutils literal"><span class="pre">httplib</span></code>. Currently,
<code class="xref py py-mod docutils literal"><span class="pre">httplib</span></code> has a hard-coded read size of <strong>8192 bytes</strong>. This
means that it will loop until the file has been read and your upload
could take a while. This is <strong>not</strong> a bug in requests. A feature is
being considered for this object to allow you, the user, to specify
what size should be returned on a read. If you have opinions on this,
please weigh in on <a class="reference external" href="https://github.com/sigmavirus24/requests-toolbelt/issues/75">this issue</a>.</p>
</div>
</dd></dl>
</div>
<div class="section" id="monitoring-your-streaming-multipart-upload">
<h2>Monitoring Your Streaming Multipart Upload<a class="headerlink" href="#monitoring-your-streaming-multipart-upload" title="Permalink to this headline">¶</a></h2>
<p>If you need to stream your <code class="docutils literal"><span class="pre">multipart/form-data</span></code> upload then you’re probably
in the situation where it might take a while to upload the content. In these
cases, it might make sense to be able to monitor the progress of the upload.
For this reason, the toolbelt provides the
<a class="reference internal" href="#requests_toolbelt.multipart.encoder.MultipartEncoderMonitor" title="requests_toolbelt.multipart.encoder.MultipartEncoderMonitor"><code class="xref py py-class docutils literal"><span class="pre">MultipartEncoderMonitor</span></code></a>. The
monitor wraps an instance of a
<a class="reference internal" href="#requests_toolbelt.multipart.encoder.MultipartEncoder" title="requests_toolbelt.multipart.encoder.MultipartEncoder"><code class="xref py py-class docutils literal"><span class="pre">MultipartEncoder</span></code></a> and is used
exactly like the encoder. It provides a similar API with some additions:</p>
<ul class="simple">
<li>The monitor accepts a function as a callback. The function is called every
time <code class="docutils literal"><span class="pre">requests</span></code> calls <code class="docutils literal"><span class="pre">read</span></code> on the monitor and passes in the monitor as
an argument.</li>
<li>The monitor tracks how many bytes have been read in the course of the
upload.</li>
</ul>
<p>You might use the monitor to create a progress bar for the upload. Here is <a class="reference external" href="https://gitlab.com/sigmavirus24/toolbelt/blob/master/examples/monitor/progress_bar.py">an
example using clint</a> which displays the progress bar.</p>
<p>To use the monitor you would follow a pattern like this:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">requests</span>
<span class="kn">from</span> <span class="nn">requests_toolbelt.multipart</span> <span class="kn">import</span> <span class="n">encoder</span>
<span class="k">def</span> <span class="nf">my_callback</span><span class="p">(</span><span class="n">monitor</span><span class="p">):</span>
<span class="c1"># Your callback function</span>
<span class="k">pass</span>
<span class="n">e</span> <span class="o">=</span> <span class="n">encoder</span><span class="o">.</span><span class="n">MultipartEncoder</span><span class="p">(</span>
<span class="n">fields</span><span class="o">=</span><span class="p">{</span><span class="s1">'field0'</span><span class="p">:</span> <span class="s1">'value'</span><span class="p">,</span> <span class="s1">'field1'</span><span class="p">:</span> <span class="s1">'value'</span><span class="p">,</span>
<span class="s1">'field2'</span><span class="p">:</span> <span class="p">(</span><span class="s1">'filename'</span><span class="p">,</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'file.py'</span><span class="p">,</span> <span class="s1">'rb'</span><span class="p">),</span> <span class="s1">'text/plain'</span><span class="p">)}</span>
<span class="p">)</span>
<span class="n">m</span> <span class="o">=</span> <span class="n">encoder</span><span class="o">.</span><span class="n">MultipartEncoderMonitor</span><span class="p">(</span><span class="n">e</span><span class="p">,</span> <span class="n">my_callback</span><span class="p">)</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="s1">'http://httpbin.org/post'</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">m</span><span class="p">,</span>
<span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s1">'Content-Type'</span><span class="p">:</span> <span class="n">m</span><span class="o">.</span><span class="n">content_type</span><span class="p">})</span>
</pre></div>
</div>
<p>If you have a very simple use case you can also do:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">requests</span>
<span class="kn">from</span> <span class="nn">requests_toolbelt.multipart.encoder</span> <span class="kn">import</span> <span class="n">MultipartEncoderMonitor</span>
<span class="k">def</span> <span class="nf">my_callback</span><span class="p">(</span><span class="n">monitor</span><span class="p">):</span>
<span class="c1"># Your callback function</span>
<span class="k">pass</span>
<span class="n">m</span> <span class="o">=</span> <span class="n">MultipartEncoderMonitor</span><span class="o">.</span><span class="n">from_fields</span><span class="p">(</span>
<span class="n">fields</span><span class="o">=</span><span class="p">{</span><span class="s1">'field0'</span><span class="p">:</span> <span class="s1">'value'</span><span class="p">,</span> <span class="s1">'field1'</span><span class="p">:</span> <span class="s1">'value'</span><span class="p">,</span>
<span class="s1">'field2'</span><span class="p">:</span> <span class="p">(</span><span class="s1">'filename'</span><span class="p">,</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'file.py'</span><span class="p">,</span> <span class="s1">'rb'</span><span class="p">),</span> <span class="s1">'text/plain'</span><span class="p">)},</span>
<span class="n">callback</span><span class="o">=</span><span class="n">my_callback</span>
<span class="p">)</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="s1">'http://httpbin.org/post'</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">m</span><span class="p">,</span>
<span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s1">'Content-Type'</span><span class="p">:</span> <span class="n">m</span><span class="o">.</span><span class="n">content_type</span><span class="p">})</span>
</pre></div>
</div>
<dl class="class">
<dt id="requests_toolbelt.multipart.encoder.MultipartEncoderMonitor">
<em class="property">class </em><code class="descclassname">requests_toolbelt.multipart.encoder.</code><code class="descname">MultipartEncoderMonitor</code><span class="sig-paren">(</span><em>encoder</em>, <em>callback=None</em><span class="sig-paren">)</span><a class="headerlink" href="#requests_toolbelt.multipart.encoder.MultipartEncoderMonitor" title="Permalink to this definition">¶</a></dt>
<dd><p>An object used to monitor the progress of a <a class="reference internal" href="#requests_toolbelt.multipart.encoder.MultipartEncoder" title="requests_toolbelt.multipart.encoder.MultipartEncoder"><code class="xref py py-class docutils literal"><span class="pre">MultipartEncoder</span></code></a>.</p>
<p>The <a class="reference internal" href="#requests_toolbelt.multipart.encoder.MultipartEncoder" title="requests_toolbelt.multipart.encoder.MultipartEncoder"><code class="xref py py-class docutils literal"><span class="pre">MultipartEncoder</span></code></a> should only be responsible for preparing and
streaming the data. For anyone who wishes to monitor it, they shouldn’t be
using that instance to manage that as well. Using this class, they can
monitor an encoder and register a callback. The callback receives the
instance of the monitor.</p>
<p>To use this monitor, you construct your <a class="reference internal" href="#requests_toolbelt.multipart.encoder.MultipartEncoder" title="requests_toolbelt.multipart.encoder.MultipartEncoder"><code class="xref py py-class docutils literal"><span class="pre">MultipartEncoder</span></code></a> as you
normally would.</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">requests_toolbelt</span> <span class="kn">import</span> <span class="p">(</span><span class="n">MultipartEncoder</span><span class="p">,</span>
<span class="n">MultipartEncoderMonitor</span><span class="p">)</span>
<span class="kn">import</span> <span class="nn">requests</span>
<span class="k">def</span> <span class="nf">callback</span><span class="p">(</span><span class="n">encoder</span><span class="p">,</span> <span class="n">bytes_read</span><span class="p">):</span>
<span class="c1"># Do something with this information</span>
<span class="k">pass</span>
<span class="n">m</span> <span class="o">=</span> <span class="n">MultipartEncoder</span><span class="p">(</span><span class="n">fields</span><span class="o">=</span><span class="p">{</span><span class="s1">'field0'</span><span class="p">:</span> <span class="s1">'value0'</span><span class="p">})</span>
<span class="n">monitor</span> <span class="o">=</span> <span class="n">MultipartEncoderMonitor</span><span class="p">(</span><span class="n">m</span><span class="p">,</span> <span class="n">callback</span><span class="p">)</span>
<span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'Content-Type'</span><span class="p">:</span> <span class="n">montior</span><span class="o">.</span><span class="n">content_type</span><span class="p">}</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="s1">'https://httpbin.org/post'</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">monitor</span><span class="p">,</span>
<span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">)</span>
</pre></div>
</div>
<p>Alternatively, if your use case is very simple, you can use the following
pattern.</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">requests_toolbelt</span> <span class="kn">import</span> <span class="n">MultipartEncoderMonitor</span>
<span class="kn">import</span> <span class="nn">requests</span>
<span class="k">def</span> <span class="nf">callback</span><span class="p">(</span><span class="n">encoder</span><span class="p">,</span> <span class="n">bytes_read</span><span class="p">):</span>
<span class="c1"># Do something with this information</span>
<span class="k">pass</span>
<span class="n">monitor</span> <span class="o">=</span> <span class="n">MultipartEncoderMonitor</span><span class="o">.</span><span class="n">from_fields</span><span class="p">(</span>
<span class="n">fields</span><span class="o">=</span><span class="p">{</span><span class="s1">'field0'</span><span class="p">:</span> <span class="s1">'value0'</span><span class="p">},</span> <span class="n">callback</span>
<span class="p">)</span>
<span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'Content-Type'</span><span class="p">:</span> <span class="n">montior</span><span class="o">.</span><span class="n">content_type</span><span class="p">}</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="s1">'https://httpbin.org/post'</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">monitor</span><span class="p">,</span>
<span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">)</span>
</pre></div>
</div>
</dd></dl>
</div>
<div class="section" id="streaming-data-from-a-generator">
<h2>Streaming Data from a Generator<a class="headerlink" href="#streaming-data-from-a-generator" title="Permalink to this headline">¶</a></h2>
<p>There are cases where you, the user, have a generator of some large quantity
of data and you already know the size of that data. If you pass the generator
to <code class="docutils literal"><span class="pre">requests</span></code> via the <code class="docutils literal"><span class="pre">data</span></code> parameter, <code class="docutils literal"><span class="pre">requests</span></code> will assume that you
want to upload the data in chunks and set a <code class="docutils literal"><span class="pre">Transfer-Encoding</span></code> header value
of <code class="docutils literal"><span class="pre">chunked</span></code>. Often times, this causes the server to behave poorly. If you
want to avoid this, you can use the
<code class="xref py py-class docutils literal"><span class="pre">StreamingIterator</span></code>. You pass it
the size of the data and the generator.</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">requests</span>
<span class="kn">from</span> <span class="nn">requests_toolbelt.streaming_iterator</span> <span class="kn">import</span> <span class="n">StreamingIterator</span>
<span class="n">generator</span> <span class="o">=</span> <span class="n">some_function</span><span class="p">()</span> <span class="c1"># Create your generator</span>
<span class="n">size</span> <span class="o">=</span> <span class="n">some_function_size</span><span class="p">()</span> <span class="c1"># Get your generator's size</span>
<span class="n">content_type</span> <span class="o">=</span> <span class="n">content_type</span><span class="p">()</span> <span class="c1"># Get the content-type of the data</span>
<span class="n">streamer</span> <span class="o">=</span> <span class="n">StreamingIterator</span><span class="p">(</span><span class="n">size</span><span class="p">,</span> <span class="n">generator</span><span class="p">)</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="s1">'https://httpbin.org/post'</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">streamer</span><span class="p">,</span>
<span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s1">'Content-Type'</span><span class="p">:</span> <span class="n">content_type</span><span class="p">})</span>
</pre></div>
</div>
<p>The streamer will handle your generator for you and buffer the data before
passing it to <code class="docutils literal"><span class="pre">requests</span></code>.</p>
<div class="versionchanged">
<p><span class="versionmodified">Changed in version 0.4.0: </span>File-like objects can be passed instead of a generator.</p>
</div>
<p>If, for example, you need to upload data being piped into standard in, you
might otherwise do:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">requests</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">sys</span><span class="o">.</span><span class="n">stdin</span><span class="p">)</span>
</pre></div>
</div>
<p>This would stream the data but would use a chunked transfer-encoding. If
instead, you know the length of the data that is being sent to <code class="docutils literal"><span class="pre">stdin</span></code> and
you want to prevent the data from being uploaded in chunks, you can use the
<a class="reference internal" href="#requests_toolbelt.streaming_iterator.StreamingIterator" title="requests_toolbelt.streaming_iterator.StreamingIterator"><code class="xref py py-class docutils literal"><span class="pre">StreamingIterator</span></code></a> to stream the
contents of the file without relying on chunking.</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">requests</span>
<span class="kn">from</span> <span class="nn">requests_toolbelt.streaming_iterator</span> <span class="kn">import</span> <span class="n">StreamingIterator</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="n">stream</span> <span class="o">=</span> <span class="n">StreamingIterator</span><span class="p">(</span><span class="n">size</span><span class="p">,</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdin</span><span class="p">)</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">stream</span><span class="p">,</span>
<span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s1">'Content-Type'</span><span class="p">:</span> <span class="n">content_type</span><span class="p">})</span>
</pre></div>
</div>
<dl class="class">
<dt id="requests_toolbelt.streaming_iterator.StreamingIterator">
<em class="property">class </em><code class="descclassname">requests_toolbelt.streaming_iterator.</code><code class="descname">StreamingIterator</code><span class="sig-paren">(</span><em>size</em>, <em>iterator</em>, <em>encoding='utf-8'</em><span class="sig-paren">)</span><a class="headerlink" href="#requests_toolbelt.streaming_iterator.StreamingIterator" title="Permalink to this definition">¶</a></dt>
<dd><p>This class provides a way of allowing iterators with a known size to be
streamed instead of chunked.</p>
<p>In requests, if you pass in an iterator it assumes you want to use
chunked transfer-encoding to upload the data, which not all servers
support well. Additionally, you may want to set the content-length
yourself to avoid this but that will not work. The only way to preempt
requests using a chunked transfer-encoding and forcing it to stream the
uploads is to mimic a very specific interace. Instead of having to know
these details you can instead just use this class. You simply provide the
size and iterator and pass the instance of StreamingIterator to requests
via the data parameter like so:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">requests_toolbelt</span> <span class="kn">import</span> <span class="n">StreamingIterator</span>
<span class="kn">import</span> <span class="nn">requests</span>
<span class="c1"># Let iterator be some generator that you already have and size be</span>
<span class="c1"># the size of the data produced by the iterator</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">StreamingIterator</span><span class="p">(</span><span class="n">size</span><span class="p">,</span> <span class="n">iterator</span><span class="p">))</span>
</pre></div>
</div>
<p>You can also pass file-like objects to <a class="reference internal" href="#requests_toolbelt.streaming_iterator.StreamingIterator" title="requests_toolbelt.streaming_iterator.StreamingIterator"><code class="xref py py-class docutils literal"><span class="pre">StreamingIterator</span></code></a> in
case requests can’t determize the filesize itself. This is the case with
streaming file objects like <code class="docutils literal"><span class="pre">stdin</span></code> or any sockets. Wrapping e.g. files
that are on disk with <code class="docutils literal"><span class="pre">StreamingIterator</span></code> is unnecessary, because
requests can determine the filesize itself.</p>
<p>Naturally, you should also set the <cite>Content-Type</cite> of your upload
appropriately because the toolbelt will not attempt to guess that for you.</p>
</dd></dl>
</div>
</div>
</div>
</div>
<footer>
<div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
<a href="user-agent.html" class="btn btn-neutral float-right" title="User-Agent Constructor" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a>
<a href="threading.html" class="btn btn-neutral float-left" title="Using requests with Threading" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a>
</div>
<hr/>
<div role="contentinfo">
<p>
© Copyright 2015, Ian Cordasco, Cory Benfield
</p>
</div>
Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script type="text/javascript">
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>