Python Overflow Error: iter index too large
<p>I have a problem with big iterator in for loop in the code below. It generates floats by reading a string list containing numbers.</p>

<pre><code>def float_generator(tekstowe):
        x = ''
        for c in tekstowe:
            if c != ' ':
                x += c
            else:
                out = float(x)
                x = ''
                yield(out)
</code></pre>

<p>I'm getting a <strong>"OverflowError: iter index too large"</strong>. I try to use really big iter numbers (like billions of values in a searched file). Is iter range somehow limited in for loops? </p>

<p>Using Python 2.7 64 bit. Thanks.</p>

<p>Looks like <code>tekstowe</code> is a sequence type that only implements <code>__getitem__</code>, not <code>__iter__</code>, so it's using the Python iterator wrapper that calls <code>__getitem__</code> with 0, then 1, 2, 3, etc., until <code>__getitem__</code> raises <code>IndexError</code>.</p>

<p>As an implementation detail, <a href="https://hg.python.org/cpython/file/2.7/Objects/iterobject.c#l57" rel="nofollow">Python 2.7.11 and higher limits the value of the index passed by the iterator wrapper to <code>LONG_MAX</code></a> (before 2.7.11, it wasn't bounds checked but it still used a <code>long</code> for index storage, so it would wrap and start indexing with negative values). This doesn't matter on most non-Windows 64 bit builds, where <code>LONG_MAX</code> is <code>2**63 - 1</code> (larger than you'd likely encounter), but on Windows, C <code>long</code>s remain 32 bit quantities even on 64 bit builds, so <code>LONG_MAX</code> remains <code>2**31 - 1</code>, which is low enough to be reached in human timescales.</p>

<p>Your options are:</p>

<ol>
<li>Change the implementation of whatever class <code>tekstowe</code> is to give it a true <code>__iter__</code> method, so it doesn't get wrapped by the sequence iterator wrapper when you use it</li>
<li>Upgrade to Python 3.4+, ideally 3.5 (2.7.10/3.4.3 and below <a href="https://hg.python.org/cpython/file/v3.4.3/Objects/iterobject.c#l57" rel="nofollow">lacks the check for overflow entirely</a>, but this could mean wraparound causes infinite looping; <a href="https://hg.python.org/cpython/file/3.5/Objects/iterobject.c#l57" rel="nofollow">3.4.4/3.5.0 added the check, and they use a signed <code>size_t</code>, testing against <code>PY_SSIZE_T_MAX</code></a>, which means it will not error until the index reaches <code>2**63 - 1</code> on any 64 bit build, Windows or otherwise)</li>
</ol>

<p>The changes to add the overflow checks were made to resolve <a href="https://bugs.python.org/issue22939" rel="nofollow">Python bug #22939</a>; the type change (from <code>long</code> to <code>Py_ssize_t</code>) for the sequence iterator's index storage occurred in 3.4.0's release, resolving <a href="https://bugs.python.org/issue17932" rel="nofollow">Python bug #17932</a>.</p>

python loops iterator