Difference iterating with tp_iternext or PyIter_Next
<p>If I write a C function that does something with an iterable then I create an Iterator first and then loop over it.</p>

<pre><code>iterator = PyObject_GetIter(sequence);
if (iterator == NULL) {
    return NULL;
}
while (( item = PyIter_Next(iterator) )) {
    ...
}
</code></pre>

<p>This works fine but I've also seen some functions using <code>tp_iternext</code>:</p>

<pre><code>iterator = PyObject_GetIter(sequence); // ....

iternext = *Py_TYPE(iterator)-&gt;tp_iternext;
while (( item = iternext(iterator) )) {
    ...
}
</code></pre>

<p>the second approach seems faster (I have only one data point: my Windows computer and my msvc compiler).</p>

<p>Is it just coincidence that the <code>iternext</code> approach is faster and is there any significant difference between these two?</p>

<p>Links to the python documentation of both:
<a href="https://docs.python.org/3/c-api/iter.html#c.PyIter_Next" rel="nofollow">PyIter_Next</a>, 
<a href="https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_iternext" rel="nofollow">tp_iternext</a>
I have read them but to me it's not clear when and why one should be preferred.</p>

<p>The <a href="https://hg.python.org/cpython/file/tip/Objects/abstract.c#l3145" rel="nofollow">source code for <code>PyIter_Next</code></a> shows that it simply retrieves the <code>tp_iternext</code> slot and calls it <strong>and clears a <code>StopIteration</code> exception that may or may not have occurred</strong>.</p>

<p>If you use <code>tp_iternext</code> explicitly you have to check for this <code>StopIteration</code> when exhausting the iterator.</p>

<hr>

<p>By the way: the documentation of <code>tp_iternext</code> also says:</p>

<blockquote>
  <p><strong><code>iternextfunc PyTypeObject.tp_iternext</code></strong></p>
  
  <p>An optional pointer to a function that returns the next item in an iterator. When the iterator is exhausted, it must return <code>NULL</code>; <strong>a
  <code>StopIteration</code> exception may or may not be set.</strong> When another error
  occurs, it must return <code>NULL</code> too. Its presence signals that the
  instances of this type are iterators.</p>
</blockquote>

<p>While there is no such mention in <code>PyIter_Next</code>'s documentation.</p>

<p>So <code>PyIter_Next</code> is the simple <em>and safe</em> way of iterating over an iterator. You can use <code>tp_iternext</code> but then you have to be careful to not trigger a <code>StopIteration</code> exception at the end.</p>

python python-c-api