Combinations of chars via a generator in Python
<p>I am trying to get all the combinations with replacement (in other words, the cartesian product) of strings with length(8) . The output is too big and I am having trouble storing it in memory all at once , so my process gets killed before it finishes.</p>

<p>This is the code I use which is way more faster than Python's stdlib itertools:</p>

<pre><code>import numpy as np

def cartesian(arrays, out=None):
"""Generate a cartesian product of input arrays.
Parameters
----------
arrays : list of array-like
    1-D arrays to form the cartesian product of.
out : ndarray
    Array to place the cartesian product in.
Returns
-------
out : ndarray
    2-D array of shape (M, len(arrays)) containing cartesian products
    formed of input arrays.
Examples
--------
&gt;&gt;&gt; cartesian(([1, 2, 3], [4, 5], [6, 7]))
array([[1, 4, 6],
       [1, 4, 7],
       [1, 5, 6],
       [1, 5, 7],
       [2, 4, 6],
       [2, 4, 7],
       [2, 5, 6],
       [2, 5, 7],
       [3, 4, 6],
       [3, 4, 7],
       [3, 5, 6],
       [3, 5, 7]])
"""
arrays = [np.asarray(x) for x in arrays]
shape = (len(x) for x in arrays)
dtype = arrays[0].dtype

ix = np.indices(shape)
ix = ix.reshape(len(arrays), -1).T

if out is None:
    out = np.empty_like(ix, dtype=dtype)

for n, arr in enumerate(arrays):
    out[:, n] = arrays[n][ix[:, n]]

return out
</code></pre>

<p>How would I make it return a generator out of the result , instead of storing everything to memory all at once ?</p>

<p>My impression from other questions is that <code>product</code> is the fastest way of iteratively generating cartesian combinations:</p>

<pre><code>In [494]: g=itertools.product([1,2,3],[4,5],[6,7])
In [495]: list(g)
Out[495]: 
[(1, 4, 6),
 (1, 4, 7),
 (1, 5, 6),
 (1, 5, 7),
 (2, 4, 6),
 (2, 4, 7),
 (2, 5, 6),
 (2, 5, 7),
 (3, 4, 6),
 (3, 4, 7),
 (3, 5, 6),
 (3, 5, 7)]
</code></pre>

<p>Your code is a mapping of <code>np.indices</code>, which is slower:</p>

<pre><code>In [499]: timeit np.indices((3,2,2)).reshape(3,-1).T
The slowest run took 11.08 times longer than the fastest. This could mean that an intermediate result is being cached.
10000 loops, best of 3: 61.6 Âµs per loop
In [500]: timeit list(itertools.product([1,2,3],[4,5],[6,7]))
100000 loops, best of 3: 3.51 Âµs per loop
</code></pre>

python python-2.7 numpy scikit-learn itertools