Most efficient way to iterate through list of lists
<p>I'm currently collecting data from quandl and is saved as a list of lists. The list looks something like this (Price data):</p>

<pre><code>['2', 1L, datetime.date(1998, 1, 2), datetime.datetime(2016, 9, 26, 1, 35, 3, 830563), datetime.datetime(2016, 9, 26, 1, 35, 3, 830563), '82.1900', '83.6200', '81.7500', '83.5000', '28.5183', 1286500.0]
</code></pre>

<p>This is typically 1 of about 5000 lists, and every once in awhile Quandl will spit back some <code>NaN</code> values that don't like being saved into the database.</p>

<pre><code>['2', 1L, datetime.date(1998, 1, 2), datetime.datetime(2016, 9, 26, 1, 35, 3, 830563), datetime.datetime(2016, 9, 26, 1, 35, 3, 830563), 'nan', 'nan', 'nan', 'nan', 'nan', 0]
</code></pre>

<p>What would be the most efficient way of iterating through the list of lists to change 'nan' values into zeros?</p>

<p>I know I could do something like this, but it seems rather inefficient. This operation will need to be performed on 11 different values * 5000 different dates * 500 companies:</p>

<pre><code>def screen_data(data):
    new_data = []
    for d in data:
        new_list = []
        for x in d:
            new_value = x
            if math.isNan(x):
                new_value = 0
            new_list.append(new_value)

        new_data.append(new_list)
    return new_data
</code></pre>

<p>I would be interested in any solution that could reduce the time. I know DataFrames might work, but not sure how it would solve the NaN issue.</p>

<p>Or if there is a way to include NaN values in an SQLServer5.6 database along with floats, changing the database is also a viable option.</p>

<p>how about this</p>

<pre><code>import math

def clean_nan(data_list,value=0):
    for i,x in enumerate(data_list):
        if math.isnan(x):
            data_list[i] = value
    return data_list 
</code></pre>

<p>(the return is optional, as the modification was made in-place, but it is needed if used with <code>map</code> or similar, assuming of course that data_list is well a list or similar container)</p>

<p>depending on how you get your data and how you work with it will determined how to use it, for instance if you do something like this</p>

<pre><code>for data in (my database/Quandl/whatever):
    #do stuff with data
</code></pre>

<p>you can change it to</p>

<pre><code>for data in (my database/Quandl/whatever):
    clean_nan(data)
    #do stuff with data
</code></pre>

<p>or use <a href="https://docs.python.org/3/library/functions.html#map" rel="nofollow">map</a> or if you are in python 2 <a href="https://docs.python.org/2.7/library/itertools.html#itertools.imap" rel="nofollow">imap</a></p>

<pre><code>for data in map(clean_nan,(my database/Quandl/whatever)):
    #do stuff with data
</code></pre>

<p>that way you get to work with your data as soon as that arrive from the database/Quandl/whatever, granted if the place where you get the data also work as a generator, that is don't process the whole thing all at once, and if it does, procure to change it to a generator if possible. In either case with this you get to work with your data as soon as possible.</p>

python performance list mysql-python