Python - Gathering File Information Recursively Leads to Memory Error
<p>I am writing a script to recurse over all directories and subdirectories of a starting folder but I'm running into memory errors (the error is <code>MemoryError</code>). My guess would be that maybe my <code>data_dicts</code> list is getting too big but I'm not sure. Any advice would be appreciated.</p>

<pre><code>import os

# example data dictionary
data_dict = {
    'filename': 'data.csv',
    'folder':   'R:/',
    'size':     300000
}

def get_file_sizes_folder(data_dicts, starting_folder):
# Given a list of file information dictionaries and a folder, iterate over the files
# in the folder to get their information and append it to the list. 
# Also recurse through subdirectories
    for entry in os.scandir(starting_folder):
        if not entry.name.startswith('.'):
            if entry.is_file():
                size = entry.stat().st_size
                filename = entry.name
                folder = os.path.dirname(entry.path)
                temp_dict = {'filename': filename, 'size': size, 'folder': folder}
                data_dicts.append(temp_dict.copy())
            else:
                print(entry.path)
                data_dicts.extend(get_file_sizes_folder(data_dicts, entry.path))

    return data_dicts

d = get_file_sizes_folder([], 'R:/')    
</code></pre>

<p>You shouldn't perform a recursion at all. Use <a href="https://docs.python.org/2.7/library/os.html#os.walk" rel="nofollow"><code>os.walk</code></a></p>

<p>Example:</p>

<pre><code>def get_file_sizes_folder(starting_folder):
    data_dicts = list()
    for root, _, files in os.walk(starting_folder):
        data_dicts.extend({
            'filename': f, 
            'size': os.path.getsize(os.path.join(root, f)),
            'folder': root,
        } for f in files)

    return data_dicts

d = get_file_sizes_folder('R:/')
</code></pre>

python recursion