Tips for wrapping files faster
<p>In a piece of code, where I go through files that are formatted differently than what I want, I experience a huge slowdown.</p>

<p>The slowdown is in the order of 2 hours for processing 500 files that have approximately 1000 lines each. I need to do that many more times so any input on why it is so slow and how to make it faster is welcome. Here's my code:</p>

<p>f is an array I have with info about locating the files.</p>

<pre><code>def athwrap(athfpath, savefpath, fdirection):
    ath = []
    with open(athfpath, 'r') as file:
        for j, line in enumerate(file):
            if j == 6:
                npts = int(line[5:12])
                dt = float(line[17:25])
                ath = np.array((fdirection, npts, dt))
            elif j &gt; 6:
                uys = [float(n) for n in line.split()]
                ath = np.hstack((ath, uys))
                if not os.path.isdir(savefpath):
                    os.makedirs(savefpath, exist_ok=True)
                np.savetxt(savefpath + f[i, 15] + '.txt', ath, fmt='%s')
    file.close()

# Search and Wrap Files
k = 1
for i in range(0, len(f)):
    athpath = path + f[i, 16] + '/' + f[i, 15] + '.ATH'
    savepath = savpath + f[i, 16] + '/'
    if os.path.isfile(athpath):
        if k % 3 == 0:
            direction = "Y"
            athwrap(athpath, savepath, direction)
        elif k % 2 == 0:
            direction = "X"
            athwrap(athpath, savepath, direction)
        else:
            direction = "Z"
            athwrap(athpath, savepath, direction)
    else:
        print("Could find file " + loc + '/' + site + '/' + f[i, 16] + '/' + f[i, 15])
    k += 1
</code></pre>

<p>Indentation was a life saver!</p>

<pre><code>def athwrap(athfpath, savefpath, fdirection):
    ath = []
    with open(athfpath, 'r') as file:
        for j, line in enumerate(file):
            if j == 6:
                npts = int(line[5:12])
                dt = float(line[17:25])
                ath = np.array((fdirection, npts, dt))
            elif j &gt; 6:
                uys = [float(n) for n in line.split()]
                ath = np.hstack((ath, uys))
    if not os.path.isdir(savefpath):
        os.makedirs(savefpath, exist_ok=True)
    np.savetxt(savefpath + f[i, 15] + '.txt', ath, fmt='%s')
</code></pre>

python wrap enumerate