Why does this piece of code gets slower with time?
<p>I'm trying to preprocess my images adding them to a 4D array. It starts off right but it gets slower with time, I thought this was due to my CPU but I tried running it on a GPU on the cloud and it still gets slower. Is this due to RAM? How can I optimize this to run faster?</p>

<pre><code>import tensorflow as tf
import os
import glob
import numpy as np
from PIL import Image
from random import randint

sess = tf.InteractiveSession()

def process_image(filename):
    im = Image.open(filename)
    array = np.array(im,dtype=np.uint8)

    #Resize and normalize
    resized = tf.image.resize_images(array, size[0], size[1], method = 0)
    normalized = tf.image.per_image_whitening(resized)
    result = sess.run(normalized)
    return result

counter_train = 0
counter_val = 0
for i, foldername in enumerate(foldernames):
    ind = 0
    index = randint(ind,ind+29)
    for j, filename in enumerate(glob.glob(foldername + '*.ppm')):
        print filename
        result = process_image(filename)

        if j == index:

            npX_val[counter_val]=result
            npClass_val[counter_val]=i
            ind += 30
            index = randint(ind,ind+29)
            counter_val += 1
        else:
            npX_train[counter_train]=result
            npClass_train[counter_train]=i
            counter_train += 1
print counter_val
print counter_train
</code></pre>

<p>I also ran pyinstrument and I get this</p>

<pre><code>3.160 &lt;module&gt;  process.py:1
ââ 2.763 &lt;module&gt;  tensorflow/__init__.py:19
   ââ 2.761 &lt;module&gt;  tensorflow/python/__init__.py:26
      ââ 2.144 &lt;module&gt;  tensorflow/contrib/__init__.py:15
      â  ââ 0.955 &lt;module&gt;  tensorflow/contrib/learn/__init__.py:65
      â  â  ââ 0.953 &lt;module&gt;  tensorflow/contrib/learn/python/__init__.py:16
      â  â     ââ 0.950 &lt;module&gt;  tensorflow/contrib/learn/python/learn/__init__.py:16
      â  â        ââ 0.889 &lt;module&gt;  tensorflow/contrib/learn/python/learn/estimators/__init__.py:16
      â  â        â  ââ 0.789 &lt;module&gt;  tensorflow/contrib/learn/python/learn/estimators/autoencoder.py:16
      â  â        â  â  ââ 0.770 &lt;module&gt;  tensorflow/contrib/learn/python/learn/estimators/base.py:16
      â  â        â  â     ââ 0.764 &lt;module&gt;  tensorflow/contrib/learn/python/learn/estimators/estimator.py:16
      â  â        â  â        ââ 0.729 &lt;module&gt;  tensorflow/contrib/learn/python/learn/learn_io/__init__.py:16
      â  â        â  â           ââ 0.724 &lt;module&gt;  tensorflow/contrib/learn/python/learn/learn_io/pandas_io.py:16
      â  â        â  â              ââ 0.724 &lt;module&gt;  pandas/__init__.py:5
      â  â        â  â                 ââ 0.307 &lt;module&gt;  pandas/core/api.py:5
      â  â        â  â                 â  ââ 0.283 &lt;module&gt;  pandas/core/groupby.py:1
      â  â        â  â                 â     ââ 0.268 &lt;module&gt;  pandas/core/frame.py:10
      â  â        â  â                 â        ââ 0.135 &lt;module&gt;  pandas/core/series.py:3
      â  â        â  â                 â        â  ââ 0.116 &lt;module&gt;  pandas/tools/plotting.py:3
      â  â        â  â                 â        â     ââ 0.112 &lt;module&gt;  pandas/tseries/converter.py:1
      â  â        â  â                 â        â        ââ 0.061 &lt;module&gt;  matplotlib/__init__.py:101
      â  â        â  â                 â        â        ââ 0.044 &lt;module&gt;  matplotlib/dates.py:111
      â  â        â  â                 â        ââ 0.102 &lt;module&gt;  pandas/core/generic.py:2
      â  â        â  â                 â           ââ 0.085 &lt;module&gt;  pandas/core/internals.py:1
      â  â        â  â                 â              ââ 0.075 &lt;module&gt;  pandas/sparse/array.py:3
      â  â        â  â                 â                 ââ 0.070 &lt;module&gt;  pandas/core/ops.py:5
      â  â        â  â                 â                    ââ 0.066 &lt;module&gt;  pandas/computation/__init__.py:2
      â  â        â  â                 â                       ââ 0.065 &lt;module&gt;  numexpr/__init__.py:22
      â  â        â  â                 ââ 0.123 &lt;module&gt;  pytz/__init__.py:9
      â  â        â  â                 â  ââ 0.110 &lt;module&gt;  pkg_resources/__init__.py:15
      â  â        â  â                 â     ââ 0.037 _call_aside  pkg_resources/__init__.py:2938
      â  â        â  â                 â     â  ââ 0.037 _initialize_master_working_set  pkg_resources/__init__.py:2953
      â  â        â  â                 â     ââ 0.036 load_module  pkg_resources/extern/__init__.py:34
      â  â        â  â                 ââ 0.114 &lt;module&gt;  pandas/core/config_init.py:11
      â  â        â  â                 â  ââ 0.083 &lt;module&gt;  pandas/formats/format.py:2
      â  â        â  â                 â     ââ 0.032 &lt;module&gt;  pandas/core/index.py:2
      â  â        â  â                 ââ 0.067 &lt;module&gt;  pandas/io/api.py:3
      â  â        â  ââ 0.053 &lt;module&gt;  tensorflow/contrib/learn/python/learn/estimators/linear.py:16
      â  â        â     ââ 0.051 &lt;module&gt;  tensorflow/contrib/linear_optimizer/__init__.py:20
      â  â        â        ââ 0.043 &lt;module&gt;  tensorflow/contrib/linear_optimizer/python/ops/sdca_ops.py:15
      â  â        ââ 0.033 &lt;module&gt;  tensorflow/contrib/learn/python/learn/dataframe/__init__.py:16
      â  ââ 0.711 &lt;module&gt;  tensorflow/contrib/distributions/__init__.py:73
      â  â  ââ 0.508 &lt;module&gt;  tensorflow/contrib/distributions/python/ops/chi2.py:15
      â  â  â  ââ 0.506 &lt;module&gt;  tensorflow/contrib/distributions/python/ops/gamma.py:15
      â  â  â     ââ 0.506 &lt;module&gt;  tensorflow/contrib/framework/__init__.py:58
      â  â  â        ââ 0.498 &lt;module&gt;  tensorflow/contrib/framework/python/ops/__init__.py:15
      â  â  â           ââ 0.489 &lt;module&gt;  tensorflow/contrib/framework/python/ops/embedding_ops.py:15
      â  â  â              ââ 0.487 &lt;module&gt;  tensorflow/contrib/layers/__init__.py:79
      â  â  â                 ââ 0.482 &lt;module&gt;  tensorflow/contrib/layers/python/layers/__init__.py:15
      â  â  â                    ââ 0.172 &lt;module&gt;  tensorflow/contrib/layers/python/layers/layers.py:17
      â  â  â                    â  ââ 0.160 &lt;module&gt;  tensorflow/python/ops/standard_ops.py:17
      â  â  â                    â     ââ 0.061 &lt;module&gt;  tensorflow/python/ops/gradients.py:15
      â  â  â                    ââ 0.131 &lt;module&gt;  tensorflow/contrib/layers/python/layers/optimizers.py:15
      â  â  â                    â  ââ 0.127 &lt;module&gt;  tensorflow/python/training/training.py:137
      â  â  â                    â     ââ 0.035 &lt;module&gt;  tensorflow/python/training/adadelta.py:16
      â  â  â                    â        ââ 0.035 &lt;module&gt;  tensorflow/python/training/training_ops.py:16
      â  â  â                    ââ 0.069 &lt;module&gt;  tensorflow/contrib/layers/python/layers/feature_column.py:68
      â  â  â                    ââ 0.053 &lt;module&gt;  tensorflow/contrib/layers/python/layers/embedding_ops.py:15
      â  â  â                    â  ââ 0.050 &lt;module&gt;  tensorflow/contrib/layers/python/ops/sparse_feature_cross_op.py:15
      â  â  â                    â     ââ 0.045 load_op_library  tensorflow/python/framework/load_library.py:40
      â  â  â                    ââ 0.048 &lt;module&gt;  tensorflow/contrib/layers/python/layers/target_column.py:16
      â  â  â                       ââ 0.046 &lt;module&gt;  tensorflow/contrib/metrics/__init__.py:135
      â  â  â                          ââ 0.039 &lt;module&gt;  tensorflow/contrib/metrics/python/ops/metric_ops.py:19
      â  â  â                             ââ 0.037 &lt;module&gt;  tensorflow/contrib/metrics/python/ops/set_ops.py:15
      â  â  â                                ââ 0.034 load_op_library  tensorflow/python/framework/load_library.py:40
      â  â  ââ 0.161 &lt;module&gt;  tensorflow/contrib/distributions/python/ops/bernoulli.py:15
      â  â     ââ 0.158 &lt;module&gt;  tensorflow/python/ops/nn.py:271
      â  â        ââ 0.087 &lt;module&gt;  tensorflow/python/ops/init_ops.py:16
      â  â           ââ 0.085 &lt;module&gt;  tensorflow/python/ops/nn_ops.py:15
      â  â              ââ 0.060 &lt;module&gt;  tensorflow/python/ops/gen_nn_ops.py:4
      â  â                 ââ 0.057 _InitOpDefLibrary  tensorflow/python/ops/gen_nn_ops.py:1630
      â  â                    ââ 0.054 Merge  google/protobuf/text_format.py:291
      â  â                       ââ 0.052 MergeLines  google/protobuf/text_format.py:331
      â  â                          ââ 0.052 _ParseOrMerge  google/protobuf/text_format.py:350
      â  â                             ââ 0.052 _MergeField  google/protobuf/text_format.py:374
      â  â                                ââ 0.052 _MergeField  google/protobuf/text_format.py:374
      â  â                                   ââ 0.038 _MergeField  google/protobuf/text_format.py:374
      â  ââ 0.265 &lt;module&gt;  tensorflow/contrib/bayesflow/__init__.py:18
      â  â  ââ 0.264 &lt;module&gt;  tensorflow/contrib/bayesflow/python/ops/stochastic_graph.py:38
      â  â     ââ 0.145 &lt;module&gt;  tensorflow/python/ops/array_ops.py:70
      â  â     â  ââ 0.068 &lt;module&gt;  tensorflow/python/ops/gen_math_ops.py:4
      â  â     â  â  ââ 0.065 _InitOpDefLibrary  tensorflow/python/ops/gen_math_ops.py:2378
      â  â     â  â     ââ 0.063 Merge  google/protobuf/text_format.py:291
      â  â     â  â        ââ 0.063 MergeLines  google/protobuf/text_format.py:331
      â  â     â  â           ââ 0.063 _ParseOrMerge  google/protobuf/text_format.py:350
      â  â     â  â              ââ 0.063 _MergeField  google/protobuf/text_format.py:374
      â  â     â  â                 ââ 0.059 _MergeField  google/protobuf/text_format.py:374
      â  â     â  â                    ââ 0.052 _MergeField  google/protobuf/text_format.py:374
      â  â     â  ââ 0.045 &lt;module&gt;  tensorflow/python/ops/gen_array_ops.py:4
      â  â     â     ââ 0.039 _InitOpDefLibrary  tensorflow/python/ops/gen_array_ops.py:2677
      â  â     â        ââ 0.038 Merge  google/protobuf/text_format.py:291
      â  â     â           ââ 0.038 MergeLines  google/protobuf/text_format.py:331
      â  â     â              ââ 0.038 _ParseOrMerge  google/protobuf/text_format.py:350
      â  â     â                 ââ 0.038 _MergeField  google/protobuf/text_format.py:374
      â  â     â                    ââ 0.035 _MergeField  google/protobuf/text_format.py:374
      â  â     ââ 0.085 &lt;module&gt;  tensorflow/python/ops/math_ops.py:210
      â  ââ 0.071 &lt;module&gt;  tensorflow/contrib/slim/__init__.py:18
      â  â  ââ 0.046 &lt;module&gt;  tensorflow/contrib/slim/python/slim/data/tfexample_decoder.py:20
      â  â     ââ 0.036 TFExampleDecoder  tensorflow/contrib/slim/python/slim/data/tfexample_decoder.py:273
      â  ââ 0.051 &lt;module&gt;  tensorflow/contrib/quantization/__init__.py:16
      â  â  ââ 0.050 &lt;module&gt;  tensorflow/contrib/quantization/python/__init__.py:15
      â  ââ 0.045 &lt;module&gt;  tensorflow/contrib/copy_graph/__init__.py:20
      â     ââ 0.043 &lt;module&gt;  tensorflow/contrib/copy_graph/python/util/copy_elements.py:27
      ââ 0.299 &lt;module&gt;  numpy/__init__.py:106
      â  ââ 0.235 &lt;module&gt;  numpy/add_newdocs.py:10
      â     ââ 0.230 &lt;module&gt;  numpy/lib/__init__.py:1
      â        ââ 0.160 &lt;module&gt;  numpy/lib/type_check.py:3
      â           ââ 0.158 &lt;module&gt;  numpy/core/__init__.py:1
      â              ââ 0.036 &lt;module&gt;  numpy/testing/__init__.py:7
      ââ 0.151 &lt;module&gt;  tensorflow/python/pywrap_tensorflow.py:11
      â  ââ 0.148 swig_import_helper  tensorflow/python/pywrap_tensorflow.py:13
      ââ 0.072 &lt;module&gt;  tensorflow/core/framework/graph_pb2.py:4
      ââ 0.039 &lt;module&gt;  tensorflow/python/platform/test.py:57
</code></pre>

<p>I don't know much about TensorFlow, but I believe the problem is <code>process_image</code> is using a bunch of globals, particularly <code>tf</code>. Every time it's called you're running TensorFlow on an ever increasing set of images. First there's <a href="https://en.wikipedia.org/wiki/1_%2B_2_%2B_3_%2B_4_%2B_%E2%8B%AF" rel="nofollow">1, then 2, then 3, then 4, 5, 6, ...</a></p>

<pre><code>1 + 2 + 3 + 4 + 5 + ... + n = n ( n + 1 ) / 2
</code></pre>

<p>So by 100 images you've actually processed 5,050. This is an O(n<sup>2</sup>) algorithm which means its runtime (and, in this case, memory) will grow exponentially as the number of images increases.</p>

<p>Again, I don't know much about TensorFlow, but perhaps leaving calling <code>sess.run</code> for the end makes more sense? Though you appear to be interested in the intermediate results?</p>

<p>And, as a very good rule of thumb, avoid globals. It's hard to tell them apart from local variables, they break the neat encapsulation of functions making the program hard to understand, and they and lead to accumulation problems like this.</p>

python performance numpy tensorflow