AWS g2.8xlarge performance and out of memory issues when using tensorflow
<p>I am currently training a recurrent net on Tensorflow for a text classification problem and am running into performance and out of memory issues. I am on AWS g2.8xlarge with Ubuntu 14.04, and a recent nightly build of tensorflow (which I downloaded on Aug 25).</p>

<p>1) Performance issue:</p>

<p>On the surface, both the CPU and GPU are highly under-utilized. I've run multiple tests on this (and have used line_profiler and memory_profiler in the process). Train durations scale linearly with number of epochs, so I tested with 1 epoch. For RNN config = 1 layer, 20 nodes, training time = 146 seconds.</p>

<p>Incidentally, that number is about 20 seconds higher/slower than the same test run on a g2.2xlarge!</p>

<p>Here is a snapshot of System Monitor and nvidia-smi (updated every 2 seconds) about 20 seconds into the run:</p>

<p><a href="http://i.stack.imgur.com/znNZD.png" rel="nofollow">SnapshotEarlyPartOfRun</a></p>

<p>As you can see, GPU utilization is at 19%. When I use nvprof, I find that the total GPU process time is about 27 seconds or so. Also, except for one vCPU, all others are very under-utilized. The numbers stay around this level, till the end of the epoch where I measure error across the entire training set, sending GPU utilization up to 45%.</p>

<p>Unless I am missing something, on the surface, each device is sitting around waiting for something to happen.</p>

<p>2) Out of memory issue:</p>

<p>If I increase the number of nodes to 200, it gives me an Out of Memory error which happens on the GPU side. As you can see from the above snapshots, only one of the four GPUs is used. I've found that the way to get tensorflow to use the GPU has to do with how you assign the model. If you don't specify anything, tensorflow will assign it to a GPU. If you specify a GPU, only it will be used. Tensorflow does not like it when I assign it to multiple devices with a "for d in ['/gpu:0',...]". I get into an issue with re-using the embedding variable. I would like to use all 4 GPUs for this (without setting up distributed tensorflow). Here is the snapshot of the Out of memory error:</p>

<p><a href="http://i.stack.imgur.com/tdHsr.png" rel="nofollow">OutofMemoryError</a></p>

<p>Any suggestions you may have for both these problems would be greatly appreciated!</p>

<p>Re (1), to improve GPU utilization did you try increasing the batch size and / or shortening the sequences you use for training?</p>

<p>Re (2), to use multiple GPUs you do need to manually assign the ops to GPU devices, I believe. The right way is to place ops on specific GPUs by doing</p>

<pre><code>with g.Device("/gpu:0"):
  ...
with g.Device("/gpu:1"):
  ...
</code></pre>

python tensorflow deep-learning recurrent-neural-network