Tensor too big for graph distribution - "InternalError: Message length was negative"
<p><strong>I tried to run the following graph:</strong>
<a href="http://i.stack.imgur.com/y5d5x.png" rel="nofollow"><img src="http://i.stack.imgur.com/y5d5x.png" alt="Graph that causes the error."></a></p>

<p><strong>Unfortunately, I receive the following error message:</strong></p>

<pre><code>tensorflow.python.framework.errors.InternalError: Message length was negativ
 [[Node: random_uniform_1_S1 = _Recv[client_terminated=false,
  recv_device= "/job:worker/replica:0/task:1/cpu:0",
  send_device="/job:worker/replica:0/task:0/cpu:0",
  send_device_incarnation=3959744268201087672,
  tensor_name="edge_18_random_uniform_1",
  tensor_type=DT_DOUBLE,
  _device="/job:worker/replica:0/task:1/cpu:0"]()]]
</code></pre>

<p>I noticed that this error message does not occur if the size of <code>random_uniform_1</code> is 800MB, but it does occur if the size is 8GB.</p>

<p>(Notice that <code>random_uniform_1</code> has to be transferred from one device to another device.)</p>

<p><strong>Question:</strong> Is there a limit on how big a tensor can be, if that tensor has to be transferred between devices?</p>

<p>Yes, currently there is a <strong>2GB limit</strong> on an individual tensor when sending it between processes. This limit is imposed by the protocol buffer representation (more precisely, by the auto-generated C++ wrappers produced by the <code>protoc</code> compiler) that is used in TensorFlow's communication layer.</p>

<p>We are investigating ways to lift this restriction. In the mean time, you can work around it by manually adding <code>tf.split()</code> or <code>tf.slice()</code>, and <code>tf.concat()</code> operations to partition the tensor for transfer. If you have very large <code>tf.Variable</code> objects, you can use <a href="https://www.tensorflow.org/versions/r0.10/api_docs/python/state_ops.html#variable-partitioners-for-sharding" rel="nofollow">variable partitioners</a> to perform this transformation automatically. Note that in your program you have multiple 8GB tensors in memory at once, so the peak memory utilization will be at least 16GB.</p>

python tensorflow distribution