Dot product of patches in tensorflow
<p>I have two square matrices of the same size and the dimensions of a square patch. I'd like to compute the dot product between every pair of patches. Essentially I would like to implement the following operation:</p>

<pre><code>def patch_dot(A, B, patch_dim):
    res_dim = A.shape[0] - patch_dim + 1
    res = np.zeros([res_dim, res_dim, res_dim, res_dim])
    for i in xrange(res_dim):
        for j in xrange(res_dim):
            for k in xrange(res_dim):
                for l in xrange(res_dim):
                    res[i, j, k, l] = (A[i:i + patch_dim, j:j + patch_dim] *
                                       B[k:k + patch_dim, l:l + patch_dim]).sum()

    return res
</code></pre>

<p>Obviously this would be an extremely inefficient implementation. Tensorflow's tf.nn.conv2d seems like a natural solution to this as I'm essentially doing a convolution, however my filter matrix isn't fixed. Is there a natural solution to this in Tensorflow, or should I start looking at implementing my own tf-op?</p>

<p>The natural way to do this is to first extract overlapping image patches of matrix B using <a href="https://www.tensorflow.org/versions/r0.10/api_docs/python/array_ops.html#extract_image_patches" rel="nofollow">tf.extract_image_patches</a>, then to apply the <a href="https://www.tensorflow.org/versions/r0.8/api_docs/python/nn.html#conv2d" rel="nofollow">tf.nn.conv2D</a> function on A and each B sub-patch using <a href="https://www.tensorflow.org/versions/r0.8/api_docs/python/functional_ops.html" rel="nofollow">tf.map_fn</a>.</p>

<p>Note that prior to use <a href="https://www.tensorflow.org/versions/r0.10/api_docs/python/array_ops.html#extract_image_patches" rel="nofollow">tf.extract_image_patches</a> and <a href="https://www.tensorflow.org/versions/r0.8/api_docs/python/nn.html#conv2d" rel="nofollow">tf.nn.conv2D</a> you need to reshape your matrices as 4D tensors of shape <code>[1, width, height, 1]</code> using <a href="https://www.tensorflow.org/versions/r0.10/api_docs/python/array_ops.html#reshape" rel="nofollow">tf.reshape</a>.</p>

<p>Also, prior to use <a href="https://www.tensorflow.org/versions/r0.8/api_docs/python/functional_ops.html" rel="nofollow">tf.map_fn</a>, you would also need to use the <a href="https://www.tensorflow.org/versions/r0.8/api_docs/python/array_ops.html#transpose" rel="nofollow">tf.transpose</a> op so that the B sub-patches are indexed by the first dimension of the tensor you use as the <code>elems</code> argument of <a href="https://www.tensorflow.org/versions/r0.8/api_docs/python/functional_ops.html" rel="nofollow">tf.map_fn</a>.</p>

python machine-learning computer-vision tensorflow linear-algebra