Using the same Embedding for Encoder and Decoder in seq2seq
<p>I am building an translation-machine based on the the seq2seq-class. The class assumes different vocabularies for the encoder and the decoder part. Thus also it expects different embeddings for the two.</p>

<p>However, I am trying to use this inside a single language. Thus I would like the two embeddings to be one. (Background is to translate laymen's terms to experts' terms, inside the same language)</p>

<p>Currently the relevant code is:</p>

<p>Encoder-Side:
in python/ops/rnn_cell.py in EmbeddingWrapper():</p>

<pre><code>with vs.variable_scope(scope or "EmbeddingWrapper"):
        additional_info_size   with vs.variable_scope(scope or type(self).__name__):
      with ops.device("/cpu:0"):
        embedding = vs.get_variable("embedding", [self._embedding_classes, self._embedding_size], initializer=initializer)
        embedded = embedding_ops.embedding_lookup(embedding, array_ops.reshape(inputs, [-1]))
</code></pre>

<p>Decoder-Side:
In python/ops/seq2seq.py in embedding_rnn_decoder():</p>

<pre><code>  with variable_scope.variable_scope(scope or "embedding_rnn_decoder"):
    with ops.device("/cpu:0"):
      embedding = variable_scope.get_variable("embedding", [num_symbols, embedding_size])
    loop_function = _extract_sksk_argmax_and_embed(
        embedding, output_projection,
        update_embedding_for_previous) if feed_previous else None
    emb_inp = (embedding_ops.embedding_lookup(embedding, i) for i in decoder_inputs)
</code></pre>

<p>Any idea how to elegantly get those two to use the same embedding-matrix?</p>

<p>You can use a reusing variable scope when you call the function which creates the second embedding. If you use a scope with the same name and set reuse=True the embedding will be reused. The documentation on <a href="https://www.tensorflow.org/versions/r0.10/how_tos/variable_scope/index.html" rel="nofollow">sharing variables</a> is relevant.</p>

python tensorflow