Tensorflow: Passing CSV with 3D feature array
<p>My current text file that I intend to use for LSTM training in Tensorflow looks like this:</p>

<pre><code>&gt; 0.2, 4.3, 1.2
&gt; 1.1, 2.2, 3.1
&gt; 3.5, 4.1, 1.1, 4300
&gt; 
&gt; 1.2, 3.3, 1.2
&gt; 1.5, 2.4, 3.1
&gt; 3.5, 2.1, 1.1, 4400
&gt; 
&gt; ...
</code></pre>

<p>There are 3 sequences 3 features vectors with only 1 label for each sample. I formatted this text file so it can be consistent with the LSTM training as the latter requires a time-steps of the sequences or in general, LSTM training requires a 3D tensor (batch, num of time-steps, num of features).</p>

<p>My question: How should I use Numpy or <code>TensorFlow.TextReader</code> in order to reformat the 3x3 sequence vectors and the singleton Labels so it can become compatible with Tensorflow?</p>

<p>Edit: I saw many tutorials on reformatting text or CSV files that has vectors and labels but unfortunately they were for 1 to 1 relationships e.g.</p>

<pre><code>0.2, 4.3, 1.2, Class1
1.1, 2.2, 3.1, Class2
3.5, 4.1, 1.1, Class3
</code></pre>

<p>becomes:</p>

<pre><code>[0.2, 4.3, 1.2, Class1], [1.1, 2.2, 3.1, Class2], [3.5, 4.1, 1.1, Class3]
</code></pre>

<p>which clearly is readable by Numpy and can build vectors easily from it dedicated for simple Feed-Forward NN tasks. But this procedure doesn't actually build an LSTM friendly CSV.</p>

<p>EDIT: The TensorFlow tutorial on CSV formats, covers only 2D arrays as an example. The <code>features = col1, col2, col3</code> doesn't assume that there might be time-steps for each sequence array and hence my question.</p>

<p><strong>UPDATE:</strong> addition to the previos answer:</p>

<pre><code>df.stack().to_csv('d:/temp/1D.csv', index=False)
</code></pre>

<p>1D.csv:</p>

<pre><code>0.2
4.3
1.2
4300.0
1.1
2.2
3.1
4300.0
3.5
4.1
1.1
4300.0
1.2
3.3
1.2
4400.0
1.5
2.4
3.1
4400.0
3.5
2.1
1.1
4400.0
</code></pre>

<p><strong>OLD answer:</strong></p>

<p>Here is a Pandas solution.</p>

<p>Assume we have the following text file:</p>

<pre><code>0.2, 4.3, 1.2
1.1, 2.2, 3.1
3.5, 4.1, 1.1, 4300

1.2, 3.3, 1.2
1.5, 2.4, 3.1
3.5, 2.1, 1.1, 4400
</code></pre>

<p>Code:</p>

<pre><code>import pandas as pd

In [95]: fn = r'D:\temp\.data\data.txt'

In [96]: df = pd.read_csv(fn, sep=',', skipinitialspace=True, header=None, names=list('abcd'))

In [97]: df
Out[97]:
     a    b    c       d
0  0.2  4.3  1.2     NaN
1  1.1  2.2  3.1     NaN
2  3.5  4.1  1.1  4300.0
3  1.2  3.3  1.2     NaN
4  1.5  2.4  3.1     NaN
5  3.5  2.1  1.1  4400.0

In [98]: df.d = df.d.bfill()

In [99]: df
Out[99]:
     a    b    c       d
0  0.2  4.3  1.2  4300.0
1  1.1  2.2  3.1  4300.0
2  3.5  4.1  1.1  4300.0
3  1.2  3.3  1.2  4400.0
4  1.5  2.4  3.1  4400.0
5  3.5  2.1  1.1  4400.0
</code></pre>

<p>now you can save it back to CSV:</p>

<pre><code>df.to_csv('d:/temp/out.csv', index=False, header=None)
</code></pre>

<p>d:/temp/out.csv:</p>

<pre><code>0.2,4.3,1.2,4300.0
1.1,2.2,3.1,4300.0
3.5,4.1,1.1,4300.0
1.2,3.3,1.2,4400.0
1.5,2.4,3.1,4400.0
3.5,2.1,1.1,4400.0
</code></pre>

python arrays csv numpy tensorflow