How to extract a single row from multiple CSV files to a new file
<p>I have hundreds of CSV files on my disk, and one file added daily and I want to extract one row from each of them and put them in a new file. Then I want to daily add values to that same file. CSV files looks like this:</p>

<pre><code>business_day,commodity,total,delivery,total_lots
.
.
20160831,CTC,,201710,10
20160831,CTC,,201711,10
20160831,CTC,,201712,10
20160831,CTC,Total,,385
20160831,HTC,,201701,30
20160831,HTC,,201702,30
.
.
</code></pre>

<p>I want to fetch the row that contains 'Total' from each file. The new file should look like:</p>

<pre><code>business_day,commodity,total,total_lots
20160831,CTC,Total,385
20160901,CTC,Total,555
.
.
</code></pre>

<p>The raw files on my disk are named '20160831_foo.CSV', '20160901_foo.CSV etc..</p>

<p>After Googling this I have yet not seen any examples on how to extract only one value from a CSV file. Any hints/help much appreciated. Happy to use pandas if that makes life easier. </p>

<p>I ended up with the following:</p>

<pre><code>import pandas as pd
import glob

list_ = []
filenames = glob.glob('c:\\Financial Data\\*_DAILY.csv')
for filename in filenames:
    df = pd.read_csv(filename, index_col = None, usecols = ['business_day', 'commodity', 'total', 'total_lots'], parse_dates = ['business_day'], infer_datetime_format = True)
    df = df[((df['commodity'] == 'CTC') &amp; (df['total'] == 'Total'))]
    list_.append(df)

df = pd.concat(list_, ignore_index = True)
df['total_lots'] = df['total_lots'].astype(int)
df = df.sort_values(['business_day'])
df = df.set_index('business_day')
</code></pre>

<p>Then I save it as my required file. </p>

python python-3.x csv pandas