Pandas df.isnull().all() count across multiple files
<p>I have 2000 csv files in a data set with 88 columns each: </p>

<pre><code>      filenames = glob.glob('path\*.csv') 
       for f in filenames:            
          df = pd.read_csv(f, error_bad_lines = False)
          df = df.isnull().all()
</code></pre>

<p>This returns a series with the column title, and True if the entire column is missing. How can count the number of Trues(completely missing columns) across the entire dataset(2000 csv files) so I can express as a % how much data is missing on a per file basis?</p>

<p>If an entire column is missing per file, I would want to add 1 and keep a running total of that   </p>

<p>The way you've phrased it, you're getting the number of missing columns per dataset. </p>

<p>However, you can get the number of missing rows per column, you can modify that code and call this:</p>

<pre><code>df.isnull().sum()
</code></pre>

<p>which will yield a count of missing rows per column. Something like: </p>

<p><code>
column1                     0
column2                     0
column3                     171
column4                     798
column5                     0
dtype: int64</code></p>

<p>Simply calling <code>.sum()</code> again will give you the sum of missing observations. </p>

<p>The total number of cells in the data frame will be equal to the columns multiplied by rows, which you can calculate by calling this: </p>

<pre><code>df.shape[0]*df.shape[1]
</code></pre>

<p>Which means you can calculate the missing percent by calling this:</p>

<pre><code>total = df.shape[0]*df.shape[1]
missing = df.isnull().sum().sum()
percent = missing/float(total)
</code></pre>

<p>Just append those values to a list, so you can save them for reference later. Try something like this: </p>

<pre><code> misscount = [] 
 for f in filenames:            
      df = pd.read_csv(f, error_bad_lines = False)
      total = df.shape[0]*df.shape[1]
      missing = df.isnull().sum().sum()
      percent = missing/float(total)
      misscount.append(percent)
</code></pre>

<h3>EDIT:</h3>

<p>based on feedback in the comments:</p>

<p>"....I actually do want the number of columns missing per the entire dataset(2000 csv files)....So if an entire column is missing, I'd want to add that to a "missing variable" like yours and then divide by the length of the entire dataset(2000)."</p>

<p>So, in order to calculate the total number of columns for a given csv file, you can call this: </p>

<pre><code>total =len(df.columns)
</code></pre>

<p>In order to calculate the total number of missing columns per csv file, you can call this:</p>

<pre><code>missing = df.isnull().all().sum()
</code></pre>

<p>So the missing column percent per csv file can be calculated like this: 
    percent = missing/float(total)</p>

<p>But it sounds like you want a running tally. So let's use this loop:</p>

<pre><code>colcount = 0
misscount = 0

for f in filenames:            
      df = pd.read_csv(f, error_bad_lines = False)
      colcount +=len(df.columns)
      misscount += df.isnull().all().sum()

percent = misscount/float(colcount)
</code></pre>

python pandas