Python: Pandas, dealing with spaced column names
<p>If I have multiple text files that I need to parse that look like so, but can vary in terms of column names, and the length of the hashtags above: <img src="https://s4.postimg.org/8p69ptj9p/feafdfdfdfdf.png" alt="txt.file"></p>

<p>How would I go about turning this into a pandas dataframe? I've tried using     <code>pd.read_table('file.txt', delim_whitespace = True, skiprows = 14)</code>, but it has all sorts of problems. My issues are... </p>

<p>All the text, asterisks, and pounds at the top needs to be ignored, but I can't just use skip rows because the size of all the junk up top can vary in length in another file. </p>

<p>The columns "stat (+/-)" and "syst (+/-)" are seen as 4 columns because of the whitespace.</p>

<p>The one pound sign is included in the column names, and I don't want that. I can't just assign the column names manually because they vary from text file to text file.</p>

<p>Any help is much obliged, I'm just not really sure where to go from after I read the file using pandas.</p>

<p>Consider reading in raw file, cleaning it line by line while writing to a new file using <code>csv</code> module. Regex is used to identify column headers using the <em>i</em> as match criteria. Below assumes more than one space separates columns:</p>

<pre><code>import os
import csv, re
import pandas as pd

rawfile = "path/To/RawText.txt"
tempfile = "path/To/TempText.txt"

with open(tempfile, 'w', newline='') as output_file:
    writer = csv.writer(output_file)    

    with open(rawfile, 'r') as data_file:
        for line in data_file:            
            if re.match('^.*i', line):                     # KEEP COLUMN HEADER ROW
                line = line.replace('\n', '')                
                row = line.split("  ")                
                writer.writerow(row)

            elif line.startswith('#') == False:            # REMOVE HASHTAG LINES
                line = line.replace('\n', '')
                row = line.split("  ")            
                writer.writerow(row)

df = pd.read_csv(tempfile)                                 # IMPORT TEMP  FILE
df.columns = [c.replace('# ', '') for c in df.columns]     # REMOVE '#' IN COL NAMES     

os.remove(tempfile)                                        # DELETE TEMP FILE
</code></pre>

python python-3.x pandas text