How to parse file and make a dataframe
<p>I have a text file in a bespoke format. I also have a parser that extracts the relevant columns from each line.  I would like to read in the file one row at a time and add them to a dataframe.</p>

<p>The problem is that appending a row to a dataframe is slow.</p>

<p>What is the right way to do this?</p>

<p>If the file is large, your code is probably bound by the time to read the file from disk, not the time to add rows to the dataframe, unless you have some concrete profiling data to suggest otherwise. For example on an SSD (read throughput ~ 400MB/s) a 4GB file will take 10 seconds just to load from disk. On an HDD (~40MB/s) this might go up to 100 seconds. If this is the case one way to speed up the loading is to convert the file to a format supported by pandas, compress it, and then use the runtime de-<code>compression</code> option of <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_table.html#pandas.read_table" rel="nofollow">read_table()</a> to decompress it on the fly while creating the dataframes.</p>

<p>Otherwise, if you are sure this is really not the issue, you have these options, if adding one by one is not OK and the file is too large to buffer in memory:</p>

<ol>
<li>Change your data format to match something supported in Pandas; e.g. a CSV file seems like a pretty standard option you could go for;</li>
<li>Use one of the other constructors listed below to create the dataframes in larger batches (e.g. of 1-10K rows), then merge them, which might be faster than adding entries one by one, while using a constant amount of memory; </li>
</ol>

<p><a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html" rel="nofollow">The other ways to construct a dataframe</a> from the data structure you extract the file into:</p>

<blockquote>
  <p><a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.from_records.html#pandas.DataFrame.from_records" rel="nofollow">DataFrame.from_records</a> constructor from tuples, also record
  arrays </p>
  
  <p><a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.from_dict.html#pandas.DataFrame.from_dict" rel="nofollow">DataFrame.from_dict</a> from dicts of Series, arrays, or dicts</p>
  
  <p><a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.from_items.html#pandas.DataFrame.from_items" rel="nofollow">DataFrame.from_items</a> from sequence of (key, value) pairs</p>
  
  <p><a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html#pandas.read_csv" rel="nofollow">pandas.read_csv</a>, <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_table.html#pandas.read_table" rel="nofollow">pandas.read_table</a>, <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_clipboard.html#pandas.read_clipboard" rel="nofollow">pandas.read_clipboard</a></p>
</blockquote>

python pandas