Process a LOT of data
<p>So I'm working with parametric energy simulations and ended up with 500GB+ of data stored in .CSV files. I need to be able to process all this data to compare the results and gain insights of the influence of different parameters. </p>

<p>Each csv file name contains information of the parameters used for the simulation so I can not merge the files. </p>

<p>I normally loaded the .csv files to python using pandas and defining a Class. but now (with all this data) there is not enough memory to do this. </p>

<p>Can you point me out a way to process this data? I need to be able to do plots and compare the csv files. </p>

<p>Thank you for your time. </p>

<p><a href="http://stackoverflow.com/questions/27203161/convert-large-csv-to-hdf5">Convert</a> the csv files to <a href="https://www.hdfgroup.org/hdf5/" rel="nofollow">hdf5</a>, which was created to deal with massive and complex datasets.  It works with <a href="http://pandas.pydata.org/pandas-docs/stable/io.html#io-hdf5" rel="nofollow">pandas</a> as well as <a href="http://www.pytables.org/FAQ.html" rel="nofollow">other</a> <a href="http://www.h5py.org/" rel="nofollow">libraries</a>.</p>

python csv dataframe bigdata