Initilize multiple dataframe columns with categorical labels
<p>Problem statement:</p>

<p>I need to load 1000's csv  files into a data frame. All files have the same columns. The values in each of the columns belong to a <strong>limited</strong> set of possible values in all cases (different per column). The length of the values lies in the 100's of chars. I do not know beforehand those values.</p>

<p>My approach has been to parse each of the files and convert into a dataframe, with categorical columns and store them in hdfs store. Later concat them all together into "in memory" dataframe.</p>

<p>As I cannot concatenate all this dataframes due to conflicting category values, I want to create an empty dataframe, with same columns and  all category values seen in the files I processed. </p>

<p>The empty categorized dataframe is my starting point to concatenate one after another.</p>

<pre><code>df=pd.DataFrame(columns=["A","B"], dtypes={"A":"category","B":"category"} categories={"A":["a","b","c"],"B":["A","B","C","D"]})

df.concat[df1,df2,df3,d4]
</code></pre>

<p>or so I wish....</p>

<p>would a different strategy work better ?</p>

<p>Something like </p>

<p>This is resolved in pandas v0.19.0, see <a href="https://github.com/pydata/pandas/issues/12699" rel="nofollow">issue in gihub</a> and <a href="http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#categorical-concatenation" rel="nofollow">in pandas docs v.1.19 dev</a>.</p>

<p>However, there is another  post with an excellent and detailed solution to this very same problem with a  pre v1.19 solution <a href="http://stackoverflow.com/questions/29709918/pandas-and-category-replacement">pandas with category replacement</a></p>

python pandas dataframe