Is it a good idea to apply ML libraries on pandas data frame?
<p>i am building a cognitive miner AI Bot. where My Bot has two task , one is train and other is predict.i'm using some/few ML functionalities. so here i have lots of documents(~200,000 docs) which i'm training. and then in predicting for a query, i'm following some steps to find most accurate matched document(by looking score, confidence on each document) from training. and some known functions i'm using finding like TF-IDF,n-gram,cosine-similarity of each tokens available in asked query. for doing this i am using core python , python third-party libraries,NoSQL database for keeping training data. </p>

<p>NOTE: all performance improvement taken care using core python as much as possible. (please don't give suggestion to use Elastic Search or python whoosh because i just want to use my silly code for another decade.:) )</p>

<p>I'm facing Performance issue. like to do score it is taking 2-3 seconds time. which is not good.i want that result should come in some milliseconds.</p>

<p>SO my question to you, if i use pandas , and try to apply all above functionality to it, will it give better performance ? or numpy matrix calculation will give better performance ? </p>

<p>so here i don't think code required to be paste. i just need experienced peoples view on my problem. and of course keeping in mind solution should be scalable. </p>

<p>It probably won't make much of a difference either way, in terms of performance.</p>

<p>Pandas is extremely efficient for loading data and munging it (grouping it in different ways, pivoting, creating new columns from existing columns, and so forth). </p>

<p>Once your data is ready for passing to a machine learning algorithm (say, in <code>sklearn</code>), then, basically, <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.as_matrix.html" rel="nofollow"><code>pd.DataFrame.as_matrix()</code></a> can transform it into a numpy array, without fundamentally affecting overall performance. It's hard to conceive of any <code>sklearn</code> prediction/classification stage whose cost doesn't dominate this.</p>

<p>The <a href="https://pypi.python.org/pypi/sklearn-pandas" rel="nofollow"><code>sklearn-pandas</code> package</a> facilitates this even further.</p>

<p>If your performance isn't satisfactory at this point, the solution lies elsewhere.</p>

python pandas numpy artificial-intelligence