Dump database table or work remotely for analysis?
<p>I have a table of <strong>80 million</strong> rows and I was given the task to do some light analysis like finding patterns for fields, which fields are mutually exclusive etc.</p>

<p>My initial instinct was to dump the whole table into a CSV so I can work with Pandas or similar since I assumed it would be faster and easier to work with. While figuring out ways on how to get the whole table into a CSV, a colleague insisted that it is overkill and the conventional approach is to work directly with the Oracle database.</p>

<p>From my software background, my understanding has been that databases are more for keeping the state of big applications and less for a human to fiddle with. What is the common approach for analysis when having such big tables? What is faster? Personally I don't mind the time it takes to dump the database but more about the time it takes to get back feedback when doing the actual analysis.</p>

<p>Directly on the database with SQL is perfectly fine for any analysis <em>when you already know what you're looking for</em>.</p>

<p>When you don't know what you're looking for, and you want to do e.g. pattern recognition, the effort to dump and process in another tool is probably worth it.</p>

<p>Also consider the possibility to connect Pandas directly to your Oracle database (which allows you to skip dumping data), <a href="http://dominicgiles.com/blog/files/bbffdb638932620b3182980fbd0e3d5b-146.html" rel="nofollow">see here for an example</a>. </p>

python oracle pandas data-analysis