How can I get a similar summary of a Pandas dataframe as in R?
<p>Different scales allow different types of operations. I would like to specify the scale of a column in a dataframe <code>df</code>. Then, <code>df.describe()</code> should take this into account.</p>

<h2>Examples</h2>

<ul>
<li><strong>Nominal scale</strong>: A nominal scale only allows to check for equivalence. Examples for this are sex, names, city names. You can basically only count how often they appear and give the most common ones (the mode).</li>
<li><strong>Ordinal scale</strong>: You can order, but not say how far one is away from another. Cloth sizes are one example. You can calculate the median / min / max for this scale.</li>
<li><strong>Quantitative scales</strong>: You can calculate the mean, standard deviation, quantiles for those scales.</li>
</ul>

<h2>Code example</h2>

<pre><code>import pandas as pd
import pandas.rpy.common as rcom
df = rcom.load_data('mtcars')
print(df.describe())
</code></pre>

<p>gives</p>

<pre><code>             mpg        cyl        disp          hp       drat         wt  \
count  32.000000  32.000000   32.000000   32.000000  32.000000  32.000000   
mean   20.090625   6.187500  230.721875  146.687500   3.596563   3.217250   
std     6.026948   1.785922  123.938694   68.562868   0.534679   0.978457   
min    10.400000   4.000000   71.100000   52.000000   2.760000   1.513000   
25%    15.425000   4.000000  120.825000   96.500000   3.080000   2.581250   
50%    19.200000   6.000000  196.300000  123.000000   3.695000   3.325000   
75%    22.800000   8.000000  326.000000  180.000000   3.920000   3.610000   
max    33.900000   8.000000  472.000000  335.000000   4.930000   5.424000   

            qsec         vs         am       gear     carb  
count  32.000000  32.000000  32.000000  32.000000  32.0000  
mean   17.848750   0.437500   0.406250   3.687500   2.8125  
std     1.786943   0.504016   0.498991   0.737804   1.6152  
min    14.500000   0.000000   0.000000   3.000000   1.0000  
25%    16.892500   0.000000   0.000000   3.000000   2.0000  
50%    17.710000   0.000000   0.000000   4.000000   2.0000  
75%    18.900000   1.000000   1.000000   4.000000   4.0000  
max    22.900000   1.000000   1.000000   5.000000   8.0000  
</code></pre>

<p>This is not good as <code>vs</code> is a binary variable which indicates if the car has a v-engine or a straight engine (<a href="http://stackoverflow.com/a/18617340/562769">source</a>). Hence the feature is of nominal scale. Hence min / max / std / mean are not applicable. It should rather be counted how often 0 and 1 appear.</p>

<p>In R, you can do the following:</p>

<pre><code>mtcars$vs = factor(mtcars$vs, levels=c(0, 1), labels=c("straight engine", "V-Engine"))
mtcars$am = factor(mtcars$am, levels=c(0, 1), labels=c("Automatic", "Manual"))
mtcars$gear = factor(mtcars$gear)
mtcars$carb = factor(mtcars$carb)
summary(mtcars)
</code></pre>

<p>and get</p>

<pre><code>      mpg             cyl             disp             hp             drat      
 Min.   :10.40   Min.   :4.000   Min.   : 71.1   Min.   : 52.0   Min.   :2.760  
 1st Qu.:15.43   1st Qu.:4.000   1st Qu.:120.8   1st Qu.: 96.5   1st Qu.:3.080  
 Median :19.20   Median :6.000   Median :196.3   Median :123.0   Median :3.695  
 Mean   :20.09   Mean   :6.188   Mean   :230.7   Mean   :146.7   Mean   :3.597  
 3rd Qu.:22.80   3rd Qu.:8.000   3rd Qu.:326.0   3rd Qu.:180.0   3rd Qu.:3.920  
 Max.   :33.90   Max.   :8.000   Max.   :472.0   Max.   :335.0   Max.   :4.930  
       wt             qsec                     vs             am     gear   carb  
 Min.   :1.513   Min.   :14.50   straight engine:18   Automatic:19   3:15   1: 7  
 1st Qu.:2.581   1st Qu.:16.89   V-Engine       :14   Manual   :13   4:12   2:10  
 Median :3.325   Median :17.71                                       5: 5   3: 3  
 Mean   :3.217   Mean   :17.85                                              4:10  
 3rd Qu.:3.610   3rd Qu.:18.90                                              6: 1  
 Max.   :5.424   Max.   :22.90                                              8: 1  
</code></pre>

<p>Is something similar also possible with Pandas?</p>

<p>I tried</p>

<pre><code>df["vs"] = df["vs"].astype('category')
</code></pre>

<p>But this makes <code>"vs"</code> disappear from the description.</p>

<p>You could have something like:</p>

<pre><code>def summary_lookalike(frame):
     frame.index = frame.index.map(str)
     return (frame.astype(str).apply(lambda x: frame.index.values + ': ' + x))

def descriptive_stats(df1, df2):
    df2 = df2.apply(lambda x: x.value_counts())
    df1, df2 = summary_lookalike(df1), summary_lookalike(df2)
    df2 = df2[df2.apply(lambda x: ~x.str.contains('nan'))].fillna('')
    return pd.concat([df1.reset_index(drop=True), df2.reset_index(drop=True)], axis=1)
</code></pre>

<p>Choosing Ordinal and Quantitative scales of data:</p>

<pre><code>df_describe = df[df.columns.difference(['vs', 'am', 'gear', 'carb'])].describe() 
</code></pre>

<p>Choosing Nominal scales of data:</p>

<pre><code>df_nominal = df.drop(df_describe.columns, axis=1)
</code></pre>

<p>Performing label replacement:</p>

<pre><code>df = pd.DataFrame(descriptive_stats(df_describe, df_nominal))
df['vs'] = df['vs'].str.replace('0.0', 'straight engine').str.replace('1.0', 'V-Engine')
df['am'] = df['am'].str.replace('0.0', 'Automatic').str.replace('1.0', 'Manual')
</code></pre>

<p>Result:</p>

<pre><code>df

                  cyl                disp                 drat  \
0         count: 32.0         count: 32.0          count: 32.0   
1        mean: 6.1875    mean: 230.721875      mean: 3.5965625   
2  std: 1.78592164695  std: 123.938693831  std: 0.534678736071   
3            min: 4.0           min: 71.1            min: 2.76   
4            25%: 4.0        25%: 120.825            25%: 3.08   
5            50%: 6.0          50%: 196.3           50%: 3.695   
6            75%: 8.0          75%: 326.0            75%: 3.92   
7            max: 8.0          max: 472.0            max: 4.93   

                   hp                 mpg               qsec  \
0         count: 32.0         count: 32.0        count: 32.0   
1      mean: 146.6875     mean: 20.090625     mean: 17.84875   
2  std: 68.5628684893  std: 6.02694805209  std: 1.7869432361   
3           min: 52.0           min: 10.4          min: 14.5   
4           25%: 96.5         25%: 15.425       25%: 16.8925   
5          50%: 123.0           50%: 19.2         50%: 17.71   
6          75%: 180.0           75%: 22.8          75%: 18.9   
7          max: 335.0           max: 33.9          max: 22.9   

                   wt                     vs               am       gear  \
0         count: 32.0  straight engine: 18.0  Automatic: 19.0              
1       mean: 3.21725         V-Engine: 14.0     Manual: 13.0              
2  std: 0.97845744299                                                      
3          min: 1.513                                          3.0: 15.0   
4        25%: 2.58125                                          4.0: 12.0   
5          50%: 3.325                                           5.0: 5.0   
6           75%: 3.61                                                      
7          max: 5.424                                                      

        carb  
0             
1   1.0: 7.0  
2  2.0: 10.0  
3   3.0: 3.0  
4  4.0: 10.0  
5             
6   6.0: 1.0  
7   8.0: 1.0
</code></pre>

<p>Timing Constraints:</p>

<pre><code>%timeit descriptive_stats(df_describe, df_nominal)
100 loops, best of 3: 12.6 ms per loop
</code></pre>

python pandas dataframe