interactive conditional histogram bucket slicing data visualization
<p>I have a df that looks like: </p>

<pre><code>df.head()
Out[1]:
        A   B   C
city0   40  12  73
city1   65  56  10
city2   77  58  71
city3   89  53  49
city4   33  98  90
</code></pre>

<p>An example df can be created by the following code:</p>

<pre><code>df = pd.DataFrame(np.random.randint(100,size=(1000000,3)), columns=list('ABC'))

indx = ['city'+str(x) for x in range(0,1000000)]
df.index = indx
</code></pre>

<p>What I want to do is:</p>

<p>a) determine appropriate histogram bucket lengths for column A and assign each city to a bucket for column A</p>

<p>b) determine appropriate histogram bucket lengths for column B and assign each city to a bucket for column B</p>

<p>Maybe the resulting df looks like (or is there a better built in way in pandas?)</p>

<pre><code>    df.head()
    Out[1]:
            A   B   C  Abkt Bbkt
    city0   40  12  73  2  1
    city1   65  56  10  4  3
    city2   77  58  71  4  3
    city3   89  53  49  5  3
    city4   33  98  90  2  5
</code></pre>

<p>Where Abkt and Bbkt are histogram bucket identifiers:</p>

<pre><code>1-20 = 1
21-40 = 2
41-60 = 3
61-80 = 4
81-100 = 5
</code></pre>

<p>Ultimately, I want to better understand the behavior of each city with respect to columns A, B and C and be able to answer questions like:</p>

<p>a) What does the distribution of Column A (or B) look like - i.e. what buckets are most/least populated.</p>

<p>b) Conditional on a particular slice/bucket of Column A, what does the distribution of Column B look like - i.e. what buckets are most/least populated.</p>

<p>c) Conditional on a particular slice/bucket of Column A and B, what does the behavior of C look like.</p>

<p>Ideally, I want to be able to visualize the data (heat maps, region identifiers etc). I'm a relative pandas/python newbie and don't know what is possible to develop. </p>

<p>If the SO community can kindly provide code examples of how I can do what I want (or a better approach if there are better pandas/numpy/scipy built in methods) I would be grateful. </p>

<p>As well, any pointers to resources that can help me better summarize/slice/dice my data and be able to visualize at intermediate steps as I proceed with my analysis. </p>

<p><strong>UPDATE:</strong></p>

<p>I am following some of the suggestions in the comments. </p>

<p>I tried:</p>

<p>1) <code>df.hist()</code></p>

<pre><code>ValueError: The first argument of bincount must be non-negative
</code></pre>

<p>2) <code>df[['A']].hist(bins=10,range=(0,10))</code></p>

<pre><code>array([[&lt;matplotlib.axes._subplots.AxesSubplot object at 0x000000A2350615C0&gt;]], dtype=object)
</code></pre>

<p>Isn't <code>#2</code> suppose to show a plot? instead of producing an object that is not rendered? I am using <code>jupyter notebook</code>. </p>

<p>Is there something I need to turn-on / enable in <code>Jupyter Notebook</code> to render the histogram objects? </p>

<p><strong>UPDATE2:</strong></p>

<p>I solved the rendering problem by: <a href="http://stackoverflow.com/questions/10511024/in-ipython-notebook-pandas-is-not-displying-the-graph-i-try-to-plot">in Ipython notebook, Pandas is not displying the graph I try to plot.</a></p>

<p><strong>UPDATE3:</strong></p>

<p>As per suggestions from the comments, I started looking through <a href="http://pandas.pydata.org/pandas-docs/stable/visualization.html" rel="nofollow">pandas visualization</a>, <a href="http://bokeh.pydata.org/en/latest/" rel="nofollow">bokeh</a> and <a href="http://stanford.edu/~mwaskom/software/seaborn/" rel="nofollow">seaborn</a>. However, I'm not sure how I can create linkages between plots. </p>

<p>Lets say I have 10 variables. I want to explore them but since 10 is a large number to explore at once, lets say I want to explore 5 at any given time (r,s,t,u,v). </p>

<p>If I want an interactive hexbin with marginal distributions plot to examine the relationship between r &amp; s, how do I also see the distribution of t, u and v given interactive region selections/slices of r&amp;s (polygons).</p>

<p>I found hexbin with marginal distribution plot here <a href="http://stanford.edu/~mwaskom/software/seaborn/examples/hexbin_marginals.html" rel="nofollow">hexbin plot</a>: </p>

<p><strong>But:</strong></p>

<p>1) How to make this interactive (allow selections of polygons)</p>

<p>2) How to link region selections of r &amp; s to other plots, for example 3 histogram plots of t,u, and v (or any other type of plot).</p>

<p>This way, I can navigate through the data more rigorously and explore the relationships in depth.</p>

<p>Here is a new solution using <a href="http://bokeh.pydata.org/en/latest/docs/user_guide/charts.html" rel="nofollow"><code>bokeh</code></a> and <a href="http://holoviews.org/#installation" rel="nofollow"><code>HoloViews</code></a>. It should respond a little more to the interactive part.</p>

<p>I try to remember that <em>simple is beautiful</em> when it comes to dataviz. </p>

<p>I used <a href="https://pypi.python.org/pypi/fake-factory" rel="nofollow"><code>faker</code></a> library in order to generate random city names to make the following graphs more realistic.</p>

<p>I will let all my codes here even if the most important part is the choice of the libraries.</p>

<pre><code>import pandas as pd
import numpy as np
from faker import Faker

def generate_random_dataset(city_number, 
                        list_identifier,
                        labels,
                        bins,
                        city_location='en_US'):

    fake = Faker(locale=city_location)

    df = pd.DataFrame(data=np.random.uniform(0, 100, len(list_identifier)]), 
                      index=[fake.city() for _ in range(city_number)], 
                      columns=list_identifier)

    for name in list_identifier:
        df[name + 'bkt'] =  pd.Series(pd.cut(df[name], bins, labels=labels))

    return df

list_identifier=list('ABC')
labels = ['Low', 'Medium', 'Average', 'Good', 'Great']
bins = np.array([-1, 20, 40, 60, 80, 101])

df = generate_random_dataset(30, list_identifier, labels, bins)

df.head()
</code></pre>

<p>will output:
<a href="http://i.stack.imgur.com/cnnxh.png" rel="nofollow"><img src="http://i.stack.imgur.com/cnnxh.png" alt="df"></a></p>

<p>Sometimes, when your dataset is small, exposing a simple chart with colors could be enough. </p>

<pre><code>from bokeh.charts import Bar, output_file, show
from bokeh.layouts import column

bar = []
for name in list_identifier:
    bar.append(Bar(df, label='index', values=name, stack=name+'bkt',
               title="percentage of " + name, legend='top_left', plot_width=1024))

output_file('cities.html')

show(column(bar))
</code></pre>

<p>Will create a new html page (cities) containing the graphs. Note that all the graphs generated with <code>bokeh</code> are interactive.</p>

<p><a href="http://i.stack.imgur.com/ZrIBp.png" rel="nofollow"><img src="http://i.stack.imgur.com/ZrIBp.png" alt="graphA"></a></p>

<p><a href="http://i.stack.imgur.com/TEK9x.png" rel="nofollow"><img src="http://i.stack.imgur.com/TEK9x.png" alt="graphB"></a></p>

<p><code>bokeh</code> can't initially plot hexbin. However, <code>HoloViews</code> can. Thus, it allows to draw interactive plots whitin <code>ipython notebook</code>.</p>

<p>The syntax is quite straightforward, you just need a Matrix with two columns and call the hist method:</p>

<pre><code>import holoviews as hv
hv.notebook_extension('bokeh')

df = generate_random_dataset(1000, list_identifier, list(range(5)), 5)

points = hv.Points(np.column_stack((df.A, df.B)))
points.hist(num_bins=5, dimension=['x', 'y'])
</code></pre>

<p><a href="http://i.stack.imgur.com/2khIc.png" rel="nofollow"><img src="http://i.stack.imgur.com/2khIc.png" alt="repartition of A and B"></a></p>

<p>To compare with @piRSquared solution, I stole a bit of code (thank you btw :) to show the data with some correlation:</p>

<pre><code>mean, cov = [0, 1], [(1, .5), (.5, 1)]
data = np.random.multivariate_normal(mean, cov, 100000)
df = pd.DataFrame(data, columns=["A", "B"])

df.index = df.index.to_series().astype(str).radd('city')

df_ = pd.cut(df[['A', 'B']].stack(), 30, labels=list(range(30))).unstack()
df_.columns = df_.columns.to_series() + 'bkt'

points = hv.Points(np.column_stack((df_.Abkt, df_.Bbkt)))
points.hist(num_bins=5, dimension=['x', 'y'])
</code></pre>

<p><a href="http://i.stack.imgur.com/ZZxM5.png" rel="nofollow"><img src="http://i.stack.imgur.com/ZZxM5.png" alt="mean covariance"></a></p>

<p>Please consider visit <code>HoloViews</code> <a href="http://holoviews.org/Tutorials/Bokeh_Backend.html" rel="nofollow">tutorial</a>.</p>

python pandas data-visualization seaborn bokeh