interactive conditional histogram bucket slicing data visualization
<p>I have a df that looks like: </p>

<pre><code>df.head()
Out[1]:
        A   B   C
city0   40  12  73
city1   65  56  10
city2   77  58  71
city3   89  53  49
city4   33  98  90
</code></pre>

<p>An example df can be created by the following code:</p>

<pre><code>df = pd.DataFrame(np.random.randint(100,size=(1000000,3)), columns=list('ABC'))

indx = ['city'+str(x) for x in range(0,1000000)]
df.index = indx
</code></pre>

<p>What I want to do is:</p>

<p>a) determine appropriate histogram bucket lengths for column A and assign each city to a bucket for column A</p>

<p>b) determine appropriate histogram bucket lengths for column B and assign each city to a bucket for column B</p>

<p>Maybe the resulting df looks like (or is there a better built in way in pandas?)</p>

<pre><code>    df.head()
    Out[1]:
            A   B   C  Abkt Bbkt
    city0   40  12  73  2  1
    city1   65  56  10  4  3
    city2   77  58  71  4  3
    city3   89  53  49  5  3
    city4   33  98  90  2  5
</code></pre>

<p>Where Abkt and Bbkt are histogram bucket identifiers:</p>

<pre><code>1-20 = 1
21-40 = 2
41-60 = 3
61-80 = 4
81-100 = 5
</code></pre>

<p>Ultimately, I want to better understand the behavior of each city with respect to columns A, B and C and be able to answer questions like:</p>

<p>a) What does the distribution of Column A (or B) look like - i.e. what buckets are most/least populated.</p>

<p>b) Conditional on a particular slice/bucket of Column A, what does the distribution of Column B look like - i.e. what buckets are most/least populated.</p>

<p>c) Conditional on a particular slice/bucket of Column A and B, what does the behavior of C look like.</p>

<p>Ideally, I want to be able to visualize the data (heat maps, region identifiers etc). I'm a relative pandas/python newbie and don't know what is possible to develop. </p>

<p>If the SO community can kindly provide code examples of how I can do what I want (or a better approach if there are better pandas/numpy/scipy built in methods) I would be grateful. </p>

<p>As well, any pointers to resources that can help me better summarize/slice/dice my data and be able to visualize at intermediate steps as I proceed with my analysis. </p>

<p><strong>UPDATE:</strong></p>

<p>I am following some of the suggestions in the comments. </p>

<p>I tried:</p>

<p>1) <code>df.hist()</code></p>

<pre><code>ValueError: The first argument of bincount must be non-negative
</code></pre>

<p>2) <code>df[['A']].hist(bins=10,range=(0,10))</code></p>

<pre><code>array([[&lt;matplotlib.axes._subplots.AxesSubplot object at 0x000000A2350615C0&gt;]], dtype=object)
</code></pre>

<p>Isn't <code>#2</code> suppose to show a plot? instead of producing an object that is not rendered? I am using <code>jupyter notebook</code>. </p>

<p>Is there something I need to turn-on / enable in <code>Jupyter Notebook</code> to render the histogram objects? </p>

<p><strong>UPDATE2:</strong></p>

<p>I solved the rendering problem by: <a href="http://stackoverflow.com/questions/10511024/in-ipython-notebook-pandas-is-not-displying-the-graph-i-try-to-plot">in Ipython notebook, Pandas is not displying the graph I try to plot.</a></p>

<p><strong>UPDATE3:</strong></p>

<p>As per suggestions from the comments, I started looking through <a href="http://pandas.pydata.org/pandas-docs/stable/visualization.html" rel="nofollow">pandas visualization</a>, <a href="http://bokeh.pydata.org/en/latest/" rel="nofollow">bokeh</a> and <a href="http://stanford.edu/~mwaskom/software/seaborn/" rel="nofollow">seaborn</a>. However, I'm not sure how I can create linkages between plots. </p>

<p>Lets say I have 10 variables. I want to explore them but since 10 is a large number to explore at once, lets say I want to explore 5 at any given time (r,s,t,u,v). </p>

<p>If I want an interactive hexbin with marginal distributions plot to examine the relationship between r &amp; s, how do I also see the distribution of t, u and v given interactive region selections/slices of r&amp;s (polygons).</p>

<p>I found hexbin with marginal distribution plot here <a href="http://stanford.edu/~mwaskom/software/seaborn/examples/hexbin_marginals.html" rel="nofollow">hexbin plot</a>: </p>

<p><strong>But:</strong></p>

<p>1) How to make this interactive (allow selections of polygons)</p>

<p>2) How to link region selections of r &amp; s to other plots, for example 3 histogram plots of t,u, and v (or any other type of plot).</p>

<p>This way, I can navigate through the data more rigorously and explore the relationships in depth.</p>

<p>In order to get the interaction effect you're looking for, you must bin all the columns you care about, together.</p>

<p>The cleanest way I can think of doing this is to <code>stack</code> into a single <code>series</code> then use <code>pd.cut</code></p>

<p>Considering your sample <code>df</code></p>

<p><a href="http://i.stack.imgur.com/8IjPO.png" rel="nofollow"><img src="http://i.stack.imgur.com/8IjPO.png" alt="enter image description here"></a></p>

<pre><code>df_ = pd.cut(df[['A', 'B']].stack(), 5, labels=list(range(5))).unstack()
df_.columns = df_.columns.to_series() + 'bkt'
pd.concat([df, df_], axis=1)
</code></pre>

<p><a href="http://i.stack.imgur.com/dlCEm.png" rel="nofollow"><img src="http://i.stack.imgur.com/dlCEm.png" alt="enter image description here"></a></p>

<hr>

<p>Let's build a better example and look at a visualization using <code>seaborn</code></p>

<pre><code>df = pd.DataFrame(dict(A=(np.random.randn(10000) * 100 + 20).astype(int),
                       B=(np.random.randn(10000) * 100 - 20).astype(int)))

import seaborn as sns

df.index = df.index.to_series().astype(str).radd('city')

df_ = pd.cut(df[['A', 'B']].stack(), 30, labels=list(range(30))).unstack()
df_.columns = df_.columns.to_series() + 'bkt'

sns.jointplot(x=df_.Abkt, y=df_.Bbkt, kind="scatter", color="k")
</code></pre>

<p><a href="http://i.stack.imgur.com/Q2XnW.png" rel="nofollow"><img src="http://i.stack.imgur.com/Q2XnW.png" alt="enter image description here"></a></p>

<hr>

<p>Or how about some data with some correlation</p>

<pre><code>mean, cov = [0, 1], [(1, .5), (.5, 1)]
data = np.random.multivariate_normal(mean, cov, 100000)
df = pd.DataFrame(data, columns=["A", "B"])

df.index = df.index.to_series().astype(str).radd('city')

df_ = pd.cut(df[['A', 'B']].stack(), 30, labels=list(range(30))).unstack()
df_.columns = df_.columns.to_series() + 'bkt'

sns.jointplot(x=df_.Abkt, y=df_.Bbkt, kind="scatter", color="k")
</code></pre>

<p><a href="http://i.stack.imgur.com/YMo9M.png" rel="nofollow"><img src="http://i.stack.imgur.com/YMo9M.png" alt="enter image description here"></a></p>

<hr>

<h3>Interactive <code>bokeh</code></h3>

<p>Without getting too complicated</p>

<pre><code>from bokeh.io import show, output_notebook, output_file

from bokeh.plotting import figure
from bokeh.layouts import row, column
from bokeh.models import ColumnDataSource, Select, CustomJS

output_notebook()

# generate random data
flips = np.random.choice((1, -1), (5, 5))
flips = np.tril(flips, -1) + np.triu(flips, 1) + np.eye(flips.shape[0])

half = np.ones((5, 5)) / 2
cov = (half + np.diag(np.diag(half))) * flips
mean = np.zeros(5)

data = np.random.multivariate_normal(mean, cov, 10000)
df = pd.DataFrame(data, columns=list('ABCDE'))

df.index = df.index.to_series().astype(str).radd('city')

# Stack and cut to get dependent relationships
b = 20
df_ = pd.cut(df.stack(), b, labels=list(range(b))).unstack()

# assign default columns x and y.  These will be the columns I set bokeh to read
df_[['x', 'y']] = df_.loc[:, ['A', 'B']]

source = ColumnDataSource(data=df_)

tools = 'box_select,pan,box_zoom,wheel_zoom,reset,resize,save'

p = figure(plot_width=600, plot_height=300)
p.circle('x', 'y', source=source, fill_color='olive', line_color='black', alpha=.5)

def gcb(like, n):
    code = """
    var data = source.get('data');
    var f = cb_obj.get('value');
    data['{0}{1}'] = data[f];
    source.trigger('change');
    """
    return CustomJS(args=dict(source=source), code=code.format(like, n))

xcb = CustomJS(
    args=dict(source=source),
    code="""
    var data = source.get('data');
    var colm = cb_obj.get('value');
    data['x'] = data[colm];
    source.trigger('change');
    """
)

ycb = CustomJS(
    args=dict(source=source),
    code="""
    var data = source.get('data');
    var colm = cb_obj.get('value');
    data['y'] = data[colm];
    source.trigger('change');
    """
)

options = list('ABCDE')
x_select = Select(options=options, callback=xcb, value='A')
y_select = Select(options=options, callback=ycb, value='B')


show(column(p, row(x_select, y_select)))
</code></pre>

<p><a href="http://i.stack.imgur.com/RTu1E.png" rel="nofollow"><img src="http://i.stack.imgur.com/RTu1E.png" alt="enter image description here"></a></p>

python pandas data-visualization seaborn bokeh