Sampling from a Computed Multivariate kernel density estimation
<p>Say I have X and Y coordinates on a map and a non-parametric distribution of "hot zones" (e.g. degree of pollution on a geographic map positioned at X and Y coordinates). My input data are heat maps.</p>

<p>I want to train a machine learning model that learns what a "hot zone" looks like, but I don't have a lot of labeled examples. All "hot zones" look pretty similar, but may be in different parts of my standardized XY coordinate map.</p>

<p>I can calculate a multivariate KDE and plot the density maps accordingly. To generate synthetic labeled data, can I "reverse" the KDE and randomly generate new image files with observations that fall within my KDE's "dense" range?</p>

<p>Is there any way to do this in python?</p>

<p>There are at least 3 high-quality kernel-density estimation implementations available for python:</p>

<ul>
<li><a href="http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gaussian_kde.html" rel="nofollow">scipy</a></li>
<li><a href="http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KernelDensity.html#sklearn.neighbors.KernelDensity" rel="nofollow">scikit-learn</a></li>
<li><a href="http://statsmodels.sourceforge.net/0.6.0/nonparametric.html" rel="nofollow">statsmodels</a></li>
</ul>

<p>My personal ranking is <strong>statsmodels > scikit-learn > scipy (best to worst)</strong> but it will depend on your use-case.</p>

<p>Some random-remarks:</p>

<ul>
<li>scikit-learn offers sampling from a fitted KDE for free (<code>kde.sample(N)</code>)</li>
<li>scikit-learn offers good cross-validation functions based on grid-search or random-search (cross-validation is highly recommended)</li>
<li>statsmodels offers cross-validation methods based on optimization (can be slow for big datasets; but very high accuracy)</li>
</ul>

<p>There are much more differences and some of these were analyzed in this very good <a href="https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/" rel="nofollow">blog post</a> by <em>Jake VanderPlas</em>. The following table is an excerpt from this post:</p>

<p><a href="http://i.stack.imgur.com/BXVWZ.png" rel="nofollow"><img src="http://i.stack.imgur.com/BXVWZ.png" alt="From: https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/ (Jake VanderPlas)"></a>
<sup><i> From: <a href="https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/" rel="nofollow">https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/</a> (Author: Jake VanderPlas) </i></sup></p>

<h3>Here is some example code using <strong>scikit-learn</strong>:</h3>

<pre><code>from sklearn.datasets import make_blobs
from sklearn.neighbors import KernelDensity
from sklearn.model_selection import GridSearchCV
import matplotlib.pyplot as plt
import numpy as np

# Create test-data
data_x, data_y = make_blobs(n_samples=100, n_features=2, centers=7, cluster_std=0.5, random_state=0)

# Fit KDE (cross-validation used!)
params = {'bandwidth': np.logspace(-1, 2, 30)}
grid = GridSearchCV(KernelDensity(), params)
grid.fit(data_x)
kde = grid.best_estimator_
bandwidth = grid.best_params_['bandwidth']

# Resample
N_POINTS_RESAMPLE = 1000
resampled = kde.sample(N_POINTS_RESAMPLE)

# Plot original data vs. resampled
fig, axs = plt.subplots(2, 2, sharex=True, sharey=True)

for i in range(100):
    axs[0,0].scatter(*data_x[i])
axs[0,1].hexbin(data_x[:, 0], data_x[:, 1], gridsize=20)

for i in range(N_POINTS_RESAMPLE):
    axs[1,0].scatter(*resampled[i])
axs[1,1].hexbin(resampled[:, 0], resampled[:, 1], gridsize=20)

plt.show()
</code></pre>

<h3>Output:</h3>

<p><a href="http://i.imgur.com/5SAvrZl.png" rel="nofollow"><img src="http://i.imgur.com/5SAvrZl.png" alt="enter image description here"></a></p>

python machine-learning statistics kernel-density