Use LeaveOneGroupOut strategy on cross_val_score in sklearn
<p>I'd like to use <code>LeaveOneGroupOut</code> strategy to evaluate my model. According to <a href="http://scikit-learn.org/stable/modules/cross_validation.html#computing-cross-validated-metrics" rel="nofollow">sklearn's document</a>, <code>cross_val_score</code> seems convenient. </p>

<p>However, the following code does not work.</p>

<pre><code>import sklearn
from sklearn import datasets
iris = datasets.load_iris()
from sklearn.model_selection import cross_val_score
clf = sklearn.svm.SVC(kernel='linear', C=1)
# cv = ShuffleSplit(n_splits=3, test_size=0.3, random_state=0) # =&gt; this works
cv = LeaveOneGroupOut # =&gt; this does not work
scores = cross_val_score(clf, iris.data, iris.target, cv=cv)
</code></pre>

<p>The error message is:</p>

<pre><code>ValueError                                Traceback (most recent call last)
&lt;ipython-input-40-435a3a7fa16c&gt; in &lt;module&gt;()
      4 from sklearn.model_selection import cross_val_score
      5 clf = sklearn.svm.SVC(kernel='linear', C=1)
----&gt; 6 scores = cross_val_score(clf, iris.data, iris.target, cv=LeaveOneGroupOut())
      7 scores

/Users/xxx/.pyenv/versions/anaconda-2.0.1/lib/python2.7/site-packages/sklearn/model_selection/_validation.pyc in cross_val_score(estimator, X, y, groups, scoring, cv, n_jobs, verbose, fit_params, pre_dispatch)
    138                                               train, test, verbose, None,
    139                                               fit_params)
--&gt; 140                       for train, test in cv.split(X, y, groups))
    141     return np.array(scores)[:, 0]
    142 

/Users/xxx/.pyenv/versions/anaconda-2.0.1/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in __call__(self, iterable)
    756             # was dispatched. In particular this covers the edge
    757             # case of Parallel used with an exhausted iterator.
--&gt; 758             while self.dispatch_one_batch(iterator):
    759                 self._iterating = True
    760             else:

/Users/xxx/.pyenv/versions/anaconda-2.0.1/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in dispatch_one_batch(self, iterator)
    601 
    602         with self._lock:
--&gt; 603             tasks = BatchedCalls(itertools.islice(iterator, batch_size))
    604             if len(tasks) == 0:
    605                 # No more tasks available in the iterator: tell caller to stop.

/Users/xxx/.pyenv/versions/anaconda-2.0.1/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in __init__(self, iterator_slice)
    125 
    126     def __init__(self, iterator_slice):
--&gt; 127         self.items = list(iterator_slice)
    128         self._size = len(self.items)
    129 

/Users/xxx/.pyenv/versions/anaconda-2.0.1/lib/python2.7/site-packages/sklearn/model_selection/_validation.pyc in &lt;genexpr&gt;(***failed resolving arguments***)
    135     parallel = Parallel(n_jobs=n_jobs, verbose=verbose,
    136                         pre_dispatch=pre_dispatch)
--&gt; 137     scores = parallel(delayed(_fit_and_score)(clone(estimator), X, y, scorer,
    138                                               train, test, verbose, None,
    139                                               fit_params)

/Users/xxx/.pyenv/versions/anaconda-2.0.1/lib/python2.7/site-packages/sklearn/model_selection/_split.pyc in split(self, X, y, groups)
     88         X, y, groups = indexable(X, y, groups)
     89         indices = np.arange(_num_samples(X))
---&gt; 90         for test_index in self._iter_test_masks(X, y, groups):
     91             train_index = indices[np.logical_not(test_index)]
     92             test_index = indices[test_index]

/Users/xxx/.pyenv/versions/anaconda-2.0.1/lib/python2.7/site-packages/sklearn/model_selection/_split.pyc in _iter_test_masks(self, X, y, groups)
    770     def _iter_test_masks(self, X, y, groups):
    771         if groups is None:
--&gt; 772             raise ValueError("The groups parameter should not be None")
    773         # We make a copy of groups to avoid side-effects during iteration
    774         groups = np.array(groups, copy=True)

ValueError: The groups parameter should not be None
scores  
</code></pre>

<p>You do not define your <strong>groups</strong> parameter which is the group according to which you are going to split your data.</p>

<p>The error comes from <strong>cross_val_score</strong> that takes this parameter in argument : in your case it is equal to <strong>None</strong>.</p>

<p>Try to follow the example below :</p>

<pre><code>from sklearn.model_selection import LeaveOneGroupOut
X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
y = np.array([1, 2, 1, 2])
groups = np.array([1, 1, 2, 2])
lol = LeaveOneGroupOut()
</code></pre>

<p>You have :</p>

<pre><code>[In] lol.get_n_splits(X, y, groups)
[Out] 2
</code></pre>

<p>Then you will be able to use :</p>

<pre><code>lol.split(X, y, groups)
</code></pre>

python machine-learning scikit-learn