Output something other than '0 pruned nodes'
<p>Every time I've used <code>xgboost</code> (not only with python), the training messages always include "0 pruned nodes" on each line.  For example:</p>

<pre><code>import pandas as pd
from sklearn import datasets
import xgboost as xgb
iris = datasets.load_iris()
dtrain = xgb.DMatrix(iris.data, label = iris.target)
params = {'max_depth': 10, 'min_child_weight': 0, 'gamma': 0, 'lambda': 0, 'alpha': 0}
bst = xgb.train(params, dtrain)
</code></pre>

<p>The output includes a long list of statements like</p>

<pre><code>[11:08:18] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 0 pruned nodes, max_depth=5
</code></pre>

<p>I've played with several combinations of tuning parameters but I always get this "0 pruned nodes" message.  How can I generate a situation where I get some pruned nodes?</p>

<p>You will have pruned nodes using <strong>regularization</strong>! Use the <code>gamma</code>parameter!</p>

<p>The objective functions contains two parts: training loss and regularization.
The regularisation in XGBoost is controlled by three parameters: <code>alpha</code>, <code>lambda</code> and <code>gamma</code> (<a href="https://github.com/dmlc/xgboost/blob/master/doc/parameter.md" rel="nofollow">doc</a>): </p>

<blockquote>
  <p>alpha [default=0] L1 regularization term on weights, increase this
  value will make model more conservative.</p>
  
  <p>lambda [default=1] L2 regularization term on weights, increase this
  value will make model more conservative.</p>
  
  <p>gamma [default=0] minimum loss reduction required to make a further
  partition on a leaf node of the tree. the larger, the more
  conservative the algorithm will be. range: [0,â]</p>
</blockquote>

<p><code>alpha</code> and <code>beta</code> are just L1 and L2 penalties on the weights and should not affect pruning.</p>

<p>BUT <code>gamma</code> is THE parameter to tune to get pruned nodes. You should increase it to get pruned nodes. Watch out that it is dependent of the objective function and that it could require value as high as 10000 or more to obtain pruned nodes. Tuning gamma is great! it will make XGBoost to converge! meaning that after a certain number of iterations the training and testing score will not change in the following iterations (all the nodes of the new trees will be pruned). At the end it is a great switch to control overfit!</p>

<p>See <a href="http://xgboost.readthedocs.io/en/latest/model.html" rel="nofollow">Introduction to Boosted Trees</a> to get the exact definition of <code>gamma</code>.</p>

python xgboost