How one can run xgboost on hadoop cluster for distributed model training?
<p>I am trying to built a CTR prediction model using XGBoost on 100 million of impressions for contextual ads and in order to achieve the same, I want to try XGboost on hadoop as I have all of the impressions data available in HDFS.</p>

<p>Can someone cite a working tutorial for the same for python? </p>

<p>There are many ways to do it:</p>

<ol>
<li><p>If in case you have some lower level logical grouping say CTR for some item department and you want to make localized models for departments then you can go for map reduce type of setting. It will make sure all data belonging to single department will end up in single YARN container and you can build a model over that data. NLineInputFormat is a clever trick to make this map only process than map reduce based process which will give you significant speed boost up.</p></li>
<li><p>You can do distributed machine learning using Spark version of XGBoost for more refer <a href="http://dmlc.ml/2016/03/14/xgboost4j-portable-distributed-xgboost-in-spark-flink-and-dataflow.html" rel="nofollow">http://dmlc.ml/2016/03/14/xgboost4j-portable-distributed-xgboost-in-spark-flink-and-dataflow.html</a></p></li>
<li><p>If in case you are in process of deciding your infrastructure as well then give AWS also a try as explained here. Its not hadoop but indeed pseudo distributed machine learning: <a href="https://xgboost.readthedocs.io/en/latest/tutorials/aws_yarn.html" rel="nofollow">https://xgboost.readthedocs.io/en/latest/tutorials/aws_yarn.html</a></p></li>
</ol>

python hadoop machine-learning xgboost