google cloud machine learning hyperparameter tuning avoid Nans
<p>I am running google cloud machine learning beta - and use the hypertune setup with tensorflow.</p>

<p>In some of the sub runs of hyperparameter tuning I have losses becoming NaNs - and that crashes the computations - which in turns stop the hyperparameter tuning job. </p>

<pre><code>Error reported to Coordinator: &lt;class  'tensorflow.python.framework.errors.InvalidArgumentError'&gt;, 
Nan in summary histogram for: softmax_linear/HistogramSummary [[Node: softmax_linear/HistogramSummary = HistogramSummary[T=DT_FLOAT, 
_device="/job:master/replica:0/task:0/cpu:0"]
(softmax_linear/HistogramSummary/tag, softmax_linear/softmax_linear)]] 
Caused by op u'softmax_linear/HistogramSummary', defined at: File   
"/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main 
</code></pre>

<p>What is the canonical way of handling these ? Should I protect the loss function ? </p>

<p>Thanks</p>

<p>You should protect the loss function by checking for NaNs.  Any crash or exception thrown by the program is treated by Cloud ML as a failure of the trial, and if enough trials fail the entire job will be failed.</p>

<p>If the trial exits cleanly without setting any hyperparameter summaries, the trial will be considered Infeasible and hyperparameters similar to those will be less likely to be tried again, but it will not be an error.</p>

python machine-learning tensorflow google-cloud-ml