Google Task Queue REST pull returning 500 occasionally
<p>I have Python a process leasing tasks from the <a href="https://cloud.google.com/appengine/docs/python/taskqueue/rest/" rel="nofollow">Google TaskQueue REST API</a> every second in the unlimited loop:</p>

<pre><code>credentials = GoogleCredentials.get_application_default()
task_api = googleapiclient.discovery.build('taskqueue', 'v1beta2', credentials=credentials)
while True:
    tasks = task_api.tasks().lease(...).execute()
    time.sleep(1)
</code></pre>

<p>The process sometimes run well for hours. But occasionally crashes often by one of HTTP error:</p>

<ul>
<li>500 Backend Error</li>
<li>503 Backend Error</li>
<li>500 An internal error happened in the backend</li>
</ul>

<p>The process is running on the Google Computing Engine server. It uses a service account key, specified by the GOOGLE_APPLICATION_CREDENTIALS env variable. Is this a Google Task Queue bug or do I miss something? E.g. do I need to reread the credentials before every lease request?</p>

<p>Since the @DalmTo has just answered in comments, I sum up his answers and add the Python solution.</p>

<p>The Google 5xx backed error is flood protection and Google recommends to implement <a href="https://developers.google.com/drive/v3/web/handle-errors#500_backend_error" rel="nofollow">exponential backoff</a>. Despite the link points to Google Drive API, the Google errors are system wide for all the APIs (GAE including). It rarely takes more then 6 retries for it to kick in and respond.</p>

<p>After digging the googleapiclient sources, I've found that the exponential backoff is already implemented in this library, so the solution is dead simple:</p>

<pre><code>tasks = task_api.tasks().lease(...).execute(num_retries=6)
</code></pre>

<p>The sources of the <code>http.py::_should_retry_response()</code> shows, that beside HTTP 5xx errors the request is repeated also when the JSON response contains <code>userRateLimitExceeded</code> or <code>rateLimitExceeded</code> error.</p>

python google-app-engine google-compute-engine task-queue