Google Cloud Tasks requeue task after 5 minutes ignoring greater dispatchDeadline setting

Question

Infrastructure: Firebase function that queues Google Cloud Task items and local NodeJS Express app that executes some long-running job. For debugging purposes I am using ngrok to tunnel task to my local app

GC Tasks docs state that the default timeout is 10 minutes which can be adjusted to anything inbetween 15 seconds and 30 minutes. But with jobs that are slightly longer than 5 minutes I am receiving another request although previous one have not yet finished.

In my nodejs app I am using @google-cloud/tasks package to create tasks. I have tried default setup (no dispatchDeadline) and custom one (see code below)

export const createQueueClient = (ref: string) => {
  const client = new google.CloudTasksClient();

  return {
    client,
    parent: client.queuePath(project, region, ref),
  };
};


export const postMessageToQueue = async <T>(
  queue: QueueNamesType,
  url: string,
  message: T,
  context: functions.https.CallableContext
) => {
  const { client, parent } = createQueueClient(queue);

  const [response] = await client.createTask({
    task: {
      dispatchDeadline: {
        seconds: 15 * 60, // <------------------- attempt to set a timeout value
      },
      httpRequest: {
        httpMethod: 'POST',
        url: url,
        body: Buffer.from(
          JSON.stringify({
            data: message,
          })
        ).toString('base64'),
        headers: {
          // forward auth header to next service
          Authorization: context.rawRequest.header('Authorization') || '',
          'Content-Type': 'application/json',
        },
      },
    },
    parent: parent,
  });

  return response;
};

ngrok:

nodejs app logs, you can clearly see that there is 5 minute difference between request:

[[11:46:31.084]] [LOG]    Received request
[[11:51:31.317]] [LOG]    Received request
[[11:51:34.006]] [LOG]    Finished request iCg7raEbrw6LlbBhjpBS

I enabled logging in panel and I am seeing 300s attempt duration and 'UNAVAILABLE' status

Time on screenshots/logs might be different due to retrying to collect more "evidences"

All jobs which are running for less than 5 minutes complete successfully and don't get requeued. Am I missing something?

Answer 1

The dispatchDeadline property worked for me. Timeout increased to 30 minute by following configuration

const {CloudTasksClient} = require('@google-cloud/tasks');
class MyService {
createMyTask(...): Promise<any> {
    return new Promise((resolve, reject) => {
        const client = new CloudTasksClient();
        const payload = {...}
        const parent = client.queuePath(GCP_PROJECT_ID, GCP_SERVICE_LOCATION, GCP_TASK_QUEUE_NAME);
        const url = TASK_URL;
        const task = {
            httpRequest: {
                httpMethod: 'POST',
                headers: {
                    'Content-Type': 'application/json',
                    'Authorization': authToken
                },
                url,
                body: Buffer.from(JSON.stringify(payload)).toString('base64'),
            },
            dispatchDeadline: { seconds: 1800 },
        };
        const request = {parent, task};
        client.createTask(request).then((result: Array<any>) => {
            resolve(`task created: ${result[0].name}`);
        }).catch((err: Error) => {
            reject(`Create task Error: ${err}`);
        });
    });
}

Along with dispatchDeadline, you also need to set retry parameters for Google Cloud Task.