'Avoid timeout in Elasticsearch re-indexing in Java

Below code returned a timeout in client (Elasticsearch Client) when number of records are higher.

CompletableFuture<BulkByScrollResponse> future = new CompletableFuture<>();
client.reindexAsync(request, RequestOptions.DEFAULT, new ActionListener<BulkByScrollResponse>() {
@Override
public void onResponse(BulkByScrollResponse bulkByScrollResponse) {
    future.complete(bulkByScrollResponse);
}

@Override
public void onFailure(Exception e) {
    future.completeExceptionally(e);
}
});
BulkByScrollResponse response = future.get(10, TimeUnit.MINUTES); // client timeout occured before this timeout

Below is the client config.

connectTimeout: 60000
socketTimeout: 600000
maxRetryTimeoutMillis: 600000

Is there a way to wait indefinitely until the re-indexing complete?

javatimeoutreindexresthighlevelclient


Solution 1:[1]

submit the reindex request as a task:

TaskSubmissionResponse task = esClient.submitReindexTask(reindex, RequestOptions.DEFAULT);

acquire the task id:

TaskId taskId = new TaskId(task.getTask());

then check the task status periodically:

        GetTaskRequest taskQuery = new GetTaskRequest(taskId.getNodeId(), taskId.getId());
        GetTaskResponse taskStatus;
        do {
            Thread.sleep(TimeUnit.MINUTES.toMillis(1));
            taskStatus = esClient.tasks()
                    .get(taskQuery, RequestOptions.DEFAULT)
                    .orElseThrow(() -> new IllegalStateException("Reindex task not found. id=" + taskId));
        } while (!taskStatus.isCompleted());

Elasticsearch java api doc about task handling just sucks.

Ref

Solution 2:[2]

I don't think its a better choice to wait indefinitely to complete the re-indexing process and give very high value for timeout as this is not a proper fix and will cause more harm than good.

Instead you should examine the response, add more debugging logging to find the root-cause and address them. Also please have a look at my tips to improve re-indexing speed, which should fix some of your underlying issues.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 puppylpg
Solution 2 Amit

Related Questions