JS: How to manage a POST request queue

Question

I've created an API that allows you to make a POST request that doesn't return an immediate result to the client and needs to be processed. I would like to allow the client to send multiple POST requests which are queued and processed one at a time. Could you recommend a package that can help me in this task? Can you give me an example of:

• how to create a queue
• how to add a POST request to the queue

Answer 1

If your task is heavy and demands using Node.js worker threads, take a look at job queues modules, like Bull, BullMQ, Bree, and so on.

The first two, require Redis queue storage, the last one, is RAM only.

Otherwise, you could use MQ brokers, like Kafka or RabbitMQ, and work with them.

So your POST request could place the job in the queue, and execute, and your client, could request the result on another endpoint by unique (job) id, or any other params. Or receive notification via WebSocket. It's for you to decide.

Answer 2

Queue URLs for an actor to visit in its run. Learn how to share your queues between actor runs. Access and manage request queues from Apify Console or via API.

Request queues enable you to enqueue and retrieve requests such as URLs with an HTTP method and other parameters. They are useful not only in web crawling but anywhere you need to process a high number of URLs and enqueue new links.

Request queue storage supports both breadth-first and depth-first crawling orders, as well as custom data attributes. It allows you to query whether specific URLs were already found, push new URLs to the queue and fetch the next URLs to process.

Answer 3

You can make your POST requests promises and chain them into .then() calls. Here you can see an implementation for sequential promise resolving: Resolve promises one after another (i.e. in sequence)?

A more old-school approach would be to create a function, let's call it doPosts, which you would call like doPosts(requestData, index) and which would send a POST request to requestData[index] and in the callback it would call doPosts(requestData, index + 1).

You could also create a for loop, where you could await sequentially the POST request.

You can also set the third parameter of the open function of XmlHttpRequest:

    var client = new XMLHttpRequest();
    client.open("POST", "/log", false);

that false above means synchronous. So, you can loop your request data and send synchronous requests.