'Using stream in mongodb is very slow

I am using the nodejs driver and the collection has ~45k documents, the query returns ~3k documents, but its almost taking 30seconds to iterate. The batch size is 1000. I have added time logs for each item, and between every 1000 elements there is a lag of 8-10s, how can I fix this?

const cursor = db.collection('item').find(query).batchSize(1000);

cursor.stream().on("data", doc => {
 console.timeLog("a");
});

.
a: 11288.358ms
a: 11288.377ms
a: 20865.905ms
a: 20866.217ms
.
.
a: 20920.683ms
a: 20920.697ms
a: 20920.713ms
a: 35908.047ms
a: 35908.316ms
.
.
node.jsmongodb


Solution 1:[1]

If you don't care about the order of your returned elements, you could try out the following sorting.

    const cursor = col
    .find({<addFilterIfNeeded>})
    .sort({ $natural: -1 })
    .stream()

With that small change applied I was able to significantly boost the read performance.

Greetings Thomas

Solution 2:[2]

According to over 40,000 developers, MongoDB is the most popular NOSQL database in use right now. The tool’s meteoric rise is likely due to its JSON structure which makes it easy for Javascript developers to use. From a developer perspective, MongoDB is a great solution for supporting modern data applications. Nevertheless, developers sometimes need to pull specific workflows out of MongoDB and integrate them into a secondary system while continuing to track any changes to the underlying MongoDB data.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Thomas Liebberger
Solution 2 Farihatul Maria

Related Questions