'appendFile nodejs not working properly in production

i'm doing an upload function with chunks, locally it's working properly, but when I go up to production, the file being fully grouped by appendFile is not complete.

the code works as follows, in the frontend I divide the file into 1mb pieces, name it with a function to be a unique name, send it to the backend and execute the appendFile on the file with that name

request.on("data", (part) => {
    chunk.push(part);
}).on("end", async () => {
    const firstChunk = chunkId === 0;
    const lastChunk = (chunkId) === (chunksQuantity) -1;
    const completedChunk = Buffer.concat(chunk);
    
    if (firstChunk && existsSync(`${__dirname}/../../uploads/videos/${fileName}`)) {
      unlinkSync(`${__dirname}/../../uploads/videos/${fileName}`);
    }

    appendFile(`${__dirname}/../../uploads/videos/${fileName}`, completedChunk, async (err)=> {
      if(err) throw err;
      if(lastChunk) {
        //I upload the video to s3 and follow our internal flow
      }
    });
  • I downloaded the file and compared the sizes between the local env that worked and the production one and it always seems that the production one doesn't work the appendFile of the last chunk, but it doesn't actually give an error in the appendFile function

I opened the files with a text editor to see the size of the writing and doing the math really missing 1 piece of 1mb approximately

node.jsfschunks


Solution 1:[1]

I didn't understand what was happening, in our production environment we use kubernetes and with that it balanced the requests because we were sending pieces of buffer of a maximum of 5mb, and when we mounted the file the pieces were in different pods.

We changed the way of the requests to transfer as streaming and everything went well.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Victor Ezequiel

Related Questions