'Broken pipe error when proxying long file upload request
Recently and overnight, Apache's error logs from a couple of nodejs servers have been filling with this:
[proxy:error] [pid xxx] (32)Broken pipe: [client xxx.xxx.xxx.xxx:59835] AH01084: pass request body failed to 127.0.0.1:9001 (127.0.0.1), referer: https://tubedu.org/videos/upload
And
[proxy_http:error] [pid xxx] [client xxx.xxx.xxx.xxx:59835] AH01097: pass request body failed to 127.0.0.1:9001 (127.0.0.1) from xxx.xxx.xxx.xxx (), referer: https://tubedu.org/videos/upload
It happens when users try to upload large files. When the transfer is done, users get a 502 bad gateway error. This happens randomly but the larger is the file, most likely it is to happen.
Thanks!
EDIT: It seems that the error happens 60 seconds before de connection is established. About the first error message, sometimes it shows that one and sometimes shows this one:
[proxy:error] [pid xxx] (104)Connection reset by peer: [client xxx.xxx.xxx.xxx:58520] AH01084: pass request body failed to 127.0.0.1:9000 (127.0.0.1), referer: https://video.hardlimit.com/videos/upload
I've tried increasing timeouts in apache2.conf like this:
Timeout 900
ProxyTimeout 900
KeepAliveTimeout 600
with no result.
Finally, I've added to my virtualhost, this:
Timeout 600
ProxyTimeout 600
and this:
ProxyPass / http://127.0.0.1:9000/ timeout=600
ProxyPassReverse / http://127.0.0.1:9000/ timeout=600
But the error always happens 60 seconds before the transfer begins.
EDIT 2: I've modified the value of /proc/sys/net/ipv4/tcp_fin_timeout to 120 (it was 60). But it didn't work.
EDIT 3: Apparently it is a new "feature" of NodeJS 12.19.0 witch kills the connection 60 seconds before it is established. I solved it by using Nginx. This web server make a new connection just when the file has been uploaded to its buffer. Then, the server transfers it thru NodeJS witch takes way less than 60 seconds.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|