'Read time out when trying to list FTP server directory in Docker container on Linux

I have service that polls FTP server with LIST command. I use local passive mode, use FTPSClient with following setting:

ftpClient.setTrustManager(TrustManagerUtils.getAcceptAllTrustManager());
ftpClient.setDataTimeout(1800000); //30 minutes
ftpClient.setBufferSize(1281000000);
ftpClient.setControlKeepAliveTimeout(120);

FTP server directory has around 700,000 files in it, so it takes time for server to respond. On my local machine by launching raw Java application it takes about 5 minutes to retrieve list of files, in Docker container on Windows it also works without problem in the same time, FileZilla - no problems, but longer. However, when I run the container on Linux machine (Ubuntu 18.04.4 LTS) it cannot retrieve the data despite the fact that it can connect to the server. After specified time in timeout setting I receive:

Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) ~[?:?] at java.net.SocketInputStream.socketRead(SocketInputStream.java:115) ~[?:?] at java.net.SocketInputStream.read(SocketInputStream.java:168) ~[?:?] at java.net.SocketInputStream.read(SocketInputStream.java:140) ~[?:?] at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) ~[?:?] at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) ~[?:?] at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) ~[?:?] at java.io.InputStreamReader.read(InputStreamReader.java:185) ~[?:?] at java.io.BufferedReader.fill(BufferedReader.java:161) ~[?:?] at java.io.BufferedReader.readLine(BufferedReader.java:326) ~[?:?] at java.io.BufferedReader.readLine(BufferedReader.java:392) ~[?:?] at org.apache.commons.net.ftp.FTPFileEntryParserImpl.readNextEntry(FTPFileEntryParserImpl.java:53) ~[app.jar:?] at org.apache.commons.net.ftp.FTPListParseEngine.readStream(FTPListParseEngine.java:142) ~[app.jar:?] at org.apache.commons.net.ftp.FTPListParseEngine.readServerList(FTPListParseEngine.java:118) ~[app.jar:?] at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3450) ~[app.jar:?] at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3371) ~[app.jar:?] at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3308) ~[app.jar:?] at org.my.org.ftp.FtpOperator.listDirectory(FtpOperator.java:30) ~[app.jar:?]

I guess this is not problem with firewall, because I have similar service that polls external server and it works properly. Both services are using local passive mode of Apache FTP client library.

Docker-compose file:

version: '2'
services:
  ingest:
    image: pl/ingest
    ports:
      - "8038:8038"

    volumes:
      - ./logs:/logs
      - ./processedFiles:/processedFiles

networks:                                
  default:                               
    driver: bridge                       
    driver_opts:                         
      com.docker.network.driver.mtu: 1500
javadockerftpapache-commons-net


Solution 1:[1]

Changing network mode from bridge to host in docker-compose made container able to list files by FTPS/FTP:

version: '2'
services:
  ingest:
    image: pl/ingest
    ports:
      - "8038:8038"
    volumes:
      - ./logs:/logs
      - ./processedFiles:/processedFiles
    network_mode: "host"

Solution 2:[2]

Having the same problem y reached to this SuperUser answer:

... active mode ftp is not suited for containerized ftp clients, due to the server actively trying to initiate a data connection to the ftp client, which is not possible for a container.

There are container workarounds but the programmatic way of solving this may be simpler: just change to passive mode after connecting.

ftpClient.enterLocalPassiveMode()

And now it works for me.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Ferdynand Kiepski
Solution 2

Related Questions