netstat misses some ports [closed]

Question

$ nmap localhost

Starting Nmap 6.40 ( http://nmap.org ) at 2019-02-12 12:59 +00
Nmap scan report for localhost (127.0.0.1)
Host is up (0.0027s latency).
Other addresses for localhost (not scanned): 127.0.0.1
Not shown: 995 closed ports
PORT    STATE SERVICE
22/tcp  open  ssh
25/tcp  open  smtp
80/tcp  open  http
111/tcp open  rpcbind
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 0.23 seconds
$ sudo netstat -lnt 
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN     
tcp6       0      0 :::111                  :::*                    LISTEN     
tcp6       0      0 :::22                   :::*                    LISTEN     
tcp6       0      0 ::1:25                  :::*                    LISTEN     
$

Why are 80 and 443 not captured by netstat?

ss does not report the missing ports either. This is found on a centos 7 box. Both 80 and 443 are actually open and working as nmap found out -- curl from another host can pull stuff as expected.

The special thing is that 80 and 443 are opened by a docker container running on this host (the commands were run on the host, not in the container, just to be clear). The other 3 (22, 25, 111) are by non-docker local programs. I'm guessing docker is doing some voodoo but I have been unable to locate anything useful.

Answer 1

As of v1.7 docker has a configuration flag --userland-proxy which can be set to true or false (I believe this can be set to false be default these days). Basically what it does (being set to false) is instead of using a proxy process to get the ingressing traffic to the container, it utilizes the iptables rules to nat/forward the traffic (hairpin NAT) to the container.

See this article for more detailed explanation. From what I was able to gather, in most cases when the userland-proxy is disabled the port will still show up in the netstat, but this is only to allocate the port, so that the other host applications wouldn't be able to bind it, but the actual data plane follows the rules specified in the iptables. At the same time I've came across a bug when it wasn't the case and the port didn't show up in the output of the netstat/ss.

I believe this is what is happening in your case. You do not see the port in the output of the netstat, but the traffic still can get to the container because userland-proxy is disabled and iptables magic is used to get there.