Traefic docker container reverse-proxy redirect fails to ports provided by other containers: Gateway timeout

Question

Setup: I have a variety of native applications and docker applications on a nas device. (simplistic example).

host
: 8080 (console)
: 81 (apache)
: <port> and more (individual nas applications)
- container:traefik
  :80
- container:nginx
  :90
- container:customcode
  :4000
- and more (individual applications)
  :<port>

(host is 192.168.1.22).

**All containers and applications work and are reachable via 'http://192.168.1.22:<port>'

I was attempting to use traefik with simplistic names to manage the ports. i.e.

• 'apache' redirect to http://192.168.1.22:81 (host exposed) - works

• 'nas' redirect to http://192.168.1.22:8080 (host exposed) - works

• 'nginx' redirect to http://192.168.1.22:90 (container exposed) - fails

The traefik setup is able to redirect to all ports on the host itself, but none of the ports exposed by docker. This works for sites on different hosts as well. for ports exposed by containers I get a 'Gateway timeout' error

(only Log file entry: "'504 Gateway Timeout' caused by: dial tcp 192.168.1.22:90: i/o timeout").

I cannot use labels on the containers as they don't (and some cannot) share networks. I just want traefik to route to the IP:Port without worrying about if the port is provided by a container or not.

traefik.toml

loglevel = "ERROR"

[Log]
  filePath = "/etc/traefik/traefik.log"
  level = "DEBUG"  

[entryPoints]
  [entryPoints.http]
  address = ":80"
  [entryPoints.https]
  address = ":443"

[api]
  dashboard = true

[providers.docker]
  watch = false
  exposedByDefault = false
  endpoint = "unix:///var/run/docker.sock"  
  
[providers.file]
  watch = true
  filename = "/etc/traefik/services.toml"

services.toml

[http]
  [http.services]
    [http.services.nas]
      [http.services.nas.loadBalancer]
        [[http.services.nas.loadBalancer.servers]]
          url = "http://192.168.1.22:8080/"
    [http.services.test90]
      [http.services.test90.loadBalancer]
        [[http.services.test90.loadBalancer.servers]]
          url = "http://192.168.1.22:90/" #this does not work#
    [http.services.test81]
      [http.services.test81.loadBalancer]
        [[http.services.test81.loadBalancer.servers]]
          url = "http://192.168.1.22:81/"

docker compose:

version: "3.5"
services:
  traefik:
    image: "traefik:2.4"
    container_name: "traefik"
    restart: always
    environment:
      - PUID=<id>
      - PGID=<id>
    ports:
      - "443:443"
      - "80:80"
    volumes:
      - "/shr/traefik/:/etc/traefik/"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.rule=Host(`traefik`)"
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.routers.traefik.entrypoints=http,https"
      - "traefik.http.routers.traefik.middlewares=auth"
      - "traefik.http.middlewares.auth.basicauth.users=admin:<pass>"

      - "traefik.http.routers.nas.entrypoints=http"
      - "traefik.http.routers.nas.rule=Host(`nas`)"
      - "traefik.http.routers.nas.service=nas@file"

      - "traefik.http.routers.test81.entrypoints=http"
      - "traefik.http.routers.test81.rule=Host(`apache`)"
      - "traefik.http.routers.test81.service=test81@file"

      - "traefik.http.routers.test90.entrypoints=http"
      - "traefik.http.routers.test90.rule=Host(`nginx`)"
      - "traefik.http.routers.test90.service=test90@file"
    networks:
      - proxy

  whoami:
    image: "traefik/whoami"
    container_name: "whoami"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami`)"
      - "traefik.http.routers.whoami.entrypoints=http"

    networks:
      - proxy

networks:
  proxy:
    external:
      name: proxy

Answer 1

You might need to assing the correct traefik network in the application that causes the problem:

In the docker-compose.yml:

labels:
[...]
 - "traefik.enable=true"
 - "traefik.docker.network=foobar"
[...]

Where "foobar" is the docker network that traefik is also in. Ideally, an external docker network.

Answer 2

It seems that adding network_mode: "host" to the docker-compose and removing the custom network fixed the issue.