'Unable to bring up prometheus using docker-compose file. Always throws the error /etc/prometheus/prometheus.yaml: no such file or directory
I have been trying to bring up a Prometheus container using docker-compose file. I have looked into the various solutions available online and none of them seem to work. Please go through my prometheus.yaml file and the docker-compose.yml file and let me know, what have I configured wrongly. My prometheus.yaml file is located at /root/prometheus/prometheus.yaml
Note: I'm trying to run the prometheus in the agent-mode and I'm running the docker-compose file from the /root path.
Error generated:
-bash-5.0# docker-compose up
[...]
prometheus | ts=2022-05-12T14:28:25.350Z caller=main.go:447 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yaml)" file=/etc/prometheus/prometheus.yaml err="open /etc/prometheus/prometheus.yaml: no such file or directory"
prometheus exited with code 2
docker-compose.yml:
version: '3'
volumes:
prometheus_data:
services:
prometheus:
image: prom/prometheus:v2.35.0
container_name: prometheus
volumes:
- ./prometheus:/etc/prometheus
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yaml'
- '--storage.agent.path=/prometheus'
- '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/consoles'
- '--web.enable-lifecycle'
- '--enable-feature=agent'
expose:
- 9090
ports:
- "9090:9090"
Update 1: Adding the directory tree structure below
-bash-5.0# pwd
/root
-bash-5.0# tree
.
|-- cadvisor
|-- docker-compose.yml
|-- docker-compose_1.yml
|-- prometheus
| |-- prometheus.yaml
| |-- prometheus.yml
| |-- prometheus_old.yaml
| `-- prometheus_old.yml
|-- prometheus.yaml
`-- prometheus.yml
1 directory, 9 files
-bash-5.0#
Update 2: I did some debugging and found out that the directory is being mounted, whereas the files are being mounted as directory.
Basically what I did was I made changes to the docker-compose.yml file as follows.
version: '3'
services:
prometheus:
image: prom/prometheus:v2.35.0
container_name: prometheus
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus-test/prometheus.yml
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
#- '--storage.agent.path=/prometheus'
- '--web.enable-lifecycle'
#- '--enable-feature=agent'
expose:
- 9090
ports:
- "9090:9090"
volumes:
prometheus_data:
In the above docker file I'm mounting my prometheus.yml file to a different location and let the prometheus configure with default config file present. Later I logged into the container and checked for the mounted files and this is what was seen.
-bash-5.0# docker container exec -it prometheus sh
/prometheus $ cd /etc
/etc $ ls -ltr
total 68
-rw-r--r-- 1 root root 18774 Feb 10 2019 services
-rw-r--r-- 1 root root 494 Aug 16 2019 nsswitch.conf
-rw-r--r-- 1 root root 118 Mar 22 21:07 localtime
-rw-r--r-- 1 root root 340 Apr 11 21:49 passwd
-rw-rw-r-- 1 root root 306 Apr 11 21:49 group
-rw------- 1 root root 136 Apr 13 00:25 shadow
drwxr-xr-x 6 root root 4096 Apr 13 00:25 network
drwxr-xr-x 3 root root 4096 Apr 15 10:54 ssl
lrwxrwxrwx 1 root root 12 May 13 10:54 mtab -> /proc/mounts
-rw-r--r-- 1 root root 174 May 13 10:54 hosts
-rw-r--r-- 1 root root 13 May 13 10:54 hostname
-rw-r--r-- 1 root root 38 May 13 10:54 resolv.conf
drwxr-xr-x 3 root root 4096 May 13 11:00 prometheus-test
drwxr-xr-x 1 nobody nobody 4096 May 13 11:00 prometheus
/etc $
/etc $ cd prometheus-test/
/etc/prometheus-test $ ls
prometheus.yml
/etc/prometheus-test $ ls -ltr
total 8
drwxr-xr-x 2 root root 4096 May 13 10:54 prometheus.yml
/etc/prometheus-test $
From the above we can observe that the prometheus.yml file is being mounted as a directory instead of a file. Can anyone please let me know about this.
OS: Ubuntu 20.04 This is a vm instance running on the ESXI server
-bash-5.0# docker version
Client: Docker Engine - Community
Version: 19.03.10
API version: 1.40
Go version: go1.13.10
Git commit: 9424aeaee9
Built: Thu May 28 22:16:52 2020
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 19.03.5
API version: 1.40 (minimum version 1.12)
Go version: go1.13.10
Git commit: 633a0ea838f10e000b7c6d6eed1623e6e988b5bb
Built: Sat May 9 16:43:52 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.3.2
GitCommit: ff48f57fc83a8c44cf4ad5d672424a98ba37ded6
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit:
-bash-5.0#
-bash-5.0#
-bash-5.0#
-bash-5.0# docker context ls
NAME DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT ORCHESTRATOR
default * Current DOCKER_HOST based configuration unix:///var/run/docker.sock swarm
-bash-5.0#
Solution 1:[1]
I think you may be over-complicating things. Given this docker-compose.yaml:
version: '3'
volumes:
prometheus_data:
services:
prometheus:
image: prom/prometheus:v2.35.0
container_name: prometheus
volumes:
- ./prometheus:/etc/prometheus
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.agent.path=/prometheus'
- '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/consoles'
- '--web.enable-lifecycle'
- '--enable-feature=agent'
And this directory structure:
$ tree .
.
??? docker-compose.yaml
??? prometheus
??? prometheus.yml
It Just Works:
$ docker-compose up
Starting prometheus ... done
Attaching to prometheus
prometheus | ts=2022-05-13T12:12:16.206Z caller=main.go:187 level=info msg="Experimental agent mode enabled."
prometheus | ts=2022-05-13T12:12:16.207Z caller=main.go:525 level=info msg="Starting Prometheus" version="(version=2.35.0, branch=HEAD, revision=6656cd29fe6ac92bab91ecec0fe162ef0f187654)"
prometheus | ts=2022-05-13T12:12:16.207Z caller=main.go:530 level=info build_context="(go=go1.18.1, user=root@cf6852b14d68, date=20220421-09:53:42)"
prometheus | ts=2022-05-13T12:12:16.207Z caller=main.go:531 level=info host_details="(Linux 5.17.5-100.fc34.x86_64 #1 SMP PREEMPT Thu Apr 28 16:02:54 UTC 2022 x86_64 02da15afa8e7 (none))"
prometheus | ts=2022-05-13T12:12:16.207Z caller=main.go:532 level=info fd_limits="(soft=1073741816, hard=1073741816)"
prometheus | ts=2022-05-13T12:12:16.207Z caller=main.go:533 level=info vm_limits="(soft=unlimited, hard=unlimited)"
prometheus | ts=2022-05-13T12:12:16.208Z caller=web.go:541 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
prometheus | ts=2022-05-13T12:12:16.208Z caller=main.go:1013 level=info msg="Starting WAL storage ..."
prometheus | ts=2022-05-13T12:12:16.212Z caller=db.go:332 level=info msg="replaying WAL, this may take a while" dir=/prometheus/wal
prometheus | ts=2022-05-13T12:12:16.213Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false
prometheus | ts=2022-05-13T12:12:16.214Z caller=db.go:383 level=info msg="WAL segment loaded" segment=0 maxSegment=1
prometheus | ts=2022-05-13T12:12:16.214Z caller=db.go:383 level=info msg="WAL segment loaded" segment=1 maxSegment=1
prometheus | ts=2022-05-13T12:12:16.215Z caller=main.go:1034 level=info fs_type=XFS_SUPER_MAGIC
prometheus | ts=2022-05-13T12:12:16.215Z caller=main.go:1037 level=info msg="Agent WAL storage started"
prometheus | ts=2022-05-13T12:12:16.215Z caller=main.go:1162 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
prometheus | ts=2022-05-13T12:12:16.216Z caller=dedupe.go:112 component=remote level=info remote_name=b4f547 url=http://10.120.23.224:9090/api/v1/write msg="Starting WAL watcher" queue=b4f547
prometheus | ts=2022-05-13T12:12:16.216Z caller=dedupe.go:112 component=remote level=info remote_name=b4f547 url=http://10.120.23.224:9090/api/v1/write msg="Starting scraped metadata watcher"
prometheus | ts=2022-05-13T12:12:16.216Z caller=main.go:1199 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=809.925µs db_storage=275ns remote_storage=282.833µs web_handler=394ns query_engine=377ns scrape=211.707µs scrape_sd=43.033µs notify=609ns notify_sd=911ns rules=143ns tracing=1.98µs
prometheus | ts=2022-05-13T12:12:16.216Z caller=main.go:930 level=info msg="Server is ready to receive web requests."
prometheus | ts=2022-05-13T12:12:16.216Z caller=dedupe.go:112 component=remote level=info remote_name=b4f547 url=http://10.120.23.224:9090/api/v1/write msg="Replaying WAL" queue=b4f547
prometheus | ts=2022-05-13T12:12:22.558Z caller=dedupe.go:112 component=remote level=info remote_name=b4f547 url=http://10.120.23.224:9090/api/v1/write msg="Done replaying WAL" duration=6.341973747s
If this fails, the first diagnostic step is probably to remove the command section from docker-compose.yaml and add an entrypoint like this:
version: '3'
volumes:
prometheus_data:
services:
prometheus:
image: prom/prometheus:v2.35.0
container_name: prometheus
volumes:
- ./prometheus:/etc/prometheus
- prometheus_data:/prometheus
entrypoint:
- sleep
- inf
This will come up and just run sleep, allowing you to docker-compose exec into the container and explore the filesystem. If you find that /etc/prometheus is empty, that suggests that the you're not running docker-compose on the same system as the docker daemon (so when it attempts to reference a host path, like ./prometheus, it doesn't find it).
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | larsks |