Grafana service fails to start due to bad directory name after Quincy upgrade
Hi Upgraded from Pacific 16.2.5 to 17.2.6 on May 8th However, Grafana fails to start due to bad folder path :/tmp# journalctl -u ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201 -n 25 -- Logs begin at Sun 2023-05-14 20:05:52 UTC, end at Tue 2023-05-16 19:07:51 UTC. -- May 16 19:05:00 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:00 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:00 fl31ca104ja0201 bash[2575021]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:00 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:00 fl31ca104ja0201 bash[2575030]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:00 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:10 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:10 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:10 fl31ca104ja0201 bash[2575273]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:10 fl31ca104ja0201 bash[2575282]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:20 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:20 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:20 fl31ca104ja0201 bash[2575369]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:20 fl31ca104ja0201 bash[2575370]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:30 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Start request r> May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:30 fl31ca104ja0201 systemd[1]: Failed to start Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ESCOC 19:07:51 UTC. -- 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Main process exited, code=exited, status=127/n/a ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Failed with result 'exit-code'. -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Scheduled restart job, restart counter is at 3. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Main process exited, code=exited, status=127/n/a ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Failed with result 'exit-code'. -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Scheduled restart job, restart counter is at 4. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. Check if path exists: #<mailto:root@fl31ca104ja0201:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#> ls /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run ls: cannot access '/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run': No such file or directory Check if grafana.fl31ca104ja0201 directory exists: #<mailto:root@fl31ca104ja0201:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#> ls -l rt/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/unit.run total 36 drwxr-xr-x 4 167 167 4096 Apr 20 08:05 data drwxr-xr-x 3 167 167 4096 Apr 20 08:13 etc -rw------- 1 167 167 48 Apr 20 08:13 unit.created -rw------- 1 167 167 390 May 8 16:12 unit.stop -rw------- 1 167 167 390 May 8 16:12 unit.poststop -rw------- 1 167 167 365 May 8 16:12 unit.meta -rw------- 1 167 167 32 May 8 16:12 unit.image -rw------- 1 167 167 38 May 8 16:12 unit.configured -rw------- 1 167 167 1063 May 8 16:12 unit.run cat unit.run set -e # grafana.fl31ca104ja0201 ! /usr/bin/docker rm -f ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana.fl31ca104ja0201 2> /dev/null ! /usr/bin/docker rm -f ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 2> /dev/null /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --init --name ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 --user 472 -e CONTAINER_IMAGE=docker.io/grafana/grafana:6.7.4 -e NODE_NAME=fl31ca104ja0201 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/grafana.ini:/etc/grafana/grafana.ini:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/certs:/etc/grafana/certs:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/data/grafana.db:/var/lib/grafana/grafana.db:Z docker.io/grafana/grafana:6.7.4 Thank you, Anantha
you could check owner of /var/lib/ceph on host with grafana container running. If its owner is root, change to 167:167 recursively. Then systemctl daemon-reload and restart the service. Good luck. Ben Adiga, Anantha <anantha.adiga@intel.com> 于2023年5月17日周三 03:57写道：
Hi
Upgraded from Pacific 16.2.5 to 17.2.6 on May 8th
However, Grafana fails to start due to bad folder path :/tmp# journalctl -u ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201 -n 25 -- Logs begin at Sun 2023-05-14 20:05:52 UTC, end at Tue 2023-05-16 19:07:51 UTC. -- May 16 19:05:00 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:00 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:00 fl31ca104ja0201 bash[2575021]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:00 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:00 fl31ca104ja0201 bash[2575030]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:00 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:10 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:10 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:10 fl31ca104ja0201 bash[2575273]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:10 fl31ca104ja0201 bash[2575282]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:20 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:20 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:20 fl31ca104ja0201 bash[2575369]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:20 fl31ca104ja0201 bash[2575370]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:30 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Start request r> May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:30 fl31ca104ja0201 systemd[1]: Failed to start Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ESCOC 19:07:51 UTC. -- 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto: -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Main process exited, code=exited, status=127/n/a ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto: -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Failed with result 'exit-code'. -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto: -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Scheduled restart job, restart counter is at 3. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto: -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Main process exited, code=exited, status=127/n/a ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto: -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Failed with result 'exit-code'. -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto: -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Scheduled restart job, restart counter is at 4. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
Check if path exists: #<mailto:root@fl31ca104ja0201:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#> ls /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run ls: cannot access '/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run': No such file or directory
Check if grafana.fl31ca104ja0201 directory exists:
#<mailto:root@fl31ca104ja0201:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#> ls -l rt/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/unit.run total 36 drwxr-xr-x 4 167 167 4096 Apr 20 08:05 data drwxr-xr-x 3 167 167 4096 Apr 20 08:13 etc -rw------- 1 167 167 48 Apr 20 08:13 unit.created -rw------- 1 167 167 390 May 8 16:12 unit.stop -rw------- 1 167 167 390 May 8 16:12 unit.poststop -rw------- 1 167 167 365 May 8 16:12 unit.meta -rw------- 1 167 167 32 May 8 16:12 unit.image -rw------- 1 167 167 38 May 8 16:12 unit.configured -rw------- 1 167 167 1063 May 8 16:12 unit.run
cat unit.run set -e # grafana.fl31ca104ja0201 ! /usr/bin/docker rm -f ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana.fl31ca104ja0201 2> /dev/null ! /usr/bin/docker rm -f ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 2> /dev/null /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --init --name ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 --user 472 -e CONTAINER_IMAGE=docker.io/grafana/grafana:6.7.4 -e NODE_NAME=fl31ca104ja0201 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/grafana.ini:/etc/grafana/grafana.ini:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/certs:/etc/grafana/certs:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/data/grafana.db:/var/lib/grafana/grafana.db:Z docker.io/grafana/grafana:6.7.4
Thank you, Anantha _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ben, Thanks for the suggestion. Changed the user and group to 167 for all files in the data and etc folders in the grafana service folder were not 167. Did a systemctl daemon-reload and restarted the grafana service , but still seeing the same error -- Logs begin at Mon 2023-05-15 19:39:34 UTC, end at Wed 2023-05-17 17:08:02 UTC. -- May 17 17:07:44 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process exited, code=exite> May 17 17:07:44 fl31ca104ja0201 bash[148899]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No > May 17 17:07:44 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with result 'exit-code'. May 17 17:07:54 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled restart job, restart > May 17 17:07:54 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 17 17:07:54 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 17 17:07:54 fl31ca104ja0201 bash[149116]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such > May 17 17:07:54 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process exited, code=exite> May 17 17:07:54 fl31ca104ja0201 bash[149118]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No > May 17 17:07:54 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with result 'exit-code'. ESCOC 2 UTC. -- a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>: Main process exited, code=exited, status=127/n/a b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>: Failed with result 'exit-code'. a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>: Scheduled restart job, restart counter is at 3. a0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. a0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>: Main process exited, code=exited, status=127/n/a b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>: Failed with result 'exit-code'. ~ Thank you, Anantha From: Ben <ruidong.gao@gmail.com> Sent: Wednesday, May 17, 2023 2:29 AM To: Adiga, Anantha <anantha.adiga@intel.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Grafana service fails to start due to bad directory name after Quincy upgrade you could check owner of /var/lib/ceph on host with grafana container running. If its owner is root, change to 167:167 recursively. Then systemctl daemon-reload and restart the service. Good luck. Ben Adiga, Anantha <anantha.adiga@intel.com<mailto:anantha.adiga@intel.com>> 于2023年5月17日周三 03:57写道： Hi Upgraded from Pacific 16.2.5 to 17.2.6 on May 8th However, Grafana fails to start due to bad folder path :/tmp# journalctl -u ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201 -n 25 -- Logs begin at Sun 2023-05-14 20:05:52 UTC, end at Tue 2023-05-16 19:07:51 UTC. -- May 16 19:05:00 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:00 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:00 fl31ca104ja0201 bash[2575021]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:00 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:00 fl31ca104ja0201 bash[2575030]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:00 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:10 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:10 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:10 fl31ca104ja0201 bash[2575273]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:10 fl31ca104ja0201 bash[2575282]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:20 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:20 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:20 fl31ca104ja0201 bash[2575369]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:20 fl31ca104ja0201 bash[2575370]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:30 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Start request r> May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:30 fl31ca104ja0201 systemd[1]: Failed to start Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ESCOC 19:07:51 UTC. -- 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Main process exited, code=exited, status=127/n/a ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Failed with result 'exit-code'. -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Scheduled restart job, restart counter is at 3. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Main process exited, code=exited, status=127/n/a ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Failed with result 'exit-code'. -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Scheduled restart job, restart counter is at 4. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. Check if path exists: #<mailto:root@fl31ca104ja0201<mailto:root@fl31ca104ja0201>:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#> ls /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run ls: cannot access '/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run': No such file or directory Check if grafana.fl31ca104ja0201 directory exists: #<mailto:root@fl31ca104ja0201<mailto:root@fl31ca104ja0201>:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#> ls -l rt/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/unit.run total 36 drwxr-xr-x 4 167 167 4096 Apr 20 08:05 data drwxr-xr-x 3 167 167 4096 Apr 20 08:13 etc -rw------- 1 167 167 48 Apr 20 08:13 unit.created -rw------- 1 167 167 390 May 8 16:12 unit.stop -rw------- 1 167 167 390 May 8 16:12 unit.poststop -rw------- 1 167 167 365 May 8 16:12 unit.meta -rw------- 1 167 167 32 May 8 16:12 unit.image -rw------- 1 167 167 38 May 8 16:12 unit.configured -rw------- 1 167 167 1063 May 8 16:12 unit.run cat unit.run set -e # grafana.fl31ca104ja0201 ! /usr/bin/docker rm -f ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana.fl31ca104ja0201 2> /dev/null ! /usr/bin/docker rm -f ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 2> /dev/null /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --init --name ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 --user 472 -e CONTAINER_IMAGE=docker.io/grafana/grafana:6.7.4<http://docker.io/grafana/grafana:6.7.4> -e NODE_NAME=fl31ca104ja0201 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/grafana.ini:/etc/grafana/grafana.ini:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/certs:/etc/grafana/certs:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/data/grafana.db:/var/lib/grafana/grafana.db:Z docker.io/grafana/grafana:6.7.4<http://docker.io/grafana/grafana:6.7.4> Thank you, Anantha _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
use this to get relevant long lines in log: journalctl -u ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201 | less -S it is '--user 472' by content of unit.run, not the default ceph user 167. Maybe set the directory owner to 472 could help. Hope it helps Ben Adiga, Anantha <anantha.adiga@intel.com> 于2023年5月18日周四 01:15写道：
Ben,
Thanks for the suggestion.
Changed the user and group to 167 for all files in the data and etc folders in the grafana service folder were not 167. Did a systemctl daemon-reload and restarted the grafana service ,
but still seeing the same error
-- Logs begin at Mon 2023-05-15 19:39:34 UTC, end at Wed 2023-05-17 17:08:02 UTC. --
May 17 17:07:44 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process exited, code=exite>
May 17 17:07:44 fl31ca104ja0201 bash[148899]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No >
May 17 17:07:44 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with result 'exit-code'.
May 17 17:07:54 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled restart job, restart >
May 17 17:07:54 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
May 17 17:07:54 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
May 17 17:07:54 fl31ca104ja0201 bash[149116]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such >
May 17 17:07:54 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process exited, code=exite>
May 17 17:07:54 fl31ca104ja0201 bash[149118]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No >
May 17 17:07:54 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with result 'exit-code'.
ESCOC
2 UTC. --
a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process exited, code=exited, status=127/n/a
b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory
a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with result 'exit-code'.
a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled restart job, restart counter is at 3.
a0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
a0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory
a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process exited, code=exited, status=127/n/a
b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory
a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with result 'exit-code'.
~
Thank you,
Anantha
*From:* Ben <ruidong.gao@gmail.com> *Sent:* Wednesday, May 17, 2023 2:29 AM *To:* Adiga, Anantha <anantha.adiga@intel.com> *Cc:* ceph-users@ceph.io *Subject:* Re: [ceph-users] Grafana service fails to start due to bad directory name after Quincy upgrade
you could check owner of /var/lib/ceph on host with grafana container running. If its owner is root, change to 167:167 recursively.
Then systemctl daemon-reload and restart the service. Good luck.
Ben
Adiga, Anantha <anantha.adiga@intel.com> 于2023年5月17日周三 03:57写道：
Hi
Upgraded from Pacific 16.2.5 to 17.2.6 on May 8th
However, Grafana fails to start due to bad folder path :/tmp# journalctl -u ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201 -n 25 -- Logs begin at Sun 2023-05-14 20:05:52 UTC, end at Tue 2023-05-16 19:07:51 UTC. -- May 16 19:05:00 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:00 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:00 fl31ca104ja0201 bash[2575021]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:00 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:00 fl31ca104ja0201 bash[2575030]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:00 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:10 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:10 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:10 fl31ca104ja0201 bash[2575273]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:10 fl31ca104ja0201 bash[2575282]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:20 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:20 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:20 fl31ca104ja0201 bash[2575369]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:20 fl31ca104ja0201 bash[2575370]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:30 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Start request r> May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:30 fl31ca104ja0201 systemd[1]: Failed to start Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ESCOC 19:07:51 UTC. -- 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory
-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Main process exited, code=exited, status=127/n/a ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory
-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Failed with result 'exit-code'.
-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Scheduled restart job, restart counter is at 3. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory
-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Main process exited, code=exited, status=127/n/a ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory
-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Failed with result 'exit-code'.
-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>: Scheduled restart job, restart counter is at 4. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
Check if path exists: #<mailto:root@fl31ca104ja0201:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#> ls /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run ls: cannot access '/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run': No such file or directory
Check if grafana.fl31ca104ja0201 directory exists:
#<mailto:root@fl31ca104ja0201:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#> ls -l rt/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/unit.run total 36 drwxr-xr-x 4 167 167 4096 Apr 20 08:05 data drwxr-xr-x 3 167 167 4096 Apr 20 08:13 etc -rw------- 1 167 167 48 Apr 20 08:13 unit.created -rw------- 1 167 167 390 May 8 16:12 unit.stop -rw------- 1 167 167 390 May 8 16:12 unit.poststop -rw------- 1 167 167 365 May 8 16:12 unit.meta -rw------- 1 167 167 32 May 8 16:12 unit.image -rw------- 1 167 167 38 May 8 16:12 unit.configured -rw------- 1 167 167 1063 May 8 16:12 unit.run
cat unit.run set -e # grafana.fl31ca104ja0201 ! /usr/bin/docker rm -f ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana.fl31ca104ja0201 2> /dev/null ! /usr/bin/docker rm -f ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 2> /dev/null /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --init --name ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 --user 472 -e CONTAINER_IMAGE=docker.io/grafana/grafana:6.7.4 -e NODE_NAME=fl31ca104ja0201 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/grafana.ini:/etc/grafana/grafana.ini:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/certs:/etc/grafana/certs:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/data/grafana.db:/var/lib/grafana/grafana.db:Z docker.io/grafana/grafana:6.7.4
Thank you, Anantha _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Ben, After chown tp 472, “systemctl daemon-reload” changes it back to 167. I also notice that these are still from docker.io while the rest are from quay /home/general# docker ps --no-trunc | grep docker 93b8c3aa33580fb6f4951849a6ff9c2e66270eb913b8579aca58371ef41f2d6c docker.io/grafana/grafana:6.7.4 "/run.sh" 10 days ago Up 10 days ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 df6b7368a54d0af7d2cdd45c0c9bad0999d58c144cb99927a3f76683652b00f2 docker.io/prom/alertmanager:v0.16.2 "/bin/alertmanager --cluster.listen-address=:9094 --web.listen-address=:9093 --cluster.peer=fl31ca104ja0201.deacluster.intel.com:9094 --config.file=/etc/alertmanager/alertmanager.yml" 10 days ago Up 10 days ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-alertmanager-fl31ca104ja0201 aa2055733fe8d426312af5572c94558e89e7cf350e7baba2c22eb6a0e20682fc docker.io/prom/prometheus:v2.7.2 "/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --web.listen-address=:9095 --storage.tsdb.retention.time=15d --storage.tsdb.retention.size=0 --web.external-url=http://fl31ca104ja0201:9095" 10 days ago Up 10 days ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-prometheus-fl31ca104ja0201 a9526f50dfacad47af298c0c1b2cf6cfd74b796b6df1945325529c79658d7356 docker.io/prom/node-exporter:v0.17.0 "/bin/node_exporter --no-collector.timex --web.listen-address=:9100 --path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs" 10 days ago Up 10 days ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-node-exporter-fl31ca104ja0201 440926ce479bdd114f43e3228cc8cbfe48b4e1a6c2c7fab58c4cd103bc0f3a0e docker.io/arcts/keepalived "./init.sh" 3 weeks ago Up 3 weeks ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-keepalived-rgw-default-default-fl31ca104ja0201-yiasjs 2813ca859a7ba0de7fcb6be74a00b9b11a23e79636c5f35fb2b6b4be31a29f89 docker.io/library/haproxy:2.3 "docker-entrypoint.sh haproxy -f /var/lib/haproxy/haproxy.cfg" 3 weeks ago Up 3 weeks ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-haproxy-rgw-default-default-fl31ca104ja0201-yvwsmz d68e2f68c45f2ea9a10267c8d964c2aaf026b4291918f4f3fb306da20a532db9 docker.io/arcts/keepalived "./init.sh" 3 weeks ago Up 3 weeks ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-keepalived-nfs-nfs-1-fl31ca104ja0201-dsynjg 40f3c0b7455f5540fdb4f428bef4e9032b0ff0f50d302352551abb208eff1f28 docker.io/library/haproxy:2.3 "docker-entrypoint.sh haproxy -f /var/lib/haproxy/haproxy.cfg" 3 weeks ago Up 3 weeks ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-haproxy-nfs-nfs-1-fl31ca104ja0201-zdbzvv From: Ben <ruidong.gao@gmail.com> Sent: Wednesday, May 17, 2023 6:32 PM To: Adiga, Anantha <anantha.adiga@intel.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Grafana service fails to start due to bad directory name after Quincy upgrade use this to get relevant long lines in log: journalctl -u ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201 | less -S it is '--user 472' by content of unit.run, not the default ceph user 167. Maybe set the directory owner to 472 could help. Hope it helps Ben Adiga, Anantha <anantha.adiga@intel.com<mailto:anantha.adiga@intel.com>> 于2023年5月18日周四 01:15写道： Ben, Thanks for the suggestion. Changed the user and group to 167 for all files in the data and etc folders in the grafana service folder were not 167. Did a systemctl daemon-reload and restarted the grafana service , but still seeing the same error -- Logs begin at Mon 2023-05-15 19:39:34 UTC, end at Wed 2023-05-17 17:08:02 UTC. -- May 17 17:07:44 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process exited, code=exite> May 17 17:07:44 fl31ca104ja0201 bash[148899]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No > May 17 17:07:44 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with result 'exit-code'. May 17 17:07:54 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled restart job, restart > May 17 17:07:54 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 17 17:07:54 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 17 17:07:54 fl31ca104ja0201 bash[149116]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such > May 17 17:07:54 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process exited, code=exite> May 17 17:07:54 fl31ca104ja0201 bash[149118]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No > May 17 17:07:54 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with result 'exit-code'. ESCOC 2 UTC. -- a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>: Main process exited, code=exited, status=127/n/a b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>: Failed with result 'exit-code'. a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>: Scheduled restart job, restart counter is at 3. a0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. a0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>: Main process exited, code=exited, status=127/n/a b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>: Failed with result 'exit-code'. ~ Thank you, Anantha From: Ben <ruidong.gao@gmail.com<mailto:ruidong.gao@gmail.com>> Sent: Wednesday, May 17, 2023 2:29 AM To: Adiga, Anantha <anantha.adiga@intel.com<mailto:anantha.adiga@intel.com>> Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io> Subject: Re: [ceph-users] Grafana service fails to start due to bad directory name after Quincy upgrade you could check owner of /var/lib/ceph on host with grafana container running. If its owner is root, change to 167:167 recursively. Then systemctl daemon-reload and restart the service. Good luck. Ben Adiga, Anantha <anantha.adiga@intel.com<mailto:anantha.adiga@intel.com>> 于2023年5月17日周三 03:57写道： Hi Upgraded from Pacific 16.2.5 to 17.2.6 on May 8th However, Grafana fails to start due to bad folder path :/tmp# journalctl -u ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201 -n 25 -- Logs begin at Sun 2023-05-14 20:05:52 UTC, end at Tue 2023-05-16 19:07:51 UTC. -- May 16 19:05:00 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:00 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:00 fl31ca104ja0201 bash[2575021]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:00 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:00 fl31ca104ja0201 bash[2575030]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:00 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:10 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:10 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:10 fl31ca104ja0201 bash[2575273]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:10 fl31ca104ja0201 bash[2575282]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:20 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:20 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:20 fl31ca104ja0201 bash[2575369]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:20 fl31ca104ja0201 bash[2575370]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:30 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Start request r> May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:30 fl31ca104ja0201 systemd[1]: Failed to start Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ESCOC 19:07:51 UTC. -- 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Main process exited, code=exited, status=127/n/a ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Failed with result 'exit-code'. -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Scheduled restart job, restart counter is at 3. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Main process exited, code=exited, status=127/n/a ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Failed with result 'exit-code'. -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Scheduled restart job, restart counter is at 4. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. Check if path exists: #<mailto:root@fl31ca104ja0201<mailto:root@fl31ca104ja0201>:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#> ls /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run ls: cannot access '/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run': No such file or directory Check if grafana.fl31ca104ja0201 directory exists: #<mailto:root@fl31ca104ja0201<mailto:root@fl31ca104ja0201>:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#> ls -l rt/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/unit.run total 36 drwxr-xr-x 4 167 167 4096 Apr 20 08:05 data drwxr-xr-x 3 167 167 4096 Apr 20 08:13 etc -rw------- 1 167 167 48 Apr 20 08:13 unit.created -rw------- 1 167 167 390 May 8 16:12 unit.stop -rw------- 1 167 167 390 May 8 16:12 unit.poststop -rw------- 1 167 167 365 May 8 16:12 unit.meta -rw------- 1 167 167 32 May 8 16:12 unit.image -rw------- 1 167 167 38 May 8 16:12 unit.configured -rw------- 1 167 167 1063 May 8 16:12 unit.run cat unit.run set -e # grafana.fl31ca104ja0201 ! /usr/bin/docker rm -f ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana.fl31ca104ja0201 2> /dev/null ! /usr/bin/docker rm -f ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 2> /dev/null /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --init --name ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 --user 472 -e CONTAINER_IMAGE=docker.io/grafana/grafana:6.7.4<http://docker.io/grafana/grafana:6.7.4> -e NODE_NAME=fl31ca104ja0201 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/grafana.ini:/etc/grafana/grafana.ini:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/certs:/etc/grafana/certs:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/data/grafana.db:/var/lib/grafana/grafana.db:Z docker.io/grafana/grafana:6.7.4<http://docker.io/grafana/grafana:6.7.4> Thank you, Anantha _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
Hi, there was a change introduced [1] for cephadm to use dashes for container names instead of dots. That still seems to be an issue somehow, in your case cephadm is complaining about the missing directory: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run when it actually should look for: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/unit.run Your unit.run file seems to reflect that correctly, I compared it to one of my virtual test clusters, besides the fact that I'm using podman the unit.run structure is similar. The only difference I can spot is that the service ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service is failing (containing dashes) while it probably should be ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana.fl31ca104ja0201.service. Could you try to start the latter service unit? You could also try to redeploy grafana if that doesn't work. Regards, Eugen [1] https://github.com/ceph/ceph/pull/42242 Zitat von "Adiga, Anantha" <anantha.adiga@intel.com>:
Hi Ben,
After chown tp 472, “systemctl daemon-reload” changes it back to 167.
I also notice that these are still from docker.io while the rest are from quay /home/general# docker ps --no-trunc | grep docker 93b8c3aa33580fb6f4951849a6ff9c2e66270eb913b8579aca58371ef41f2d6c docker.io/grafana/grafana:6.7.4 "/run.sh"
10 days ago Up 10 days ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 df6b7368a54d0af7d2cdd45c0c9bad0999d58c144cb99927a3f76683652b00f2 docker.io/prom/alertmanager:v0.16.2 "/bin/alertmanager --cluster.listen-address=:9094 --web.listen-address=:9093 --cluster.peer=fl31ca104ja0201.deacluster.intel.com:9094 --config.file=/etc/alertmanager/alertmanager.yml" 10 days ago Up 10 days ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-alertmanager-fl31ca104ja0201 aa2055733fe8d426312af5572c94558e89e7cf350e7baba2c22eb6a0e20682fc docker.io/prom/prometheus:v2.7.2 "/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --web.listen-address=:9095 --storage.tsdb.retention.time=15d --storage.tsdb.retention.size=0 --web.external-url=http://fl31ca104ja0201:9095" 10 days ago Up 10 days ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-prometheus-fl31ca104ja0201 a9526f50dfacad47af298c0c1b2cf6cfd74b796b6df1945325529c79658d7356 docker.io/prom/node-exporter:v0.17.0 "/bin/node_exporter --no-collector.timex --web.listen-address=:9100 --path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs"
10 days ago Up 10 days ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-node-exporter-fl31ca104ja0201 440926ce479bdd114f43e3228cc8cbfe48b4e1a6c2c7fab58c4cd103bc0f3a0e docker.io/arcts/keepalived "./init.sh"
3 weeks ago Up 3 weeks ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-keepalived-rgw-default-default-fl31ca104ja0201-yiasjs 2813ca859a7ba0de7fcb6be74a00b9b11a23e79636c5f35fb2b6b4be31a29f89 docker.io/library/haproxy:2.3 "docker-entrypoint.sh haproxy -f /var/lib/haproxy/haproxy.cfg"
3 weeks ago Up 3 weeks ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-haproxy-rgw-default-default-fl31ca104ja0201-yvwsmz d68e2f68c45f2ea9a10267c8d964c2aaf026b4291918f4f3fb306da20a532db9 docker.io/arcts/keepalived "./init.sh"
3 weeks ago Up 3 weeks ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-keepalived-nfs-nfs-1-fl31ca104ja0201-dsynjg 40f3c0b7455f5540fdb4f428bef4e9032b0ff0f50d302352551abb208eff1f28 docker.io/library/haproxy:2.3 "docker-entrypoint.sh haproxy -f /var/lib/haproxy/haproxy.cfg"
3 weeks ago Up 3 weeks ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-haproxy-nfs-nfs-1-fl31ca104ja0201-zdbzvv
From: Ben <ruidong.gao@gmail.com> Sent: Wednesday, May 17, 2023 6:32 PM To: Adiga, Anantha <anantha.adiga@intel.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Grafana service fails to start due to bad directory name after Quincy upgrade
use this to get relevant long lines in log:
journalctl -u ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201 | less -S
it is '--user 472' by content of unit.run, not the default ceph user 167. Maybe set the directory owner to 472 could help.
Hope it helps
Ben
Adiga, Anantha <anantha.adiga@intel.com<mailto:anantha.adiga@intel.com>> 于2023年5月18日周四 01:15写道： Ben,
Thanks for the suggestion. Changed the user and group to 167 for all files in the data and etc folders in the grafana service folder were not 167. Did a systemctl daemon-reload and restarted the grafana service ,
but still seeing the same error
-- Logs begin at Mon 2023-05-15 19:39:34 UTC, end at Wed 2023-05-17 17:08:02 UTC. -- May 17 17:07:44 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process exited, code=exite> May 17 17:07:44 fl31ca104ja0201 bash[148899]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No
May 17 17:07:44 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with result 'exit-code'. May 17 17:07:54 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled restart job, restart > May 17 17:07:54 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 17 17:07:54 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 17 17:07:54 fl31ca104ja0201 bash[149116]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such
May 17 17:07:54 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process exited, code=exite> May 17 17:07:54 fl31ca104ja0201 bash[149118]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No
May 17 17:07:54 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with result 'exit-code'. ESCOC 2 UTC. -- a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>: Main process exited, code=exited, status=127/n/a b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>: Failed with result 'exit-code'. a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>: Scheduled restart job, restart counter is at 3. a0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. a0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>: Main process exited, code=exited, status=127/n/a b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>: Failed with result 'exit-code'. ~
Thank you, Anantha
From: Ben <ruidong.gao@gmail.com<mailto:ruidong.gao@gmail.com>> Sent: Wednesday, May 17, 2023 2:29 AM To: Adiga, Anantha <anantha.adiga@intel.com<mailto:anantha.adiga@intel.com>> Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io> Subject: Re: [ceph-users] Grafana service fails to start due to bad directory name after Quincy upgrade
you could check owner of /var/lib/ceph on host with grafana container running. If its owner is root, change to 167:167 recursively. Then systemctl daemon-reload and restart the service. Good luck.
Ben
Adiga, Anantha <anantha.adiga@intel.com<mailto:anantha.adiga@intel.com>> 于2023年5月17日周三 03:57写道： Hi
Upgraded from Pacific 16.2.5 to 17.2.6 on May 8th
However, Grafana fails to start due to bad folder path :/tmp# journalctl -u ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201 -n 25 -- Logs begin at Sun 2023-05-14 20:05:52 UTC, end at Tue 2023-05-16 19:07:51 UTC. -- May 16 19:05:00 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:00 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:00 fl31ca104ja0201 bash[2575021]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:00 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:00 fl31ca104ja0201 bash[2575030]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:00 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:10 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:10 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:10 fl31ca104ja0201 bash[2575273]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:10 fl31ca104ja0201 bash[2575282]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:20 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:20 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:20 fl31ca104ja0201 bash[2575369]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:20 fl31ca104ja0201 bash[2575370]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:30 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Start request r> May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:30 fl31ca104ja0201 systemd[1]: Failed to start Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ESCOC 19:07:51 UTC. -- 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Main process exited, code=exited, status=127/n/a ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Failed with result 'exit-code'. -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Scheduled restart job, restart counter is at 3. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Main process exited, code=exited, status=127/n/a ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Failed with result 'exit-code'. -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Scheduled restart job, restart counter is at 4. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
Check if path exists: #<mailto:root@fl31ca104ja0201<mailto:root@fl31ca104ja0201>:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#> ls /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run ls: cannot access '/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run': No such file or directory
Check if grafana.fl31ca104ja0201 directory exists:
#<mailto:root@fl31ca104ja0201<mailto:root@fl31ca104ja0201>:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#> ls -l rt/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/unit.run total 36 drwxr-xr-x 4 167 167 4096 Apr 20 08:05 data drwxr-xr-x 3 167 167 4096 Apr 20 08:13 etc -rw------- 1 167 167 48 Apr 20 08:13 unit.created -rw------- 1 167 167 390 May 8 16:12 unit.stop -rw------- 1 167 167 390 May 8 16:12 unit.poststop -rw------- 1 167 167 365 May 8 16:12 unit.meta -rw------- 1 167 167 32 May 8 16:12 unit.image -rw------- 1 167 167 38 May 8 16:12 unit.configured -rw------- 1 167 167 1063 May 8 16:12 unit.run
cat unit.run set -e # grafana.fl31ca104ja0201 ! /usr/bin/docker rm -f ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana.fl31ca104ja0201 2> /dev/null ! /usr/bin/docker rm -f ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 2> /dev/null /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --init --name ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 --user 472 -e CONTAINER_IMAGE=docker.io/grafana/grafana:6.7.4<http://docker.io/grafana/grafana:6.7.4> -e NODE_NAME=fl31ca104ja0201 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/grafana.ini:/etc/grafana/grafana.ini:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/certs:/etc/grafana/certs:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/data/grafana.db:/var/lib/grafana/grafana.db:Z docker.io/grafana/grafana:6.7.4<http://docker.io/grafana/grafana:6.7.4>
Thank you, Anantha _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Adiga, Anantha
-
Ben
-
Eugen Block