Re: Ubuntu 24.02 LTS Ceph status warning
Is apparmor configured differently on those hosts? Or is it running only on the misbehaving host? Zitat von Dominique Ramaekers <dominique.ramaekers@cometal.be>:
'ceph config get mgr container_image' gives quay.io/ceph/ceph@sha256:200087c35811bf28e8a8073b15fa86c07cce85c575f1ccd62d1d6ddbfdc6770a => OK
'ceph health detail' gives HEALTH_WARN failed to probe daemons or devices [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices host hvs004 `cephadm gather-facts` failed: cephadm exited with an error code: 1, stderr: Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/var/lib/ceph/dd4b0610-b4d2-11ec-bb58-d1b32ae31585/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 5579, in <module> ... File "/var/lib/ceph/dd4b0610-b4d2-11ec-bb58-d1b32ae31585/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/host_facts.py", line 722, in _fetch_apparmor ValueError: too many values to unpack (expected 2) host hvs004 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/dd4b0610-b4d2-11ec-bb58-d1b32ae31585/config/ceph.conf Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/var/lib/ceph/dd4b0610-b4d2-11ec-bb58-d1b32ae31585/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 5579, in <module> ... File "/var/lib/ceph/dd4b0610-b4d2-11ec-bb58-d1b32ae31585/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/host_facts.py", line 722, in _fetch_apparmor ValueError: too many values to unpack (expected 2)
I do think it's a ceph version issue, so I started to compaire the hvs004 host with a good behaving host hvs001. I did find this: 'root@hvs001~#:cephadm shell ceph -v' gives ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable) 'root@hvs004~#:cephadm shell ceph -v' gives ceph version 19.3.0-5346-gcc481a63 (cc481a63bc03a534cb8e2e961293d6509ba59401) squid (dev)
It seems only the shell uses a wrong docker image so I took a list from both of them: hvs0001 ---------- REPOSITORY TAG IMAGE ID CREATED SIZE quay.io/ceph/ceph v19 37996728e013 2 weeks ago 1.28GB quay.io/ceph/ceph v18.2 2bc0b0f4375d 2 months ago 1.22GB quay.io/ceph/ceph v18 a27483cc3ea0 6 months ago 1.26GB quay.io/ceph/ceph v17 5a04c8b3735d 9 months ago 1.27GB quay.io/ceph/ceph-grafana 9.4.7 954c08fa6188 10 months ago 633MB quay.io/ceph/grafana 9.4.12 2bacad6d85d8 17 months ago 330MB quay.io/prometheus/prometheus v2.43.0 a07b618ecd1d 19 months ago 234MB quay.io/prometheus/alertmanager v0.25.0 c8568f914cd2 22 months ago 65.1MB quay.io/prometheus/node-exporter v1.5.0 0da6a335fe13 22 months ago 22.5MB quay.io/ceph/ceph v17.2 0912465dcea5 2 years ago 1.34GB quay.io/ceph/ceph v17.2.3 0912465dcea5 2 years ago 1.34GB quay.io/ceph/ceph-grafana 8.3.5 dad864ee21e9 2 years ago 558MB quay.ceph.io/ceph-ci/ceph master c5ce177c6a5d 2 years ago 1.38GB quay.io/prometheus/prometheus v2.33.4 514e6a882f6e 2 years ago 204MB quay.io/prometheus/node-exporter v1.3.1 1dbe0e931976 2 years ago 20.9MB quay.io/prometheus/alertmanager v0.23.0 ba2b418f427c 3 years ago 57.5MB quay.io/ceph/ceph-grafana 6.7.4 557c83e11646 3 years ago 486MB quay.io/prometheus/prometheus v2.18.1 de242295e225 4 years ago 140MB quay.io/prometheus/alertmanager v0.20.0 0881eb8f169f 4 years ago 52.1MB quay.io/prometheus/node-exporter v0.18.1 e5a616e4b9cf 5 years ago 22.9MB
hvs004 --------- REPOSITORY TAG IMAGE ID CREATED SIZE quay.ceph.io/ceph-ci/ceph main 6e76ca06f33a 11 days ago 1.41GB quay.io/ceph/ceph <none> 37996728e013 2 weeks ago 1.28GB quay.io/ceph/ceph v17 9cea3956c04b 18 months ago 1.16GB quay.io/prometheus/node-exporter v1.5.0 0da6a335fe13 22 months ago 22.5MB quay.io/prometheus/node-exporter v1.3.1 1dbe0e931976 2 years ago 20.9MB
I pulled on hvs004 the v19 tagged image and my cephadm shell ceph -v gave the correct version.
It seems my docker images aren't automatically managed by ceph?
Can I fix this, or do I have to pull the correct images and remove the wrong ones myself?
-----Oorspronkelijk bericht----- Van: Eugen Block <eblock@nde.ag> Verzonden: vrijdag 11 oktober 2024 13:03 Aan: ceph-users@ceph.io Onderwerp: [ceph-users] Re: Ubuntu 24.02 LTS Ceph status warning
I don't think the warning is related to a specific ceph version. The orchestrator uses the default image anyway, you can get it via:
ceph config get mgr container_image
'ceph health detail' should reveal which host or daemons misbehaves. I would then look into cephadm.log on that host to find more hints, what exactly goes wrong. You should also look into the active MGR log, it could also give you hints why that service fails.
Zitat von Dominique Ramaekers <dominique.ramaekers@cometal.be>:
I manage a 4 hosts cluster on Ubuntu 22.04 LTS with ceph installed trough cephad and containers on Docker.
Last month, I've migrated to the latest Ceph 19.2. All went great.
Last week I've upgraded one of my hosts to Ubuntu 24.04.1 LTS Now I get the following warning in cephadm shell -- ceph status: Failed to apply 1 service(s): osd.all-available-devices failed to probe daemons or devices
Outside the ceph shell: Ceph -v results in 'ceph version 19.2.0~git20240301.4c76c50 (4c76c50a73f63ba48ccdf0adccce03b00d1d80c7) squid (dev)'
Inside the shell: 'ceph version 19.3.0-5346-gcc481a63 (cc481a63bc03a534cb8e2e961293d6509ba59401) squid (dev)' All osd's, mon's , mgr's and mds's are on 19.2.0` (image id 37996728e013)
Do I get the warning because the Ubuntu package of ceph is still on a development version? Or can I have another underling problem?
Thanks for the help. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Apparmor profiles for Ceph are apparently very limited on Ubuntu. On hvs001 (not misbehaving host) with services osd's, mgr, prometheus, mon - /bin/prometheus (93021) docker-default - /usr/bin/ceph-mgr (3574755) docker-default - /bin/alertmanager (3578797) docker-default - /usr/bin/ceph-mds (4051355) docker-default - /bin/node_exporter (4068210) docker-default On hvs004 (misbehaving host) with services osd's, Grafana - /bin/node_exporter (5389) docker-default - /usr/share/grafana/bin/grafana (854839) docker-default None of the hosts have ceph-profiles in /etc/apparmor.d. Are the apparmor profiles created on initialization of each service? I don't have any apparmor="DENIED" in syslog... I'v read David Orman's mail, but one bug report relates to podman and the other to runc. I'm not sure (not convinced) if I'm affected to one of these bugs. I've disabled apparmor on the misbehaving host, reset the host and it seems the issue is resolved... I'll keep the cluster running like this for now, just to make sure all stays well. Then I'll re-enable apparmor and if the issue reappears, I'll submit a bug report on Ubuntu. @Eugen and @David, thanks for the input!
-----Oorspronkelijk bericht----- Van: Eugen Block <eblock@nde.ag> Verzonden: woensdag 16 oktober 2024 17:24 Aan: Dominique Ramaekers <dominique.ramaekers@cometal.be> CC: ceph-users@ceph.io Onderwerp: Re: [ceph-users] Re: Ubuntu 24.02 LTS Ceph status warning
Is apparmor configured differently on those hosts? Or is it running only on the misbehaving host?
Zitat von Dominique Ramaekers <dominique.ramaekers@cometal.be>:
'ceph config get mgr container_image' gives
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG h
Ac2hhMjU2OjIwMDA4N2MzNTgxMWJmMjhlOGE4MDczYjE1ZmE4NmMwN2 NjZTg1YzU3NWYxY
2NkNjJkMWQ2ZGRiZmRjNjc3MGE=/C18099323945AF9F4A1D330A5B5F56543 074169706
2B694BDF90785FC96D992F?c=1&i=1&docs=1 => OK
'ceph health detail' gives HEALTH_WARN failed to probe daemons or devices [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices host hvs004 `cephadm gather-facts` failed: cephadm exited with an error code: 1, stderr: Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/var/lib/ceph/dd4b0610-b4d2-11ec-bb58- d1b32ae31585/cephadm.a58127a8ee
d242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.p y", line 5579, in <module>
... File "/var/lib/ceph/dd4b0610-b4d2-11ec-bb58- d1b32ae31585/cephadm.a58127a8ee
d242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib /host
_facts.py", line 722, in _fetch_apparmor ValueError: too many values to unpack (expected 2) host hvs004 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/dd4b0610-b4d2-11ec-bb58-d1b32ae31585/config/ceph.conf Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/var/lib/ceph/dd4b0610-b4d2-11ec-bb58- d1b32ae31585/cephadm.a58127a8ee
d242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.p y", line 5579, in <module>
... File "/var/lib/ceph/dd4b0610-b4d2-11ec-bb58- d1b32ae31585/cephadm.a58127a8ee
d242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib /host
_facts.py", line 722, in _fetch_apparmor ValueError: too many values to unpack (expected 2)
I do think it's a ceph version issue, so I started to compaire the hvs004 host with a good behaving host hvs001. I did find this: 'root@hvs001~#:cephadm shell ceph -v' gives ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable) 'root@hvs004~#:cephadm shell ceph -v' gives ceph version 19.3.0-5346-gcc481a63 (cc481a63bc03a534cb8e2e961293d6509ba59401) squid (dev)
It seems only the shell uses a wrong docker image so I took a list from both of them: hvs0001 ---------- REPOSITORY TAG IMAGE ID CREATED SIZE
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG g=/ADFE53E9DD9FA29720D92941A564893CB2CFA88EEE099C9A09763A6628C6 F3BD?c=1&i=1&docs=1 v19 37996728e013 2 weeks
ago 1.28GB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG g=/ADFE53E9DD9FA29720D92941A564893CB2CFA88EEE099C9A09763A6628C6 F3BD?c=1&i=1&docs=1 v18.2 2bc0b0f4375d 2 months
ago 1.22GB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG g=/ADFE53E9DD9FA29720D92941A564893CB2CFA88EEE099C9A09763A6628C6 F3BD?c=1&i=1&docs=1 v18 a27483cc3ea0 6 months
ago 1.26GB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG g=/ADFE53E9DD9FA29720D92941A564893CB2CFA88EEE099C9A09763A6628C6 F3BD?c=1&i=1&docs=1 v17 5a04c8b3735d 9 months
ago 1.27GB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG gtZ3JhZmFuYQ==/1111DCD4A88C0C2E0223375011289337FCA5859049F048BFA 1D957A6F9437591?c=1&i=1&docs=1 9.4.7 954c08fa6188 10
months ago 633MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2dyY WZhbmE=/68B473B21D47C3F07F8461CE79C63FB5C33025F9FB2344B554A38F2 29177F32D?c=1&i=1&docs=1 9.4.12 2bacad6d85d8 17
months ago 330MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL3Byb21ldGhldXM=/2CD01AE3BF4B76D6045EB8CFAB7D5AB4D2E1D6E4E4 D24A14F5B7061F3A23CDBC?c=1&i=1&docs=1 v2.43.0 a07b618ecd1d 19
months ago 234MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL2FsZXJ0bWFuYWdlcg==/4DB820C3D138493D338A2DFD7075A018E5D6DC 2F30EDE4E347B1A7E3B356AD13?c=1&i=1&docs=1 v0.25.0 c8568f914cd2 22
months ago 65.1MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL25vZGUtZXhwb3J0ZXI=/A423BD4C42A7129B12B8D543C456F8086899136 A61C0BEDF4814BDFD0E8F9323?c=1&i=1&docs=1 v1.5.0 0da6a335fe13 22
months ago 22.5MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG g=/ADFE53E9DD9FA29720D92941A564893CB2CFA88EEE099C9A09763A6628C6 F3BD?c=1&i=1&docs=1 v17.2 0912465dcea5 2 years
ago 1.34GB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG g=/ADFE53E9DD9FA29720D92941A564893CB2CFA88EEE099C9A09763A6628C6 F3BD?c=1&i=1&docs=1 v17.2.3 0912465dcea5 2 years
ago 1.34GB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG gtZ3JhZmFuYQ==/1111DCD4A88C0C2E0223375011289337FCA5859049F048BFA 1D957A6F9437591?c=1&i=1&docs=1 8.3.5 dad864ee21e9 2 years
ago 558MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5jZXBoLmlvL2NlcGg tY2kvY2VwaA==/5B87D1E443C50F1093DBB1CEB6CA0111A93277F371E38F7CF E2007DA95A2BD1A?c=1&i=1&docs=1 master c5ce177c6a5d 2 years
ago 1.38GB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL3Byb21ldGhldXM=/2CD01AE3BF4B76D6045EB8CFAB7D5AB4D2E1D6E4E4 D24A14F5B7061F3A23CDBC?c=1&i=1&docs=1 v2.33.4 514e6a882f6e 2 years
ago 204MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL25vZGUtZXhwb3J0ZXI=/A423BD4C42A7129B12B8D543C456F8086899136 A61C0BEDF4814BDFD0E8F9323?c=1&i=1&docs=1 v1.3.1 1dbe0e931976 2 years
ago 20.9MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL2FsZXJ0bWFuYWdlcg==/4DB820C3D138493D338A2DFD7075A018E5D6DC 2F30EDE4E347B1A7E3B356AD13?c=1&i=1&docs=1 v0.23.0 ba2b418f427c 3 years
ago 57.5MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG gtZ3JhZmFuYQ==/1111DCD4A88C0C2E0223375011289337FCA5859049F048BFA 1D957A6F9437591?c=1&i=1&docs=1 6.7.4 557c83e11646 3 years
ago 486MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL3Byb21ldGhldXM=/2CD01AE3BF4B76D6045EB8CFAB7D5AB4D2E1D6E4E4 D24A14F5B7061F3A23CDBC?c=1&i=1&docs=1 v2.18.1 de242295e225 4 years
ago 140MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL2FsZXJ0bWFuYWdlcg==/4DB820C3D138493D338A2DFD7075A018E5D6DC 2F30EDE4E347B1A7E3B356AD13?c=1&i=1&docs=1 v0.20.0 0881eb8f169f 4 years
ago 52.1MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL25vZGUtZXhwb3J0ZXI=/A423BD4C42A7129B12B8D543C456F8086899136 A61C0BEDF4814BDFD0E8F9323?c=1&i=1&docs=1 v0.18.1 e5a616e4b9cf 5 years
ago 22.9MB
hvs004 --------- REPOSITORY TAG IMAGE ID CREATED SIZE
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5jZXBoLmlvL2NlcGg tY2kvY2VwaA==/5B87D1E443C50F1093DBB1CEB6CA0111A93277F371E38F7CF E2007DA95A2BD1A?c=1&i=1&docs=1 main 6e76ca06f33a 11 days
ago 1.41GB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG g=/ADFE53E9DD9FA29720D92941A564893CB2CFA88EEE099C9A09763A6628C6 F3BD?c=1&i=1&docs=1 <none> 37996728e013 2 weeks
ago 1.28GB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG g=/ADFE53E9DD9FA29720D92941A564893CB2CFA88EEE099C9A09763A6628C6 F3BD?c=1&i=1&docs=1 v17 9cea3956c04b 18
months ago 1.16GB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL25vZGUtZXhwb3J0ZXI=/A423BD4C42A7129B12B8D543C456F8086899136 A61C0BEDF4814BDFD0E8F9323?c=1&i=1&docs=1 v1.5.0 0da6a335fe13 22
months ago 22.5MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL25vZGUtZXhwb3J0ZXI=/A423BD4C42A7129B12B8D543C456F8086899136 A61C0BEDF4814BDFD0E8F9323?c=1&i=1&docs=1 v1.3.1 1dbe0e931976 2 years
ago 20.9MB
I pulled on hvs004 the v19 tagged image and my cephadm shell ceph -v gave the correct version.
It seems my docker images aren't automatically managed by ceph?
Can I fix this, or do I have to pull the correct images and remove the wrong ones myself?
-----Oorspronkelijk bericht----- Van: Eugen Block <eblock@nde.ag> Verzonden: vrijdag 11 oktober 2024 13:03 Aan: ceph-users@ceph.io Onderwerp: [ceph-users] Re: Ubuntu 24.02 LTS Ceph status warning
I don't think the warning is related to a specific ceph version. The orchestrator uses the default image anyway, you can get it via:
ceph config get mgr container_image
'ceph health detail' should reveal which host or daemons misbehaves. I would then look into cephadm.log on that host to find more hints, what exactly goes wrong. You should also look into the active MGR log, it could also give you hints why that service fails.
Zitat von Dominique Ramaekers <dominique.ramaekers@cometal.be>:
I manage a 4 hosts cluster on Ubuntu 22.04 LTS with ceph installed trough cephad and containers on Docker.
Last month, I've migrated to the latest Ceph 19.2. All went great.
Last week I've upgraded one of my hosts to Ubuntu 24.04.1 LTS Now I get the following warning in cephadm shell -- ceph status: Failed to apply 1 service(s): osd.all-available-devices failed to probe daemons or devices
Outside the ceph shell: Ceph -v results in 'ceph version 19.2.0~git20240301.4c76c50 (4c76c50a73f63ba48ccdf0adccce03b00d1d80c7) squid (dev)'
Inside the shell: 'ceph version 19.3.0-5346-gcc481a63 (cc481a63bc03a534cb8e2e961293d6509ba59401) squid (dev)' All osd's, mon's , mgr's and mds's are on 19.2.0` (image id 37996728e013)
Do I get the warning because the Ubuntu package of ceph is still on a development version? Or can I have another underling problem?
Thanks for the help. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
My knowledge of AppArmor and Ubuntu is too limited to give a qualified answer, I'll leave it for others to respond. Zitat von Dominique Ramaekers <dominique.ramaekers@cometal.be>:
Apparmor profiles for Ceph are apparently very limited on Ubuntu.
On hvs001 (not misbehaving host) with services osd's, mgr, prometheus, mon - /bin/prometheus (93021) docker-default - /usr/bin/ceph-mgr (3574755) docker-default - /bin/alertmanager (3578797) docker-default - /usr/bin/ceph-mds (4051355) docker-default - /bin/node_exporter (4068210) docker-default
On hvs004 (misbehaving host) with services osd's, Grafana - /bin/node_exporter (5389) docker-default - /usr/share/grafana/bin/grafana (854839) docker-default
None of the hosts have ceph-profiles in /etc/apparmor.d. Are the apparmor profiles created on initialization of each service?
I don't have any apparmor="DENIED" in syslog...
I'v read David Orman's mail, but one bug report relates to podman and the other to runc. I'm not sure (not convinced) if I'm affected to one of these bugs.
I've disabled apparmor on the misbehaving host, reset the host and it seems the issue is resolved...
I'll keep the cluster running like this for now, just to make sure all stays well. Then I'll re-enable apparmor and if the issue reappears, I'll submit a bug report on Ubuntu.
@Eugen and @David, thanks for the input!
-----Oorspronkelijk bericht----- Van: Eugen Block <eblock@nde.ag> Verzonden: woensdag 16 oktober 2024 17:24 Aan: Dominique Ramaekers <dominique.ramaekers@cometal.be> CC: ceph-users@ceph.io Onderwerp: Re: [ceph-users] Re: Ubuntu 24.02 LTS Ceph status warning
Is apparmor configured differently on those hosts? Or is it running only on the misbehaving host?
Zitat von Dominique Ramaekers <dominique.ramaekers@cometal.be>:
'ceph config get mgr container_image' gives
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG h
Ac2hhMjU2OjIwMDA4N2MzNTgxMWJmMjhlOGE4MDczYjE1ZmE4NmMwN2 NjZTg1YzU3NWYxY
2NkNjJkMWQ2ZGRiZmRjNjc3MGE=/C18099323945AF9F4A1D330A5B5F56543 074169706
2B694BDF90785FC96D992F?c=1&i=1&docs=1 => OK
'ceph health detail' gives HEALTH_WARN failed to probe daemons or devices [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices host hvs004 `cephadm gather-facts` failed: cephadm exited with an error code: 1, stderr: Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/var/lib/ceph/dd4b0610-b4d2-11ec-bb58- d1b32ae31585/cephadm.a58127a8ee
d242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.p y", line 5579, in <module>
... File "/var/lib/ceph/dd4b0610-b4d2-11ec-bb58- d1b32ae31585/cephadm.a58127a8ee
d242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib /host
_facts.py", line 722, in _fetch_apparmor ValueError: too many values to unpack (expected 2) host hvs004 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/dd4b0610-b4d2-11ec-bb58-d1b32ae31585/config/ceph.conf Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/var/lib/ceph/dd4b0610-b4d2-11ec-bb58- d1b32ae31585/cephadm.a58127a8ee
d242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.p y", line 5579, in <module>
... File "/var/lib/ceph/dd4b0610-b4d2-11ec-bb58- d1b32ae31585/cephadm.a58127a8ee
d242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib /host
_facts.py", line 722, in _fetch_apparmor ValueError: too many values to unpack (expected 2)
I do think it's a ceph version issue, so I started to compaire the hvs004 host with a good behaving host hvs001. I did find this: 'root@hvs001~#:cephadm shell ceph -v' gives ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable) 'root@hvs004~#:cephadm shell ceph -v' gives ceph version 19.3.0-5346-gcc481a63 (cc481a63bc03a534cb8e2e961293d6509ba59401) squid (dev)
It seems only the shell uses a wrong docker image so I took a list from both of them: hvs0001 ---------- REPOSITORY TAG IMAGE ID CREATED SIZE
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG g=/ADFE53E9DD9FA29720D92941A564893CB2CFA88EEE099C9A09763A6628C6 F3BD?c=1&i=1&docs=1 v19 37996728e013 2 weeks
ago 1.28GB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG g=/ADFE53E9DD9FA29720D92941A564893CB2CFA88EEE099C9A09763A6628C6 F3BD?c=1&i=1&docs=1 v18.2 2bc0b0f4375d 2 months
ago 1.22GB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG g=/ADFE53E9DD9FA29720D92941A564893CB2CFA88EEE099C9A09763A6628C6 F3BD?c=1&i=1&docs=1 v18 a27483cc3ea0 6 months
ago 1.26GB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG g=/ADFE53E9DD9FA29720D92941A564893CB2CFA88EEE099C9A09763A6628C6 F3BD?c=1&i=1&docs=1 v17 5a04c8b3735d 9 months
ago 1.27GB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG gtZ3JhZmFuYQ==/1111DCD4A88C0C2E0223375011289337FCA5859049F048BFA 1D957A6F9437591?c=1&i=1&docs=1 9.4.7 954c08fa6188 10
months ago 633MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2dyY WZhbmE=/68B473B21D47C3F07F8461CE79C63FB5C33025F9FB2344B554A38F2 29177F32D?c=1&i=1&docs=1 9.4.12 2bacad6d85d8 17
months ago 330MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL3Byb21ldGhldXM=/2CD01AE3BF4B76D6045EB8CFAB7D5AB4D2E1D6E4E4 D24A14F5B7061F3A23CDBC?c=1&i=1&docs=1 v2.43.0 a07b618ecd1d 19
months ago 234MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL2FsZXJ0bWFuYWdlcg==/4DB820C3D138493D338A2DFD7075A018E5D6DC 2F30EDE4E347B1A7E3B356AD13?c=1&i=1&docs=1 v0.25.0 c8568f914cd2 22
months ago 65.1MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL25vZGUtZXhwb3J0ZXI=/A423BD4C42A7129B12B8D543C456F8086899136 A61C0BEDF4814BDFD0E8F9323?c=1&i=1&docs=1 v1.5.0 0da6a335fe13 22
months ago 22.5MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG g=/ADFE53E9DD9FA29720D92941A564893CB2CFA88EEE099C9A09763A6628C6 F3BD?c=1&i=1&docs=1 v17.2 0912465dcea5 2 years
ago 1.34GB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG g=/ADFE53E9DD9FA29720D92941A564893CB2CFA88EEE099C9A09763A6628C6 F3BD?c=1&i=1&docs=1 v17.2.3 0912465dcea5 2 years
ago 1.34GB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG gtZ3JhZmFuYQ==/1111DCD4A88C0C2E0223375011289337FCA5859049F048BFA 1D957A6F9437591?c=1&i=1&docs=1 8.3.5 dad864ee21e9 2 years
ago 558MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5jZXBoLmlvL2NlcGg tY2kvY2VwaA==/5B87D1E443C50F1093DBB1CEB6CA0111A93277F371E38F7CF E2007DA95A2BD1A?c=1&i=1&docs=1 master c5ce177c6a5d 2 years
ago 1.38GB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL3Byb21ldGhldXM=/2CD01AE3BF4B76D6045EB8CFAB7D5AB4D2E1D6E4E4 D24A14F5B7061F3A23CDBC?c=1&i=1&docs=1 v2.33.4 514e6a882f6e 2 years
ago 204MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL25vZGUtZXhwb3J0ZXI=/A423BD4C42A7129B12B8D543C456F8086899136 A61C0BEDF4814BDFD0E8F9323?c=1&i=1&docs=1 v1.3.1 1dbe0e931976 2 years
ago 20.9MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL2FsZXJ0bWFuYWdlcg==/4DB820C3D138493D338A2DFD7075A018E5D6DC 2F30EDE4E347B1A7E3B356AD13?c=1&i=1&docs=1 v0.23.0 ba2b418f427c 3 years
ago 57.5MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG gtZ3JhZmFuYQ==/1111DCD4A88C0C2E0223375011289337FCA5859049F048BFA 1D957A6F9437591?c=1&i=1&docs=1 6.7.4 557c83e11646 3 years
ago 486MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL3Byb21ldGhldXM=/2CD01AE3BF4B76D6045EB8CFAB7D5AB4D2E1D6E4E4 D24A14F5B7061F3A23CDBC?c=1&i=1&docs=1 v2.18.1 de242295e225 4 years
ago 140MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL2FsZXJ0bWFuYWdlcg==/4DB820C3D138493D338A2DFD7075A018E5D6DC 2F30EDE4E347B1A7E3B356AD13?c=1&i=1&docs=1 v0.20.0 0881eb8f169f 4 years
ago 52.1MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL25vZGUtZXhwb3J0ZXI=/A423BD4C42A7129B12B8D543C456F8086899136 A61C0BEDF4814BDFD0E8F9323?c=1&i=1&docs=1 v0.18.1 e5a616e4b9cf 5 years
ago 22.9MB
hvs004 --------- REPOSITORY TAG IMAGE ID CREATED SIZE
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5jZXBoLmlvL2NlcGg tY2kvY2VwaA==/5B87D1E443C50F1093DBB1CEB6CA0111A93277F371E38F7CF E2007DA95A2BD1A?c=1&i=1&docs=1 main 6e76ca06f33a 11 days
ago 1.41GB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG g=/ADFE53E9DD9FA29720D92941A564893CB2CFA88EEE099C9A09763A6628C6 F3BD?c=1&i=1&docs=1 <none> 37996728e013 2 weeks
ago 1.28GB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9jZXBoL2NlcG g=/ADFE53E9DD9FA29720D92941A564893CB2CFA88EEE099C9A09763A6628C6 F3BD?c=1&i=1&docs=1 v17 9cea3956c04b 18
months ago 1.16GB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL25vZGUtZXhwb3J0ZXI=/A423BD4C42A7129B12B8D543C456F8086899136 A61C0BEDF4814BDFD0E8F9323?c=1&i=1&docs=1 v1.5.0 0da6a335fe13 22
months ago 22.5MB
https://lsems.gravityzone.bitdefender.com/scan/cXVheS5pby9wcm9tZXRoZ XVzL25vZGUtZXhwb3J0ZXI=/A423BD4C42A7129B12B8D543C456F8086899136 A61C0BEDF4814BDFD0E8F9323?c=1&i=1&docs=1 v1.3.1 1dbe0e931976 2 years
ago 20.9MB
I pulled on hvs004 the v19 tagged image and my cephadm shell ceph -v gave the correct version.
It seems my docker images aren't automatically managed by ceph?
Can I fix this, or do I have to pull the correct images and remove the wrong ones myself?
-----Oorspronkelijk bericht----- Van: Eugen Block <eblock@nde.ag> Verzonden: vrijdag 11 oktober 2024 13:03 Aan: ceph-users@ceph.io Onderwerp: [ceph-users] Re: Ubuntu 24.02 LTS Ceph status warning
I don't think the warning is related to a specific ceph version. The orchestrator uses the default image anyway, you can get it via:
ceph config get mgr container_image
'ceph health detail' should reveal which host or daemons misbehaves. I would then look into cephadm.log on that host to find more hints, what exactly goes wrong. You should also look into the active MGR log, it could also give you hints why that service fails.
Zitat von Dominique Ramaekers <dominique.ramaekers@cometal.be>:
I manage a 4 hosts cluster on Ubuntu 22.04 LTS with ceph installed trough cephad and containers on Docker.
Last month, I've migrated to the latest Ceph 19.2. All went great.
Last week I've upgraded one of my hosts to Ubuntu 24.04.1 LTS Now I get the following warning in cephadm shell -- ceph status: Failed to apply 1 service(s): osd.all-available-devices failed to probe daemons or devices
Outside the ceph shell: Ceph -v results in 'ceph version 19.2.0~git20240301.4c76c50 (4c76c50a73f63ba48ccdf0adccce03b00d1d80c7) squid (dev)'
Inside the shell: 'ceph version 19.3.0-5346-gcc481a63 (cc481a63bc03a534cb8e2e961293d6509ba59401) squid (dev)' All osd's, mon's , mgr's and mds's are on 19.2.0` (image id 37996728e013)
Do I get the warning because the Ubuntu package of ceph is still on a development version? Or can I have another underling problem?
Thanks for the help. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Dominique Ramaekers
-
Eugen Block