/var/lib/ceph/crash/posted does not exist
Hi guys. I'm missing: /var/lib/ceph/crash/posted on each node, one node where 'mon' fails: -> $ ceph health detail HEALTH_WARN 1 failed cephadm daemon(s); 1/3 mons down, quorum podster3,podster2 [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) daemon mon.podster1.mine.priv on podster1.mine.priv is in error state [WRN] MON_DOWN: 1/3 mons down, quorum podster3,podster2 mon.podster1.mine.priv (rank 0) addr [v2:10.1.1.61:3300/0,v1:10.1.1.61:6789/0] is down (out of quorum) started logging: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create Is this expected - on newly deployed cluster with 'cephadm' - that this path does not exist, with 'crash' module already enabled? Is this the SOP that user creates (also perhaps do other necessary) this path manually in order for 'crash' to be functional? many thanks, L.
Is it possible that you have ceph-crash installed (non-cephadm)? Maybe other ceph packages as well? I recommend to remove all ceph packages except ceph-common if you intend to use cephadm. But ceph-crash is not the reason for the failing mon, not sure if you tried to connect those two things. Zitat von lejeczek <peljasz@yahoo.co.uk>:
Hi guys.
I'm missing: /var/lib/ceph/crash/posted on each node, one node where 'mon' fails: -> $ ceph health detail HEALTH_WARN 1 failed cephadm daemon(s); 1/3 mons down, quorum podster3,podster2 [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) daemon mon.podster1.mine.priv on podster1.mine.priv is in error state [WRN] MON_DOWN: 1/3 mons down, quorum podster3,podster2 mon.podster1.mine.priv (rank 0) addr [v2:10.1.1.61:3300/0,v1:10.1.1.61:6789/0] is down (out of quorum)
started logging: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Is this expected - on newly deployed cluster with 'cephadm' - that this path does not exist, with 'crash' module already enabled? Is this the SOP that user creates (also perhaps do other necessary) this path manually in order for 'crash' to be functional?
many thanks, L. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
no 'ceph-crush' installed as far as I can tell. -> $ rpm -qa \*ceph\* centos-release-ceph-squid-1.0-1.el9s.noarch python3-ceph-argparse-19.2.3-1.el9s.x86_64 libcephfs2-19.2.3-1.el9s.x86_64 python3-cephfs-19.2.3-1.el9s.x86_64 python3-ceph-common-19.2.3-1.el9s.x86_64 ceph-common-19.2.3-1.el9s.x86_64 cephadm-19.2.3-1.el9s.noarch cluster was deployed with -> $ cephadm bootstrap ... What also is "interesting" is: -> $ cephadm shell --name crash.$HOSTNAME Inferring fsid 9f4f9dba-72c7-11f0-8052-525400519d29 Inferring config /var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/crash.podster1.mine.priv/config Using ceph image with id 'aade1b12b8e6' and tag 'v19' created on 2025-07-17 19:53:27 +0000 UTC quay.io/ceph/ceph@sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102a753902f33ee16c26b6cee Error: statfs /var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/crash.podster1.mine.priv/keyring: no such file or directory and that is for all nodes. But I find: /var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/crash/posted/ on all/each node - which path _was_ created by the cluster (as opposed to human) also, I find: /var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/crash.podster3/keyring also on each node, which seems to contain other configs - as 'shell --name ...' above, was looking for. No, failing mon I thought I should have mentioned it only as trace of originating issue - but I did not think of causality. Is this some hostnames / FQDN misconfiguration ? but if so, then that was/is cluster/cephadm on its own. OS resolver check /etc/hosts - DNS also involved later - which contains: -> $ cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.1.1.61 podster1.mine.priv podster1 10.1.1.62 podster2.mine.priv podster2 10.1.1.63 podster3.mine.priv podster3 I added hosts to the cluster - and 'bootstraped' - with their FQDNs. thanks, L.
My Ceph cluster(s) usually run on openSUSE, in this case Leap 15.6. The host directory /var/lib/ceph/crash/posted is created by the 'ceph-base' package, so I removed that (this is a test machine for all kinds of purposes). I redeployed with cephadm (I don't use FQDN) and it creates the directory /var/lib/ceph/{FSID}/crash/posted for me. I don't see a warning about this directory. So just to clarify, is it complaining about the host directory /var/lib/ceph/crash/posted not being present or the "mapped" directory within the ceph-crash container? There have been a couple of releases which don't handle crash well in general. I haven't look too deep into the latest though. As for the cephadm shell, what das 'cephadm ls --no-detail | grep crash' show you? If you deployed with FQDN, I'd expect that crash is also deployed with FQDN. But $HOSTNAME could be the shortname, so you probably should use a different approach, for example 'cephadm shell --name crash.$(hostname -f) or something similar. Zitat von lejeczek <peljasz@yahoo.co.uk>:
no 'ceph-crush' installed as far as I can tell.
-> $ rpm -qa \*ceph\* centos-release-ceph-squid-1.0-1.el9s.noarch python3-ceph-argparse-19.2.3-1.el9s.x86_64 libcephfs2-19.2.3-1.el9s.x86_64 python3-cephfs-19.2.3-1.el9s.x86_64 python3-ceph-common-19.2.3-1.el9s.x86_64 ceph-common-19.2.3-1.el9s.x86_64 cephadm-19.2.3-1.el9s.noarch
cluster was deployed with -> $ cephadm bootstrap ...
What also is "interesting" is: -> $ cephadm shell --name crash.$HOSTNAME Inferring fsid 9f4f9dba-72c7-11f0-8052-525400519d29 Inferring config /var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/crash.podster1.mine.priv/config Using ceph image with id 'aade1b12b8e6' and tag 'v19' created on 2025-07-17 19:53:27 +0000 UTC quay.io/ceph/ceph@sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102a753902f33ee16c26b6cee Error: statfs /var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/crash.podster1.mine.priv/keyring: no such file or directory
and that is for all nodes. But I find: /var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/crash/posted/ on all/each node - which path _was_ created by the cluster (as opposed to human) also, I find: /var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/crash.podster3/keyring also on each node, which seems to contain other configs - as 'shell --name ...' above, was looking for.
No, failing mon I thought I should have mentioned it only as trace of originating issue - but I did not think of causality. Is this some hostnames / FQDN misconfiguration ? but if so, then that was/is cluster/cephadm on its own. OS resolver check /etc/hosts - DNS also involved later - which contains: -> $ cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.1.1.61 podster1.mine.priv podster1 10.1.1.62 podster2.mine.priv podster2 10.1.1.63 podster3.mine.priv podster3
I added hosts to the cluster - and 'bootstraped' - with their FQDNs.
thanks, L. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ok, seems that FQDN vs short hostnames does not play a role - not here at least. I'm must have either, bootstraped in the past differently than this one current cluster or remember it wrongly. this works: -> $ cephadm shell --name crash.podster1 and there is - in the container - /var/lib/ceph/crash/posted and I think I read about it somewhere - pitty that if it was a bug, it's still not fixed in 19.2.3 - and it helped: chowning ceph:ceph not just on _posted_ but the parent /var/lib/ceph/crash - for it was owned by 'nobody' What should be the permission of '/etc/ceph/' content - on the host - ?
On 8/18/25 2:11 PM, lejeczek wrote:
Ok, seems that FQDN vs short hostnames does not play a role - not here at least. I'm must have either, bootstraped in the past differently than this one current cluster or remember it wrongly. this works: -> $ cephadm shell --name crash.podster1 and there is - in the container - /var/lib/ceph/crash/posted and I think I read about it somewhere - pitty that if it was a bug, it's still not fixed in 19.2.3 - and it helped: chowning ceph:ceph not just on _posted_ but the parent /var/lib/ceph/crash - for it was owned by 'nobody' What should be the permission of '/etc/ceph/' content - on the host - ?
This sounds similar to a bug I reported a while back: * https://tracker.ceph.com/issues/64548 * see my ML post https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/VACLBNVXTYNS... Regards Christian
participants (3)
-
Christian Rohmann
-
Eugen Block
-
lejeczek