16.2.13: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Hi, I'm having an issue with crash daemons on Pacific 16.2.13 hosts. ceph-crash throws the following error on all hosts: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create ceph-crash runs in docker, the container has the directory mounted: -v /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash:/var/lib/ceph/crash:z The mount works correctly: 18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# ls -al crash/posted/ total 8 drwx------ 2 nobody nogroup 4096 May 6 2021 . drwx------ 3 nobody nogroup 4096 May 6 2021 .. 18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# touch crash/posted/a 18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# docker exec -it c0cd2b8022d8 bash [root@ceph02 /]# ls -al /var/lib/ceph/crash/posted/ total 8 drwx------ 2 nobody nobody 4096 Jun 1 18:26 . drwx------ 3 nobody nobody 4096 May 6 2021 .. -rw-r--r-- 1 root root 0 Jun 1 18:26 a I.e. the directory actually exists and is correctly mounted in the crash container, yet ceph-crash says it doesn't exist. How can I convince it that the directory is there? Best regards, Zakhar
Hi Zakhar, I'm going to guess that it's a permissions issue arising from https://github.com/ceph/ceph/pull/48804, which was included in 16.2.13. You may need to change the directory permissions, assuming that you manage the directories yourself. If this is managed by cephadm or something like that, then that seems like some sort of missing migration in the upgrade. Josh On Thu, Jun 1, 2023 at 12:34 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi,
I'm having an issue with crash daemons on Pacific 16.2.13 hosts. ceph-crash throws the following error on all hosts:
ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
ceph-crash runs in docker, the container has the directory mounted: -v
/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash:/var/lib/ceph/crash:z
The mount works correctly:
18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# ls -al crash/posted/ total 8 drwx------ 2 nobody nogroup 4096 May 6 2021 . drwx------ 3 nobody nogroup 4096 May 6 2021 ..
18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# touch crash/posted/a
18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# docker exec -it c0cd2b8022d8 bash
[root@ceph02 /]# ls -al /var/lib/ceph/crash/posted/ total 8 drwx------ 2 nobody nobody 4096 Jun 1 18:26 . drwx------ 3 nobody nobody 4096 May 6 2021 .. -rw-r--r-- 1 root root 0 Jun 1 18:26 a
I.e. the directory actually exists and is correctly mounted in the crash container, yet ceph-crash says it doesn't exist. How can I convince it that the directory is there?
Best regards, Zakhar _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks, Josh. The cluster is managed by cephadm. On Thu, 1 Jun 2023, 23:07 Josh Baergen, <jbaergen@digitalocean.com> wrote:
Hi Zakhar,
I'm going to guess that it's a permissions issue arising from https://github.com/ceph/ceph/pull/48804, which was included in 16.2.13. You may need to change the directory permissions, assuming that you manage the directories yourself. If this is managed by cephadm or something like that, then that seems like some sort of missing migration in the upgrade.
Josh
On Thu, Jun 1, 2023 at 12:34 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi,
I'm having an issue with crash daemons on Pacific 16.2.13 hosts. ceph-crash throws the following error on all hosts:
ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
ceph-crash runs in docker, the container has the directory mounted: -v
/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash:/var/lib/ceph/crash:z
The mount works correctly:
18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# ls -al crash/posted/ total 8 drwx------ 2 nobody nogroup 4096 May 6 2021 . drwx------ 3 nobody nogroup 4096 May 6 2021 ..
18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# touch crash/posted/a
18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# docker exec -it c0cd2b8022d8 bash
[root@ceph02 /]# ls -al /var/lib/ceph/crash/posted/ total 8 drwx------ 2 nobody nobody 4096 Jun 1 18:26 . drwx------ 3 nobody nobody 4096 May 6 2021 .. -rw-r--r-- 1 root root 0 Jun 1 18:26 a
I.e. the directory actually exists and is correctly mounted in the crash container, yet ceph-crash says it doesn't exist. How can I convince it that the directory is there?
Best regards, Zakhar _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Any other thoughts on this, please? Should I file a bug report? /Z On Fri, 2 Jun 2023 at 06:11, Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Thanks, Josh. The cluster is managed by cephadm.
On Thu, 1 Jun 2023, 23:07 Josh Baergen, <jbaergen@digitalocean.com> wrote:
Hi Zakhar,
I'm going to guess that it's a permissions issue arising from https://github.com/ceph/ceph/pull/48804, which was included in 16.2.13. You may need to change the directory permissions, assuming that you manage the directories yourself. If this is managed by cephadm or something like that, then that seems like some sort of missing migration in the upgrade.
Josh
On Thu, Jun 1, 2023 at 12:34 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi,
I'm having an issue with crash daemons on Pacific 16.2.13 hosts. ceph-crash throws the following error on all hosts:
ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
ceph-crash runs in docker, the container has the directory mounted: -v
/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash:/var/lib/ceph/crash:z
The mount works correctly:
18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# ls -al crash/posted/ total 8 drwx------ 2 nobody nogroup 4096 May 6 2021 . drwx------ 3 nobody nogroup 4096 May 6 2021 ..
18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# touch crash/posted/a
18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# docker exec -it c0cd2b8022d8 bash
[root@ceph02 /]# ls -al /var/lib/ceph/crash/posted/ total 8 drwx------ 2 nobody nobody 4096 Jun 1 18:26 . drwx------ 3 nobody nobody 4096 May 6 2021 .. -rw-r--r-- 1 root root 0 Jun 1 18:26 a
I.e. the directory actually exists and is correctly mounted in the crash container, yet ceph-crash says it doesn't exist. How can I convince it that the directory is there?
Best regards, Zakhar _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I've opened a bug report https://tracker.ceph.com/issues/61589, which unfortunately received no attention. I fixed the issue by manually setting directory ownership for /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash and /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash/posted to 167:167, which in my system is the user ID crash process uses inside the crash container. /Z On Mon, 5 Jun 2023 at 11:24, Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Any other thoughts on this, please? Should I file a bug report?
/Z
On Fri, 2 Jun 2023 at 06:11, Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Thanks, Josh. The cluster is managed by cephadm.
On Thu, 1 Jun 2023, 23:07 Josh Baergen, <jbaergen@digitalocean.com> wrote:
Hi Zakhar,
I'm going to guess that it's a permissions issue arising from https://github.com/ceph/ceph/pull/48804, which was included in 16.2.13. You may need to change the directory permissions, assuming that you manage the directories yourself. If this is managed by cephadm or something like that, then that seems like some sort of missing migration in the upgrade.
Josh
On Thu, Jun 1, 2023 at 12:34 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi,
I'm having an issue with crash daemons on Pacific 16.2.13 hosts. ceph-crash throws the following error on all hosts:
ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
ceph-crash runs in docker, the container has the directory mounted: -v
/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash:/var/lib/ceph/crash:z
The mount works correctly:
18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# ls -al crash/posted/ total 8 drwx------ 2 nobody nogroup 4096 May 6 2021 . drwx------ 3 nobody nogroup 4096 May 6 2021 ..
18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# touch crash/posted/a
18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# docker exec -it c0cd2b8022d8 bash
[root@ceph02 /]# ls -al /var/lib/ceph/crash/posted/ total 8 drwx------ 2 nobody nobody 4096 Jun 1 18:26 . drwx------ 3 nobody nobody 4096 May 6 2021 .. -rw-r--r-- 1 root root 0 Jun 1 18:26 a
I.e. the directory actually exists and is correctly mounted in the crash container, yet ceph-crash says it doesn't exist. How can I convince it that the directory is there?
Best regards, Zakhar _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I wonder if a redeploy of the crash service would fix that, did you try that? Zitat von Zakhar Kirpichenko <zakhar@gmail.com>:
I've opened a bug report https://tracker.ceph.com/issues/61589, which unfortunately received no attention.
I fixed the issue by manually setting directory ownership for /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash and /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash/posted to 167:167, which in my system is the user ID crash process uses inside the crash container.
/Z
On Mon, 5 Jun 2023 at 11:24, Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Any other thoughts on this, please? Should I file a bug report?
/Z
On Fri, 2 Jun 2023 at 06:11, Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Thanks, Josh. The cluster is managed by cephadm.
On Thu, 1 Jun 2023, 23:07 Josh Baergen, <jbaergen@digitalocean.com> wrote:
Hi Zakhar,
I'm going to guess that it's a permissions issue arising from https://github.com/ceph/ceph/pull/48804, which was included in 16.2.13. You may need to change the directory permissions, assuming that you manage the directories yourself. If this is managed by cephadm or something like that, then that seems like some sort of missing migration in the upgrade.
Josh
On Thu, Jun 1, 2023 at 12:34 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi,
I'm having an issue with crash daemons on Pacific 16.2.13 hosts. ceph-crash throws the following error on all hosts:
ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
ceph-crash runs in docker, the container has the directory mounted: -v
/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash:/var/lib/ceph/crash:z
The mount works correctly:
18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# ls -al crash/posted/ total 8 drwx------ 2 nobody nogroup 4096 May 6 2021 . drwx------ 3 nobody nogroup 4096 May 6 2021 ..
18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# touch crash/posted/a
18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# docker exec -it c0cd2b8022d8 bash
[root@ceph02 /]# ls -al /var/lib/ceph/crash/posted/ total 8 drwx------ 2 nobody nobody 4096 Jun 1 18:26 . drwx------ 3 nobody nobody 4096 May 6 2021 .. -rw-r--r-- 1 root root 0 Jun 1 18:26 a
I.e. the directory actually exists and is correctly mounted in the crash container, yet ceph-crash says it doesn't exist. How can I convince it that the directory is there?
Best regards, Zakhar _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Eugen Block
-
Josh Baergen
-
Zakhar Kirpichenko