ceph-crash NOT reporting crashes due to wrong permissions on /var/lib/ceph/crash/posted (Debian / Ubuntu packages)
Hey ceph-users, I just noticed issues with ceph-crash using the Debian /Ubuntu packages (package: ceph-base): While the /var/lib/ceph/crash/posted folder is created by the package install, it's not properly chowned to ceph:ceph by the postinst script. This might also affect RPM based installs somehow, but I did not look into that. I opened a bug report with all the details and two ideas to fix this: https://tracker.ceph.com/issues/64548 The wrong ownership causes ceph-crash to NOT work at all. I myself missed quite a few crash reports. All of them were just sitting around on the machines, but were reported right after I did chown ceph:ceph /var/lib/ceph/crash/posted systemctl restart ceph-crash.service You might want to check if you might be affected as well. Failing to post crashes to the local cluster results in them not being reported back via telemetry. Regards Christian
On 23.02.24 16:18, Christian Rohmann wrote:
I just noticed issues with ceph-crash using the Debian /Ubuntu packages (package: ceph-base):
While the /var/lib/ceph/crash/posted folder is created by the package install, it's not properly chowned to ceph:ceph by the postinst script.
[...]
You might want to check if you might be affected as well. Failing to post crashes to the local cluster results in them not being reported back via telemetry.
Sorry to bluntly bump this again, but did nobody else notice this on your clusters? Call me egoistic, but the more clusters return crash reports the more stable my Ceph likely becomes ;-) Regards Christian
On 23.02.24 16:18, Christian Rohmann wrote:
I just noticed issues with ceph-crash using the Debian /Ubuntu packages (package: ceph-base):
While the /var/lib/ceph/crash/posted folder is created by the package install, it's not properly chowned to ceph:ceph by the postinst script. ... You might want to check if you might be affected as well. Failing to post crashes to the local cluster results in them not being reported back via telemetry.
Sorry to bluntly bump this again, but did nobody else notice this on your clusters? Call me egoistic, but the more clusters return crash reports the more stable my Ceph likely becomes ;-)
I do observe the ownership does not match ceph:ceph on Debian with v17.2.7. $ sudo ls -l /var/lib/ceph/crash | grep posted drwxr-xr-x 2 root root 4096 Feb 10 19:23 posted The issue seems to be that the postinst script does not recursively chown and only chowns subdirectories directly under /var/lib/ceph: https://github.com/ceph/ceph/blob/91e8cea0d31775de0e59936b3608a9a453353a45/d... The rpm spec looks to do subdirectories under /var/lib/ceph as well, but explicitly lists everything out instead of globs, and also includes posted: https://github.com/ceph/ceph/blob/91e8cea0d31775de0e59936b3608a9a453353a45/c... Tyler
Hi, El 2/3/24 a las 18:00, Tyler Stachecki escribió:
On 23.02.24 16:18, Christian Rohmann wrote:
I just noticed issues with ceph-crash using the Debian /Ubuntu packages (package: ceph-base):
While the /var/lib/ceph/crash/posted folder is created by the package install, it's not properly chowned to ceph:ceph by the postinst script. ... You might want to check if you might be affected as well. Failing to post crashes to the local cluster results in them not being reported back via telemetry. Sorry to bluntly bump this again, but did nobody else notice this on your clusters? Call me egoistic, but the more clusters return crash reports the more stable my Ceph likely becomes ;-) I do observe the ownership does not match ceph:ceph on Debian with v17.2.7. $ sudo ls -l /var/lib/ceph/crash | grep posted drwxr-xr-x 2 root root 4096 Feb 10 19:23 posted
The issue seems to be that the postinst script does not recursively chown and only chowns subdirectories directly under /var/lib/ceph: https://github.com/ceph/ceph/blob/91e8cea0d31775de0e59936b3608a9a453353a45/d...
The rpm spec looks to do subdirectories under /var/lib/ceph as well, but explicitly lists everything out instead of globs, and also includes posted: https://github.com/ceph/ceph/blob/91e8cea0d31775de0e59936b3608a9a453353a45/c...
This seems to have been fixed in Proxmox recently: * master (reef?): https://lists.proxmox.com/pipermail/pve-devel/2024-February/061803.html * quincy: https://lists.proxmox.com/pipermail/pve-devel/2024-February/061798.html Not sure this has been upstreamed. Cheers Eneko Lacunza Zuzendari teknikoa | Director técnico Binovo IT Human Project Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/
participants (3)
-
Christian Rohmann
-
Eneko Lacunza
-
Tyler Stachecki