- ceph-users - lists.ceph.io

16.2.14: [progress WARNING root] complete: ev {UUID} does not exist
by Zakhar Kirpichenko 03 May '24

03 May '24

Hi, Mgr of my cluster logs this every few seconds: [progress WARNING root] complete: ev 7de5bb74-790b-4fda-8838-e4af4af18c62 does not exist [progress WARNING root] complete: ev fff93fce-b630-4141-81ee-19e7a3e61483 does not exist [progress WARNING root] complete: ev a02f6966-5b9f-49e8-89c4-b4fb8e6f4423 does not exist [progress WARNING root] complete: ev 8d318560-ff1a-477f-9386-43f6b51080bf does not exist [progress WARNING root] complete: ev ff3740a9-6434-470a-808f-a2762fb542a0 does not exist [progress WARNING root] complete: ev 7d0589f1-545e-4970-867b-8482ce48d7f0 does not exist [progress WARNING root] complete: ev 78d57e43-5be5-43f0-8b1a-cdc60e410892 does not exist I would appreciate an advice on what these warnings mean and how they can be resolved. Best regards, Zakhar

4 7

Reset health.
by Albert Shih 03 May '24

03 May '24

Hi, Very basic question : 2 days ago I reboot all the cluster. Everything work fine. But I'm guessing during the shutdown 4 osd was mark as crash [WRN] RECENT_CRASH: 4 daemons have recently crashed osd.381 crashed on host cthulhu5 at 2024-03-20T18:33:12.017102Z osd.379 crashed on host cthulhu4 at 2024-03-20T18:47:13.838839Z osd.376 crashed on host cthulhu3 at 2024-03-20T18:50:00.877536Z osd.373 crashed on host cthulhu1 at 2024-03-20T18:56:46.887394Z is they are any way to «clean» that ? Because otherwise my icinga complain.... I don't like to add a downtime in icinga. Thanks. -- Albert SHIH 🦫 🐸 France Heure locale/Local time: ven. 22 mars 2024 22:24:35 CET

3 2

cephadm custom crush location hooks
by Eugen Block 03 May '24

03 May '24

Hi, we've been using custom crush location hooks for some OSDs [1] for years. Since we moved to cephadm, we always have to manually edit the unit.run file of those OSDs because the path to the script is not mapped into the containers. I don't want to define custom location hooks for all OSDs globally in the OSD spec, even if those are limited to two hosts only in our case. But I'm not aware of a method to target only specific OSDs to have some files mapped into the container [2]. Is my assumption correct that we'll have to live with the manual intervention until we reorganize our osd tree? Or did I miss something? Thanks! Eugen [1] https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-location… [2] https://docs.ceph.com/en/latest/cephadm/services/#mounting-files-with-extra…

2 8

'ceph fs status' no longer works?
by Erich Weiler 03 May '24

03 May '24

Hi All, For a while now I've been using 'ceph fs status' to show current MDS active servers, filesystem status, etc. I recently took down my MDS servers and added RAM to them (one by one, so the filesystem stayed online). After doing that with my four MDS servers (I had two active and two standby), all looks OK, 'ceph -s' reports HEALTH_OK. But when I do 'ceph fs status' now, I get this: # ceph fs status Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1811, in _handle_command return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/status/module.py", line 109, in handle_fs_status assert metadata AssertionError This is on ceph 18.2.1 reef. This is very odd - can anyone think of a reason why 'ceph fs status' would stop working after taking each of the servers down for maintenance? The filesystem is online and working just fine however. This ceph instance is deployed via the cephadm method on RHEL 9.3, so the everything is containerized in podman. Thanks again, erich

4 6

NVME node disks maxed out during rebalance after adding to existing cluster
by Szabo, Istvan (Agoda) 02 May '24

02 May '24

Hi, I have slow heartbeat in front and back with the extra node added to the cluster and this occasionally causing slow ops and failed osd reports. I'm extending our cluster with +3 relatively differently configured servers compared to the original 12. Our cluster (latest octopus) is an objectstore cluster with 12x identical node (8x15.3 TB SSD inside with 4osd on each, 512GB mem, 96 vcore cpu ...) and it hosts a 4:2 EC data pool. The +3 nodes - currently done only 1 and now the 2nd is in progress but have the issue during rebalance - have 8x 15.3TB NVME drives with 4x osd on each. The NVME drive specification:https://i.ibb.co/BVmLKnf/currentnvme.png [https://i.ibb.co/BVmLKnf/currentnvme.png] The old server SSD spec: https://i.ibb.co/dkD3VKx/oldssd.png [https://i.ibb.co/dkD3VKx/oldssd.png] Iostat on new nvme: https://i.ibb.co/PF0hrVW/iostat.png [https://i.ibb.co/PF0hrVW/iostat.png] Rebalance is ongoing with the slowest option like max_backfill/op priority/max recovery = 1 But it generates a very huge iowait, seems like the disk not fast enough (but why in the previous node didn't have this issue)? Here is the metrics about the disk which is running the backfill/rebalance now (FYI we have 3.2B objects in the cluster): https://i.ibb.co/LNXCRbj/disks.png [https://i.ibb.co/LNXCRbj/disks.png] Wonder what I'm missing or how this can happen? Here you can see gigantic latencies, failed osds and slow ops: https://i.ibb.co/Jn0sj9g/laten.png [https://i.ibb.co/Jn0sj9g/laten.png] Thank you for your help ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Reconstructing an OSD server when the boot OS is corrupted
by Peter van Heusden 02 May '24

02 May '24

Dear Ceph Community We have 5 OSD servers running Ceph v15.2.17. The host operating system is Ubuntu 20.04. One of the servers has suffered corruption to its boot operating system. Using a system rescue disk it is possible to mount the root filesystem but it is not possible to boot the operating system at the moment. The OSDs are configured with (spinning disk) data drives, WALs and DBs on partitions of SSDs, but from my examination of the filesystem the configuration in /var/lib/ceph appears to be corrupted. So my question is: what is the best option for repair going forward? Is it possible to do a clean install of the operating system and scan the existing drives in order to reconstruct the OSD configuration? Thank you, Peter P.S. the cause of the original corruption is likely due to an unplanned power outage, an event that hopefully will not recur.

7 9

service:mgr [ERROR] "Failed to apply:
by Roberto Maggi ＠ Debian 02 May '24

02 May '24

Hi you all, it is a couple of days I'm facing this problem. Although I already destroyed the cluster a couple of times I continuously get these error I instruct ceph to place 3 daemons ceph orch apply mgr 3 --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" but root@cephstage01:/# ceph -s cluster: id: 0827effc-302a-45a6-bb97-758d2a9e6835 health: HEALTH_ERR Failed to apply 6 service(s): alertmanager,grafana,mgr,mon,node-exporter,prometheus failed to probe daemons or devices 2 mgr modules have failed 1 mgr modules have recently crashed OSD count 0 < osd_pool_default_size 3 services: mon: 2 daemons, quorum cephstage01,cephstagedatanode01 (age 21m) mgr: cephstage01.ukwnlo(active, since 5h), standbys: cephstagedatanode01.vqbwbz osd: 0 root@cephstage01:/# ceph orch ls --service_name mgr --format yaml service_type: mgr service_name: mgr placement: hosts: - cephstage01:10.20.20.81 - cephstage02:10.20.20.82 - cephstage03:10.20.20.83 status: created: '2024-05-02T13:52:41.346547Z' last_refresh: '2024-05-02T14:06:31.823024Z' running: 2 size: 1 events: - 2024-05-02T13:51:49.993145Z service:mgr [INFO] "service was created" - '2024-05-02T13:53:45.566950Z service:mgr [ERROR] "Failed to apply: Cannot place <ServiceSpec for service_name=mgr> on cephstage02, cephstage03: Unknown hosts"' and when I check those hosts root@cephstage01:/# ceph cephadm check-host cephstage03 cephstage03 (None) ok docker (/usr/bin/docker) is present systemctl is present lvcreate is present Unit chrony.service is enabled and running Hostname "cephstage03" matches what is expected. Host looks OK they look fine. every time I recreate the cluster it gives me the same errors: does anybody have any idea? thanks in advance

2 1

Ceph Day NYC 2024 Slides
by Matt Vandermeulen 02 May '24

02 May '24

Hi folks! Thanks for a great Ceph Day event in NYC! I wanted to make sure I posted my slides before I forget (and encourage others to do the same). Feel free to reach out in the Ceph Slack https://ceph.io/en/community/connect/ How we Operate Ceph at Scale (DigitalOcean): - https://do-matt-ams3.ams3.digitaloceanspaces.com/2024%20Ceph%20Day%20NYC%20… - https://do-matt-sfo3.sfo3.digitaloceanspaces.com/2024%20Ceph%20Day%20NYC%20… Discards in Ceph (DigitalOcean): - https://do-matt-ams3.ams3.digitaloceanspaces.com/2024%20Ceph%20Day%20NYC%20… - https://do-matt-sfo3.sfo3.digitaloceanspaces.com/2024%20Ceph%20Day%20NYC%20… Matt

2 2

After dockerized ceph cluster to Pacific, the fsid changed in the output of 'ceph -s'
by wjsherry075＠outlook.com 02 May '24

02 May '24

Hello, I had a problem after I finished the 'cephadm adopt' from services to docker containers for mon and mgr. The fsid of `ceph -s` is not the same as the /etc/ceph/ceph.conf. The ceph.conf is correct, but `ceph -s` is incorrect. I followed the https://docs.ceph.com/en/quincy/cephadm/adoption/ ``` 2024-04-25T19:49:17.460652+0000 mgr.cloud-lab-test-mon01 (mgr.65113) 109 : cephadm [ERR] cephadm exited with an error code: 1, stderr:ERROR: fsid does not match ceph.conf Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection yield (conn, connr) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm code, '\n'.join(err))) orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:ERROR: fsid does not match ceph.conf #ceph health detail shows the warning: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices ``` Does anyone else have any ideas? Thanks, Sherry

2 1

Ceph client cluster compatibility
by Nima AbolhassanBeigi 02 May '24

02 May '24

Hi, We are trying to upgrade our OS version from ubuntu 18.04 to ubuntu 22.04. Our ceph cluster version is 16.2.13 (pacific). The problem is that the ubuntu packages for the ceph pacific release will not be supported for ubuntu 22.04. We were wondering if the ceph client (version 18.2, reef) on ubuntu 22.04 can work with lower version clusters. Thanks in advance Regards Nima

2 1