November 2024 - ceph-users

Unable to add OSD
by tpDev Tester 07 Nov '24

07 Nov '24

Hi, I try to add OSDs to my new Cluster (Ubuntu 24.04 + podman), Four devices are listed as available: root@ceph-1:~# ceph-volume inventory Device Path Size Device nodes rotates available Model name /dev/nvme0n1 1.82 TB nvme0n1 False True KINGSTON SNV2S2000G /dev/sda 1.82 TB sda True True WDC WD20EFZX-68A /dev/sdb 16.37 TB sdb True True TOSHIBA MG09ACA1 /dev/sdc 931.51 GB sdc True True WDC WD10EFRX-68J but none of them can be added: root@ceph-1:~# ceph orch daemon add osd ceph-1:/dev/sda Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1862, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 184, in handle_command return dispatch[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 499, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 120, in <lambda> wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 109, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/orchestrator/module.py", line 1374, in _daemon_add_osd raise_if_exception(completion) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 241, in raise_if_exception raise e RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/mon.ceph-1/config Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 5579, in <module> File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 5567, in main File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 409, in _infer_config File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 437, in _infer_image File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 3288, in command_ceph_volume File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 918, in get_container_mounts_for_type File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/daemons/ceph.py", line 422, in get_ceph_mounts_for_type File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/host_facts.py", line 760, in selinux_enabled File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/host_facts.py", line 743, in kernel_security File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/host_facts.py", line 722, in _fetch_apparmor ValueError: too many values to unpack (expected 2) I found some hints with apparmor and MongoDB, but that does not help in this case. Any help appreciated. Kind regards Thomas

3 3

Ceph Multisite Version Compatibility
by Alex Hussein-Kershaw (HE/HIM) 06 Nov '24

06 Nov '24

Hi folks. I'm looking for some guidance on RGW multisite version sync compatibility. Particularly between Octopus and Squid. Context is I have two sites in a multisite pair replicating all S3 data. One is on Squid, one is on Octopus. Should I expect the multisite sync to just work between these versions? I'm observing that both sites "radosgw-admin sync status" reports that we're in sync, but objects from the Octopus zone are not replicated to the Squid zone (but the opposite direction is fine). Might be this just isn't a valid setup, but failing to a reference that claims something like "must be within +/- 1 version of the other zones"). Thanks, Alex

2 4

Backfill full osds
by Szabo, Istvan (Agoda) 06 Nov '24

06 Nov '24

Hi, We have a 60% full cluster where unfortunately pg scaler hasn't beek used and during the time generated gigantic pgs which when moved from 1 osd to the other it gets full and blocks write. Tried to add 2 new node but before they would he utilised other osd gets full. Trying now backfill full osds to change weight and crush reweight but feels like running after something which can't be catched. What's the best practice to come out of this situation? Remap pg manually to empty osd with some remapper tool? I scare it will not check pg size. Increasing now the pg will fill up I guess again because cleaning happens after pg increase. Istvan ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 4

Pacific: mgr loses osd removal queue
by Eugen Block 06 Nov '24

06 Nov '24

Hi, I'm not sure if this has been asked before, or if there's an existing tracker issue already. It's difficult to reproduce it on my lab clusters. I'm testing some new SSD OSDs on a Pacific cluster (16.2.15) and noticed that if we instruct the orchestrator to remove two or three OSDs (issuing the command 'ceph orch osd rm {ID}' a couple of times), it eventually only removes the first in the queue. I've been watching 'ceph orch osd rm status' to see the progress, and then the rest of the queued OSDs suddenly vanish from the status and never get removed. Then I have to issue the command again. If I remove one OSD by one, so no others in the queue, they are all successfully removed. Why is this happening? Is this a bug in Pacific? Has someone seen this in newer releases? Thanks! Eugen

1 0

Setting temporary CRUSH "constraint" for planned cross-datacenter downtime
by Niklas Hambüchen 06 Nov '24

06 Nov '24

My server provider usually does infrastructure maintenance and planned downtimes on a per-datacenter-building granularity, and thus I have a Ceph cluster with that set as the "datacenter" failure domain in CRUSH. However, it now has a planned maintenance that affects two buildings simultaneously. How can I tell Ceph to balance data such that DC1 and DC2 specifically should be treated as the same failure domain? (All other datacenters remain independent; ideally with minimal necessary rebalancing.) Thanks!

3 3

OSD refuse to start
by Albert Shih 06 Nov '24

06 Nov '24

Hi everyone, I'm running currently ceph reef 18.2.4 with podman with Debian 12 Today I've to reboot the cluster (firmware alerts), everything goes as the plan but....one osd on one server won't start. I try everything in https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-osd/ but no luck. I don't find any message in the dmesg. Zero message with journalctl zero message with systemctl status At the end I reboot once more time the server and everything work fine again. Is anyone encounter something like that ? Is that “normal” Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: mar. 05 nov. 2024 15:11:40 CET

2 2

Ceph Steering Committee (CLT) Meeting Minutes 2024-11-04
by Neha Ojha 05 Nov '24

05 Nov '24

Hi everybody, These are the highlights from this week's meeting: - Election: - CEC Election is live: https://vote.heliosvoting.org/helios/elections/b2fa20e6-1a5d-4f49-a747-0f8f… - https://pad.ceph.com/p/css-vote-2024q4 - Feedback from the Ceph Ambassadors meeting - October 2024: - https://pad.ceph.com/p/ceph-ambassadors-meeting - 17.2.8 - ongoing discussion with CERN to help make el8 ceph packages available for 17.2.8 - QE validation runs in dev leads review phase - Ceph Tentacle Award - criteria/logistics needs to be discussed - Ceph Windows client builds - cloudbase is reviewing if they will continue this next year - Ceph Foundation had funded it in 2021/22, no longer funding it - Shaman also builds, but we don't distribute. (Signing costs $$) - Usage metrics unknown, the Ceph Foundation has it on agenda for their 2025 budget discussion Thanks, Neha

1 0

Assistance Required: Ceph OSD Out of Memory (OOM) Issue
by Md Mosharaf Hossain 03 Nov '24

03 Nov '24

Dear Ceph Community, I hope this message finds you well. I am encountering an out-of-memory (OOM) issue with one of my Ceph OSDs, which is repeatedly getting killed by the OOM killer on my system. Below are the relevant details from the log: *OOM Log*: [Wed Oct 30 13:14:48 2024] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/system-ceph\x2dosd.slice,task=ceph-osd,pid=6213,uid=64045 [Wed Oct 30 13:14:48 2024] Out of memory: Killed process 6213 (ceph-osd) total-vm:216486528kB, anon-rss:211821164kB, file-rss:0kB, shmem-rss:0kB, UID:64045 pgtables:418836kB oom_score_adj:0 [Wed Oct 30 13:14:58 2024] oom_reaper: reaped process 6213 (ceph-osd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB *Ceph OSD Log*: 2024-10-30T13:15:30.207+0600 7f906c74dd80 0 _get_class not permitted to load lua 2024-10-30T13:15:30.211+0600 7f906c74dd80 0 <cls> /build/ceph-15.2.17/src/cls/hello/cls_hello.cc:312: loading cls_hello 2024-10-30T13:15:30.215+0600 7f906c74dd80 0 _get_class not permitted to load kvs 2024-10-30T13:15:30.219+0600 7f906c74dd80 0 _get_class not permitted to load queue 2024-10-30T13:15:30.223+0600 7f906c74dd80 0 <cls> /build/ceph-15.2.17/src/cls/cephfs/cls_cephfs.cc:198: loading cephfs 2024-10-30T13:15:30.223+0600 7f906c74dd80 0 osd.13 299547 crush map has features 432629239337189376, adjusting msgr requires for clients 2024-10-30T13:15:30.223+0600 7f906c74dd80 0 osd.13 299547 crush map has features 432629239337189376 was 8705, adjusting msgr requires for mons 2024-10-30T13:15:30.223+0600 7f906c74dd80 0 osd.13 299547 crush map has features 3314933000854323200, adjusting msgr requires for osds 2024-10-30T13:15:30.223+0600 7f906c74dd80 1 osd.13 299547 check_osdmap_features require_osd_release unknown -> octopus 2024-10-30T13:15:31.023+0600 7f906c74dd80 0 osd.13 299547 load_pgs *Environment Details*: - Ceph Version: 15.2.17 (Octopus) - OSD: osd.13 - Kernel: Linux kernel version It seems that the OSD process is consuming a substantial amount of memory (total-vm: 216486528kB, anon-rss: 211821164kB), leading to OOM kills on the node. The OSD service restarts but continues to showing consumption excessive memory and OSD get down. Could you please provide guidance or suggestions on how to mitigate this issue? Are there any known memory management settings, configuration adjustments, or OSD-specific tuning parameters that could help prevent this from recurring? Any help would be greatly appreciated. Thank you for your time and assistance! Regards Mosharaf Hossain Manager, Product Development Bangladesh Online (BOL) Level 8, SAM Tower, Plot 4, Road 22, Gulshan 1, Dhaka 1212, Bangladesh Tel: +880 9609 000 999, +880 2 58815559, Ext: 14191, Fax: +880 2 2222 95757 Cell: +880 1787 680828, Web: www.bol-online.com

2 2

MDS and stretched clusters
by Sake Ceph 01 Nov '24

01 Nov '24

Hi all We deployed successfully a stretched cluster and all is working fine. But is it possible to assign the active MDS services in one DC and the standby-replay in the other? We're running 18.2.4, deployed via cephadm. Using 4 MDS servers with 2 active MDS on pinnend ranks and 2 in standby-replay mode.

5 8