- ceph-users - lists.ceph.io

1 stray daemon(s) not managed by cephadm
by Michel Niyoyita 08 Nov '24

08 Nov '24

Hello team, I have run a ceph cluster using cephadm Reef , on ubuntu 22.04 , the cluster is composed by 3 mons and 3 hosts running OSDs, after deployment my cluster is in warning state with : [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm . I have run ceph heath detail and the out put is : HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon mon.ceph-osd3 on host ceph-osd3 not managed by cephadm could you please help to solve this problem? Best regards Michel

3 2

Fwd: Call for participation: Software Defined Storage devroom at FOSDEM 2025
by Jan Fajerski 08 Nov '24

08 Nov '24

----- Forwarded message from Jan Fajerski <jan(a)fajerski.name> ----- Date: Mon, 28 Oct 2024 17:31:49 +0100 From: Jan Fajerski <jan(a)fajerski.name> To: fosdem(a)lists.fosdem.org Cc: Subject: [devroom-managers] Call for participation: Software Defined Storage devroom at FOSDEM 2025 FOSDEM is a free software event that offers open source communities a place to meet, share ideas and collaborate. It is well known for being highly developer-oriented and in the past brought together 8000+ participants from all over the world. Its home is in the city of Brussels (Belgium). FOSDEM 2025 will take place as an in-person event during the weekend of February 1/2 2025. More details about the event can be found at http://fosdem.org/ ** Call For Participation The Software Defined Storage devroom will go into its 9. iteration for talks around Open Source Software Defined Storage projects, management tools and real world deployments. Presentation topics could include but are not limited too: - Your work on a SDS project like Ceph, Gluster, Garage, OpenEBS, CORTX or Longhorn - Your work on or with SDS related projects like Samba, OpenStack SWIFT or Container Storage Interface - Management tools for SDS deployments - Monitoring tools for SDS clusters ** Important dates: - Dec 1, 2024: submission deadline for talk proposals - Dec 8, 2024: announcement of the final schedule - Feb 1, 2025: Software Defined Storage dev room Talk proposals will be reviewed by a steering committee: - Niels de Vos (IBM) - Jan Fajerski (Red Hat) - Hugo Gonzalez Labrador (CERN) - Abhishek Lekshmanan (CERN) - Richard Bachmann (CERN) We also welcome additional volunteers to help with making this devroom a success. Please submit talks at https://fosdem.org/submit. - Important! Select the "Software Defined Storage devroom" track. Hope to hear from you soon! And please forward this announcement. If you have any further questions, please write to the mailing list at storage-devroom(a)lists.fosdem.org and we will try to answer as soon as possible. Thanks! _______________________________________________ devroom-managers mailing list devroom-managers(a)lists.fosdem.org https://lists.fosdem.org/listinfo/devroom-managers ----- End forwarded message -----

1 0

Re: osd removal leaves 'stray daemon'
by Frédéric Nass 08 Nov '24

08 Nov '24

Hi, We're encountering this unexpected behavior as well. This tracker [1] was created 4 months ago. Regards, Frédéric. [1] https://tracker.ceph.com/issues/67018 ----- Le 6 Déc 22, à 8:41, Holger Naundorf naundorf(a)rz.uni-kiel.de a écrit : > Hello, > a mgr failover did not change the situation - the osd still shows up in > the 'ceph node ls' - I assume that this is more or less 'working as > intended' as I did ask for the OSD to be kept in the CRUSH map to be > replacd later - but as we are still not so experienced with Ceph here I > wanted to get some input from other sites. > > Regards, > Holger > > On 30.11.22 16:28, Adam King wrote: >> I typically don't see this when I do OSD replacement. If you do a mgr >> failover ("ceph mgr fail") and wait a few minutes does this still show >> up? The stray daemon/host warning is roughly equivalent to comparing the >> daemons in `ceph node ls` and `ceph orch ps` and seeing if there's >> anything in the former but not the latter. Sometimes I have seen the mgr >> will have some out of data info in the node ls and a failover will >> refresh it. >> >> On Fri, Nov 25, 2022 at 6:07 AM Holger Naundorf <naundorf(a)rz.uni-kiel.de >> <mailto:naundorf@rz.uni-kiel.de>> wrote: >> >> Hello, >> I have a question about osd removal/replacement: >> >> I just removed an osd where the disk was still running but had read >> errors, leading to failed deep scrubs - as the intent is to replace >> this >> as soon as we manage to get a spare I removed it with the >> '--replace' flag: >> >> # ceph orch osd rm 224 --replace >> >> After all placement groups are evacuated I now have 1 osd down/out >> and showing as 'destroyed': >> >> # ceph osd tree >> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF >> (...) >> 214 hdd 14.55269 osd.214 up 1.00000 1.00000 >> 224 hdd 14.55269 osd.224 destroyed 0 1.00000 >> 234 hdd 14.55269 osd.234 up 1.00000 1.00000 >> (...) >> >> All as expected - but now the health check complains that the >> (destroyed) osd is not managed: >> >> # ceph health detail >> HEALTH_WARN 1 stray daemon(s) not managed by cephadm >> [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm >> stray daemon osd.224 on host ceph19 not managed by cephadm >> >> Is this expected behaviour and I have to live with the yellow check >> until we get a replacement disk and recreate the osd or did something >> not finish correctly? >> >> Regards, >> Holger >> >> -- >> Dr. Holger Naundorf >> Christian-Albrechts-Universität zu Kiel >> Rechenzentrum / HPC / Server und Storage >> Tel: +49 431 880-1990 >> Fax: +49 431 880-1523 >> naundorf(a)rz.uni-kiel.de <mailto:naundorf@rz.uni-kiel.de> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> <mailto:ceph-users@ceph.io> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> <mailto:ceph-users-leave@ceph.io> >> > > -- > Dr. Holger Naundorf > Christian-Albrechts-Universität zu Kiel > Rechenzentrum / HPC / Server und Storage > Tel: +49 431 880-1990 > Fax: +49 431 880-1523 > naundorf(a)rz.uni-kiel.de > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 4

Squid 19.2.0 balancer causes restful requests to be lost
by Chris Palmer 07 Nov '24

07 Nov '24

I've just upgraded a test cluster from 18.2.4 to 19.2.0. Package install on centos 9 stream. Very smooth upgrade. Only one problem so far... The MGR restful api calls work fine. EXCEPT whenever the balancer kicks in to find any new plans. During the few seconds that the balancer takes to run, all REST calls seem to be completely dropped. The MGR log file normally logs the POST requests, but the ones during these few seconds don't appear at all. This causes our monitoring to keep raising alarms. The cluster is in a completely stable state, HEALTH_OK, very little activity, just the occasional scrubs. We use the restful API for monitoring (using the Ceph for Zabbix Agent 2 plugin, as Zabbix is the over-arching monitoring platform in the data centre). I haven't yet checked the memory leak problems that we (like many) were having, because I have been chasing this new problem. The problem is quite repeatable. To diagnose I use the zabbix_get utility to query every second. Whenever the MGR log file shows the balancer kick in the REST requests time out (after 3 seconds - not sure whether the utility or the MGR is timing them out - I suspect the utility). They normally complete after a small fraction of a second. With the balancer disabled the REST interface works reliably again. The problem does not occur pre-squid. Anyone any ideas, or shall I raise a bug? Thanks, Chris

6 9

Recreate Destroyed OSD
by Dave Hall 07 Nov '24

07 Nov '24

Hello. Sorry if it appears that I am reposting the same issue under a different topic. However, I feel that the problem has moved and I now have different questions. At this point I have, I believe, removed all traces of OSD.12 from my cluster - based on steps in the Reef docs at https://docs.ceph.com/en/reef/rados/operations/add-or-rm-osds/#. I have further located and removed the WAL/DB LV on an associated NVMe drive (shared with 3 other OSDs). I don't believe the instructions for replacing an OSD (ceph-volume lvm prepare) still apply, so I have been trying to work with the instructions under ADDING AN OSD (MANUAL). However, since my installation is containerized (Podman), it is unclear which steps should be issued on the host and which within 'cephadm shell'. There is also another ambiguity: In step 3 the instruction is to 'mkfs -t {fstype}' and then to 'mount -o user_xattr'. However, which fs type? After this, in step 4, the 'ceph-osd -i {osd-id} --mkfs --mkkey' gets throws errors about the keyring file. So, are these the right instructions to be using in a containerized installation? Are there, in general, alternate documents for containerized installations? Lastly, the above cited instructions don't say anything about the separate WAL/DB LV. Please advise. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

5 17

[RGW] Enable per user/bucket performance counters
by Nathan MALO 07 Nov '24

07 Nov '24

Hi, I come to you with a quite simple question. I am running a REEF cluster in 18.2.2, deployed in k3s via rook 1.14.8. I want to enable per user and per bucket performance counters on my RGW. According to this doc https://docs.ceph.com/en/latest/radosgw/metrics/#user-bucket-counter-caches I need to run ceph config set client.rgw rgw_{user,bucket}_counters_cache true. But when I run the commands in the ceph toolbox, I end up with the following errors: bash-4.4$ ceph config set client.rgw rgw_user_counters_cache true Error EINVAL: unrecognized config option 'rgw_user_counters_cache' bash-4.4$ ceph config set client.rgw rgw_bucket_counters_cache true Error EINVAL: unrecognized config option 'rgw_bucket_counters_cache' This doc https://www.ibm.com/docs/en/storage-ceph/7?topic=counters-viewing-performan… mentions a different command regarding the client.rgw. So I did the following: bash-4.4$ ceph auth list | grep rgw client.bootstrap-rgw caps: [mon] allow profile bootstrap-rgw client.rgw.ceph.s3.a bash-4.4$ ceph config set client.rgw.ceph.s3.a rgw_user_counters_cache true Error EINVAL: unrecognized config option 'rgw_user_counters_cache' bash-4.4$ ceph config set client.rgw.ceph.s3.a rgw_bucket_counters_cache true Error EINVAL: unrecognized config option 'rgw_bucket_counters_cache' When I run all this commands with --force, the keys are created. I restarted my RGWs pods but the keys do not seem to be taken into account as I do not have any ceph_rgw_op* metric. Did anyone successfully managed to enable this ? What am I missing ? Thank your for your help !

2 1

Unable to add OSD
by tpDev Tester 07 Nov '24

07 Nov '24

Hi, I try to add OSDs to my new Cluster (Ubuntu 24.04 + podman), Four devices are listed as available: root@ceph-1:~# ceph-volume inventory Device Path Size Device nodes rotates available Model name /dev/nvme0n1 1.82 TB nvme0n1 False True KINGSTON SNV2S2000G /dev/sda 1.82 TB sda True True WDC WD20EFZX-68A /dev/sdb 16.37 TB sdb True True TOSHIBA MG09ACA1 /dev/sdc 931.51 GB sdc True True WDC WD10EFRX-68J but none of them can be added: root@ceph-1:~# ceph orch daemon add osd ceph-1:/dev/sda Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1862, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 184, in handle_command return dispatch[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 499, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 120, in <lambda> wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 109, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/orchestrator/module.py", line 1374, in _daemon_add_osd raise_if_exception(completion) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 241, in raise_if_exception raise e RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/mon.ceph-1/config Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 5579, in <module> File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 5567, in main File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 409, in _infer_config File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 437, in _infer_image File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 3288, in command_ceph_volume File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 918, in get_container_mounts_for_type File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/daemons/ceph.py", line 422, in get_ceph_mounts_for_type File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/host_facts.py", line 760, in selinux_enabled File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/host_facts.py", line 743, in kernel_security File "/var/lib/ceph/16a56cdf-9bb4-11ef-b530-001e06456e10/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/host_facts.py", line 722, in _fetch_apparmor ValueError: too many values to unpack (expected 2) I found some hints with apparmor and MongoDB, but that does not help in this case. Any help appreciated. Kind regards Thomas

3 3

Ceph Multisite Version Compatibility
by Alex Hussein-Kershaw (HE/HIM) 06 Nov '24

06 Nov '24

Hi folks. I'm looking for some guidance on RGW multisite version sync compatibility. Particularly between Octopus and Squid. Context is I have two sites in a multisite pair replicating all S3 data. One is on Squid, one is on Octopus. Should I expect the multisite sync to just work between these versions? I'm observing that both sites "radosgw-admin sync status" reports that we're in sync, but objects from the Octopus zone are not replicated to the Squid zone (but the opposite direction is fine). Might be this just isn't a valid setup, but failing to a reference that claims something like "must be within +/- 1 version of the other zones"). Thanks, Alex

2 4

Backfill full osds
by Szabo, Istvan (Agoda) 06 Nov '24

06 Nov '24

Hi, We have a 60% full cluster where unfortunately pg scaler hasn't beek used and during the time generated gigantic pgs which when moved from 1 osd to the other it gets full and blocks write. Tried to add 2 new node but before they would he utilised other osd gets full. Trying now backfill full osds to change weight and crush reweight but feels like running after something which can't be catched. What's the best practice to come out of this situation? Remap pg manually to empty osd with some remapper tool? I scare it will not check pg size. Increasing now the pg will fill up I guess again because cleaning happens after pg increase. Istvan ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 4

Pacific: mgr loses osd removal queue
by Eugen Block 06 Nov '24

06 Nov '24

Hi, I'm not sure if this has been asked before, or if there's an existing tracker issue already. It's difficult to reproduce it on my lab clusters. I'm testing some new SSD OSDs on a Pacific cluster (16.2.15) and noticed that if we instruct the orchestrator to remove two or three OSDs (issuing the command 'ceph orch osd rm {ID}' a couple of times), it eventually only removes the first in the queue. I've been watching 'ceph orch osd rm status' to see the progress, and then the rest of the queued OSDs suddenly vanish from the status and never get removed. Then I have to issue the command again. If I remove one OSD by one, so no others in the queue, they are all successfully removed. Why is this happening? Is this a bug in Pacific? Has someone seen this in newer releases? Thanks! Eugen

1 0