- ceph-users - lists.ceph.io

Slow ops during recovery for RGW index pool only when degraded OSD is primary
by Wesley Dillingham 04 Apr '24

04 Apr '24

I am fighting an issue on an 18.2.0 cluster where a restart of an OSD which supports the RGW index pool causes crippling slow ops. If the OSD is marked with primary-affinity of 0 prior to the OSD restart no slow ops are observed. If the OSD has a primary affinity of 1 slow ops occur. The slow ops only occur during the recovery period of the OMAP data and further only occur when client activity is allowed to pass to the cluster. Luckily I am able to test this during periods when I can disable all client activity at the upstream proxy. Given the behavior of the primary affinity changes preventing the slow ops I think this may be a case of recovery being more detrimental than backfill. I am thinking that causing an pg_temp acting set by forcing backfill may be the right method to mitigate the issue. [1] I believe that reducing the PG log entries for these OSDs would accomplish that but I am also thinking a tuning of osd_async_recovery_min_cost [2] may also accomplish something similar. Not sure the appropriate tuning for that config at this point or if there may be a better approach. Seeking any input here. Further if this issue sounds familiar or sounds like another condition within the OSD may be at hand I would be interested in hearing your input or thoughts. Thanks! [1] https://docs.ceph.com/en/latest/dev/peering/#concepts [2] https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/#confval… Respectfully, *Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes(a)wesdillingham.com

3 5

question about rbd_read_from_replica_policy
by Noah Elias Feldt 04 Apr '24

04 Apr '24

Hello, I have a question about a setting for RBD. How exactly does "rbd_read_from_replica_policy" with the value "localize" work? According to the RBD documentation, read operations will be sent to the closest OSD as determined by the CRUSH map. How does the client know exactly which OSD I am closest to? The client is not in the CRUSH map. I can't find much more information about it. How does this work? Thanks noah feldt infrastrutur _____ [Inline image OWAPstImg850138] Mittwald CM Service GmbH & Co. KG Königsberger Straße 4-6 32339 Espelkamp Tel.: 05772 / 293-100 Fax: 05772 / 293-333 support(a)mittwald.de<mailto:support@mittwald.de> https://www.mittwald.de<https://www.mittwald.de/> Geschäftsführer: Robert Meyer, Florian Jürgens USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen Informationen zur Datenverarbeitung im Rahmen unserer Geschäftstätigkeit gemäß Art. 13-14 DSGVO sind unter www.mittwald.de/ds<http://www.mittwald.de/ds> abrufbar. ________________________________

3 2

RBD image metric
by Szabo, Istvan (Agoda) 04 Apr '24

04 Apr '24

Hi, Trying to pull out some metrics from ceph about the rbd images sizes but haven't found anything only pool related metrics. Wonder is there any metric about images or I need to create by myself to collect it with some third party tool? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 3

CEPHADM_HOST_CHECK_FAILED
by arnoud＠fuga.cloud 04 Apr '24

04 Apr '24

Hi, I’ve added some new nodes to our Ceph cluster. Only did the host add, had not added the OSD’s yet. Due to a configuration error I had to reinstall some of them. But I forgot to remove the nodes from Ceph first. I did a “ceph orch host rm peon7 --offline —force” before re-adding them to the cluster. All the nodes are showing up in the host list (all the peons are the new ones): # ceph orch host ls HOST ADDR LABELS STATUS ceph1 10.103.0.71 ceph2 10.103.0.72 ceph3 10.103.0.73 ceph4 10.103.0.74 compute1 10.103.0.11 compute2 10.103.0.12 compute3 10.103.0.13 compute4 10.103.0.14 controller1 10.103.0.8 controller2 10.103.0.9 controller3 10.103.0.10 peon1 10.103.0.41 peon2 10.103.0.42 peon3 10.103.0.43 peon4 10.103.0.44 peon5 10.103.0.45 peon6 10.103.0.46 peon7 10.103.0.47 peon8 10.103.0.48 peon9 10.103.0.49 peon10 10.103.0.50 peon12 10.103.0.52 peon13 10.103.0.53 peon14 10.103.0.54 peon15 10.103.0.55 peon16 10.103.0.56 But Ceph status still shows an error, which I can’t seem to get rid off. [WRN] CEPHADM_HOST_CHECK_FAILED: 2 hosts fail cephadm check host peon7 (10.103.0.47) failed check: Can't communicate with remote host `10.103.0.47`, possibly because python3 is not installed there or you are missing NOPASSWD in sudoers. [Errno 113] Connect call failed ('10.103.0.47', 22) host peon12 (10.103.0.52) failed check: Can't communicate with remote host `10.103.0.52`, possibly because python3 is not installed there or you are missing NOPASSWD in sudoers. [Errno 113] Connect call failed ('10.103.0.52', 22) [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: 'peon7' Module 'cephadm' has failed: ‘peon7' From the mgr log: Apr 04 08:33:46 controller2 bash[4031857]: debug 2024-04-04T08:33:46.876+0000 7f2bb5710700 -1 mgr.server reply reply (5) Input/output error Module 'cephadm' has experienced an error and cannot handle commands: 'peon7' Any idea how to clear this error? # ceph --version ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Regards, Arnoud de Jonge.

2 1

RGW services crashing randomly with same message
by Reid Guyett 03 Apr '24

03 Apr '24

Hello, We are currently experiencing a lot of rgw service crashes that all seem to terminate with the same message. We have kept our RGW services at 17.2.5 but the rest of the cluster is 17.2.7 due to a bug introduced in 17.2.7. terminate called after throwing an instance of > 'ceph::buffer::v15_2_0::end_of_buffer' > what(): End of buffer > *** Caught signal (Aborted) ** > in thread 7fcfd1772700 thread_name:radosgw > ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy > (stable) > What would be the best way to figure out what is causing the error? The rest of the crash is quite long in our logs with 10k lines of history. Thanks, Reid

1 0

cephadm auto disk preparation and OSD installation incomplete
by Kuhring, Mathias 03 Apr '24

03 Apr '24

Dear ceph community, We have trouble with new disks not being properly prepared resp. OSDs not being fully installed by cephadm. We just added one new node each with ~40 HDDs each to two of our ceph clusters. In one cluster all but 5 disks got installed automatically. In the other none got installed. We are on ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) on both clusters. (I haven't added new disks since the last upgrade if I recall correctly). This is our OSD service definition: ``` 0|0[root@ceph-3-10 ~]# ceph orch ls osd --export service_type: osd service_id: all-available-devices service_name: osd.all-available-devices placement: host_pattern: '*' spec: data_devices: all: true filter_logic: AND objectstore: bluestore --- service_type: osd service_id: unmanaged service_name: osd.unmanaged unmanaged: true spec: filter_logic: AND objectstore: bluestore ``` Usually, new disks are installed properly (as expected due to all-available-devices). This time, I can see that LVs were created (via `lsblk`, `lvs`, `cephadm ceph-volume lvm list`). And OSDs are entered to the crushmap. However, they are not assigned to a host yet, nor do they have a type or weight, e.g.: ``` 0|0[root@ceph-2-10 ~]# ceph osd tree | grep "0 osd" 518 0 osd.518 down 0 1.00000 519 0 osd.519 down 0 1.00000 520 0 osd.520 down 0 1.00000 521 0 osd.521 down 0 1.00000 522 0 osd.522 down 0 1.00000 ``` And there is also no OSD daemon created (no docker container). So, OSD creation is somehow stuck halfway. I thought of fully cleaning up the OSD/disks. Hopping cephadm might pick them up properly next time. Just zapping was not possible, e.g. `cephadm ceph-volume lvm zap --destroy /dev/sdab` results in these errors: ``` /usr/bin/docker: stderr stderr: wipefs: error: /dev/sdab: probing initialization failed: Device or resource busy /usr/bin/docker: stderr --> failed to wipefs device, will try again to workaround probable race condition ``` So, I cleaned up more manually with purging them from crush and "resetting" disk and LV with dd and dmsetup, resp.: ``` ceph osd purge 480 --force dd if=/dev/zero of=/dev/sdab bs=1M count=1 dmsetup remove ceph--e10e0f08--8705--441a--8caa--4590de22a611-osd--block--d464211c--f513--4513--86c1--c7ad63e6c142 ``` ceph-volume still reported the old volumes, but then zapping actually got rid of them (only cleaned out the left-over entries, I guess). Now, cephadm was able to get one OSD up, when I did this cleanup for only one disk. When I did it in bulk for the rest, they all got stuck again the same way. Looking into ceph-volume logs (here for osd.522 as representative): ``` 0|0[root@ceph-2-11 /var/log/ceph/55633ec3-6c0c-4a02-990c-0f87e0f7a01f]# ll *20240316 -rw-r--r-- 1 ceph ceph 613789 Mar 14 17:10 ceph-osd.522.log-20240316 -rw-r--r-- 1 root root 42473553 Mar 16 03:13 ceph-volume.log-20240316 ``` ceph-volume only reports keyring creation: ``` [2024-03-14 16:10:19,509][ceph_volume.util.prepare][INFO ] Creating keyring file for osd.522 [2024-03-14 16:10:19,510][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-522/keyring --create-keyring --name osd.522 --add-key AQBfIfNlinc7EBAAHeFicrjmLEjRPGSjuFuLiQ== ``` In the OSD logs I found a couple of these, but don't know if they are related: ``` 2024-03-14T16:10:54.706+0000 7fab26988540 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) ``` Has anyone seen this behaviour before? Or could tell me where I should look next to troubleshoot this (which logs)? Any help is appreciated. Best Wishes, Mathias

2 4

quincy-> reef upgrade non-cephadm
by Christopher Durham 03 Apr '24

03 Apr '24

Hi, I am upgrading my test cluster from 17.2.6 (quincy) to 18.2.2 (reef). As it was an rpm install, i am following the directions here: Reef — Ceph Documentation | | | | Reef — Ceph Documentation | | | The upgrade worked, but I have some observations and questions before I move to my production cluster: 1. I see no systemd units with the fsid in them, as described in the document above. Both before and after the upgrade, my mon and other units are: ceph-mon@<server>.serviceceph-osd(a)[N].service etc Should I be concerned? 2. Does order matter? Based on past upgrades, I do not think so, but I wanted to be sure. For example, can I update: mon/mds/radosgw/mgrs first, then afterwards update the osds? This is what i have done in previous updates and and all was well. 3. Again on order, if a server serves say, a mon and mds, I can't really easily update one without the other, based on shared libraries and such. It appears that that is ok, based on my test cluster, but wanted to be sure. Again if an mds is one of the servers to update, I know I have to updatethe remaining one after max_mds is set to 1 and others are stopped, first. 4. After upgrade of my mgr node I get: "Module [several module names] has missing NOTIFY_TYPES member" in ceph-mgr.<server>.log But the mgr starts up eventually The system is Rocky Linux 8.9 Thanks for any thoughts -Chris

2 1

ceph orchestrator for osds
by Jeffrey Turmelle 03 Apr '24

03 Apr '24

Running on Octopus: While attempting to install a bunch of new OSDs on multiple hosts, I ran some ceph orchestrator commands to install them, such as ceph orch apply osd --all-available-devices ceph orch apply osd -I HDD_drive_group.yaml I assumed these were just helper processes, and they would be short-lived. In fact, they didn’t actually work and I ended up installing each drive by hand like this: ceph orch daemon add osd ceph4.iri.columbia.edu:/dev/sdag However, now I have these services running: # ceph orch ls --service-type=osd NAME RUNNING REFRESHED AGE PLACEMENT IMAGE NAME IMAGE ID osd.HDD_drive_group 2/2 7m ago 6w ceph[456].iri.columbia.edu docker.io/ceph/ceph:v15 2cf504fded39 osd.None 54/0 7m ago - <unmanaged> docker.io/ceph/ceph:v15 2cf504fded39 osd.all-available-devices 1/0 7m ago - <unmanaged> docker.io/ceph/ceph:v15 2cf504fded39 I’m certain none of these actually created any of my running OSD daemons, but I’m not sure if it’s ok to remove them. For example: ceph orch daemon rm osd.all-available-devices ceph orch daemon rm osd.HDD_drive_group ceph orch daemon rm osd.None Does anyone have any insight to this? I can just leave them there, they don’t seem to be doing anything, but on the other hand, I don’t want any new devices to be automatically loaded or any other unintended consequences of these. Thanks for any guidance, Jeff Turmelle International Research Institute for Climate & Society <https://iri.columbia.edu/> The Climate School <https://climate.columbia.edu/> at Columbia University <https://columbia.edu/>

2 1

Re: Multi-MDS
by quaglio＠bol.com.br 03 Apr '24

03 Apr '24

1 0

IO500 CFS ISC 2024
by IO500 Committee 03 Apr '24

03 Apr '24

Call for Submission Stabilization Period: Monday, April 1st - Friday, April 15th, 2024 Submission Deadline: Tuesday, May 3rd, 2024 AoE The IO500 is now accepting and encouraging submissions for the upcoming 14th semi-annual IO500 Production and Research lists, in conjunction with ISC24. Once again, we are also accepting submissions to both the Production and Research 10 Client Node Challenges to encourage the submission of small scale results. View the requirements for submitting to each list on the IO500 Webpage. The new ranked lists will be announced at the BoF [1]. We hope to see many new results. Background Following the success of the Top500 in collecting and analyzing historical trends in supercomputer technology and evolution, the IO500 was created in 2017, published its first list at SC17, and has grown continually since then. The benchmarks represent community accepted standards, including being used in Request for Proposals for new HPC platforms. The benchmarks showcase the IO access pattern extremes giving a full picture of storage system potential performance. The list is about much more than just the raw rank; all submissions help the community by collecting and publishing a wider corpus of data. The multi-fold goals of the benchmark suite are as follows: - Represent naive and optimized access patterns for the execution of a rich variety of HPC applications, their achievable performance and the documentation of how the numbers are achieved. - Support small to extreme-scale Research and Production HPC systems using flexible storage APIs - Maximizing simplicity in running the benchmark suite while offering tunable parameters Specifically, the benchmark suite includes a hero-run of both IOR and mdtest configured however possible to maximize performance and establish an upper-bound for performance. It also includes runs with highly prescribed parameters in an attempt to determine a lower performance bound. Finally, it includes a namespace search as this has been determined to be a highly sought-after feature in HPC storage systems that has historically not been well-measured. Supported Storage APIs are those that are part of IOR and mdtest. Extending these tools with a public pull request can be done to enable new storage APIs. The goals of the community are also multi-fold: 1. Gather historical data for the sake of analysis and to aid predictions of storage futures 2. Collect tuning information to share valuable performance optimizations across the community 3. Encourage vendors and designers to optimize for workloads beyond "hero runs" 4. Establish bounded expectations for users, procurers, and administrators 5. Understand and be able to reproduce performance on storage systems Using the IO500 Reproducibility guidelines, each submission is labeled according to the breadth of details provided and the access to the deployed storage software that enables the community to reproduce the results and study system design changes over time. The IO500 follows a two-staged approach. First, there will be a two-week stabilization period during which we encourage the community to verify that the benchmark runs properly on a variety of storage systems. During this period the benchmark may be updated based upon feedback from the community. The final benchmark will then be released. We expect that runs compliant with the rules made during the stabilization period will be valid as a final submission unless a significant defect is found. 10 Client Node I/O Challenge The 10 Client Node Challenge is conducted using the regular IO500 benchmark, however, with the rule that exactly 10 client nodes must be used to run the benchmark. You may use any shared storage with any number of servers. We will announce the results in the Production and Research lists as well as in separate derived lists. Birds-of-a-Feather We encourage you to submit [2] to join our community, and to attend the ISC’24 BoF [1] on Tuesday, May 12, 2024 at 10:05am - 11:05am CEST, where we will announce the new IO500 Production and Research lists and their 10 Client Node counterparts. Be Part of the Community Submissions of all sizes are welcome; the webpage has customizable sorting, so it is possible to submit on a small system and still get a very good per-client score, for example. We will also highlight new and interesting results with invited talk(s) at the BoF. [1] https://io500.org/pages/bof-isc24 [2] https://io500.org/submission [3] https://io500.org/rules-submission

1 0