- ceph-users - lists.ceph.io

Cephfs snapshot
by s.dhivagar.cse＠gmail.com 28 Oct '24

28 Oct '24

Hi.. If we enable cephfs snapshot will we face performance issue? And does snapshots take up any specific storage?

1 0

RGW Graphs in cephadm setup
by bkennedy＠cfl.rr.com 26 Oct '24

26 Oct '24

We recently upgraded all our clusters to rocky 9.4 and reef 18.2.4. Two of the clusters show the rgw metrics in the ceph dashboard and the other two don't. I made sure the firewalls were open for ceph-exporter and that Prometheus was gathering the stats on all 4 clusters. For the clusters that are not working, when I do a search on the Prometheus web console at port 9095 on the performance monitoring node using "{instance="rgw1"}, I noticed that the instance_id shows a long number. I did the same search on the clusters that show rgw stats in grafana and they have an instance_id that looks like the drop down in the Grafana dashboard. E.g. "rgw.rgw.rgw1.biewae". So it seems that for some reason, on the clusters that are not working, the ceph-exporter running on the rgw nodes is not pulling the proper instance_id rgw instance. Is there a way to fix the instance_id that ceph-exporter is pulling? I can see this instance id when browsing to the servers http://rgw1:9926/metrics portal. If I run a podman ps, I can see the container running and it has the unique id .biewae showing. I used cephadm to enter the ceph-exporter container on the rgw to see what the logs were saying and I see the same thing repeated: 1 pid path is empty; process metrics won't be fetched for: ceph-client.rgw.rgw.rgw1.biewae.2.94759457265864 Its only the rgw's that show this error in the ceph-exporter log file, the mons and osd nodes show no errors in the log and show up in grafana properly with data. I redeployed ceph-exporter and rgw to the same node in that order, but no change. I did set the default image container to 18.2.4 prior to redeploy cause I ran into that issue as well. Regards, -Brent Existing Clusters: Test: Reef 18.2.0 ( all virtual on nvme ) US Production(HDD): Reef 18.2.4 Cephadm with 11 osd servers, 5 mons, 4 rgw, 2 iscsigw UK Production(HDD): Reef 18.2.4 Cephadm with 20 osd servers, 5 mons, 4 rgw, 2 iscsigw US Production(SSD): Reef 18.2.4 Cephadm with 6 osd servers, 5 mons, 4 rgw UK Production(SSD): Reef 18.2.4 cephadm with 6 osd servers, 5 mons, 4 rgw

3 3

Availability of Ceph Reef in Debian 12(bookworm) for arm64
by Karun Josy 26 Oct '24

26 Oct '24

Hello, By default, Debian 12.7 bookwork has only the Pacific (16.x) package for cephadm installation. Is there any plan to have reef or squid for arm64 in Debian 12.x? As a workaround to install ceph, I used curl method to download the cephadm binary from "rpm-reef" $ curl --silent --remote-name --location https://download.ceph.com/rpm-reef/el9/noarch/cephadm $ chmod +x cephadm $ sudo ./cephadm add-repo --version 18.2.4 This helps to create a ceph cluster in Reef version.

2 1

centos9 or el9/rocky9
by Marc 25 Oct '24

25 Oct '24

Sorry for posting off topic, a bit to lazy to create yet another account somewhere. I still need to make this upgrade to different os. I have now some vms on centos9 stream. What annoys me a lot is that tcp wrapper support is not default added to ssh. (I am using auto fed dns blacklists to refuse access) Can anyone tell me if this is the same in el9/rocky9?

5 5

pgs not deep-scrubbed in time and pgs not scrubbed in time
by Götz Reinicke 25 Oct '24

25 Oct '24

Hello Ceph Community, My cluster was hit by a power outage some month ago. Luckily no data was destroyed and powering up the nodes and services went well. But till than some pgs are still shown as not scrubbed in time. Googling and searching the list showed some debugging hints like „ceph pg deep-scrub“ the pgs or restarting osd deamons. Nothing „solved“ that issue here. I’m on ceph version 18.2.4 now. Is there anything special what I can do to have thous pgs scrubbed? I like having the cluster health state ok not warning :) Or will time solve the problem when the pgs are in there regular cycle for being scrubbed again? Thanks for hints and suggestion . Best regards Götz

6 6

IO500 SC24 List Call for Submissions
by IO500 Committee 24 Oct '24

24 Oct '24

Call for Submission Submission Deadline: Nov 10th, 2024 AoE The IO500 is now accepting and encouraging submissions for the upcoming 15th semi-annual IO500 Production and Research lists, in conjunction with SC24. We are also accepting submissions to both the Production and Research 10 Client Node Challenges to encourage the submission of small scale results. View the requirements for submitting to each list on the IO500 Webpage. The new ranked lists will be announced at the BoF [1]. We look forward to seeing many new results! Background Following the success of the Top500 in collecting and analyzing historical trends in supercomputer technology and evolution, the IO500 was created in 2017, published its first list at SC17, and has grown continually since then. The benchmarks represent community accepted standards, including being used in Request for Proposals for new HPC platforms. The benchmarks showcase the IO access pattern extremes giving a full picture of storage system potential performance. The list is about much more than just the raw rank; all submissions help the community by collecting and publishing a wider corpus of data. The multi-fold goals of the benchmark suite are as follows: Represent naive and optimized access patterns for the execution of a rich variety of HPC applications, their achievable performance and the documentation of how the numbers are achieved. Support small to extreme-scale Research and Production HPC systems using flexible storage APIs Maximizing simplicity in running the benchmark suite while offering tunable parameters Specifically, the benchmark suite includes a hero-run of both IOR and mdtest configured however possible to maximize performance and establish an upper-bound for performance. It also includes runs with highly prescribed parameters in an attempt to determine a lower performance bound. Finally, it includes a namespace search as this has been determined to be a highly sought-after feature in HPC storage systems that has historically not been well-measured. Supported Storage APIs are those that are part of IOR and mdtest. Extending these tools with a public pull request can be done to enable new storage APIs. The goals of the community are also multi-fold: Gather historical data for the sake of analysis and to aid predictions of storage futures Collect tuning information to share valuable performance optimizations across the community Encourage vendors and designers to optimize for workloads beyond "hero runs" Establish bounded expectations for users, procurers, and administrators Understand and be able to reproduce performance on storage systems Using the IO500 Reproducibility guidelines, each submission is labeled according to the breadth of details provided and the access to the deployed storage software that enables the community to reproduce the results and study system design changes over time. This year, since there are no changes to the benchmark, we are skipping the stabilization period and just releasing the final benchmark. 10 Client Node I/O Challenge The 10 Client Node Challenge is conducted using the regular IO500 benchmark, however, with the rule that exactly 10 client nodes must be used to run the benchmark. You may use any shared storage with any number of servers. We will announce the results in the Production and Research lists as well as in separate derived lists. Birds-of-a-Feather We encourage you to submit [2] to join our community, and to attend the SC’24 BoF [1] on Tuesday, Nov 19, 2024 at 12:15pm - 1:15pm EST, where we will announce the new IO500 Production and Research lists and their 10 Client Node counterparts. Be Part of the Community Submissions of all sizes are welcome; the webpage has customizable sorting, so it is possible to submit on a small system and still get a very good per-client score, for example. We will also highlight new and interesting results with invited talk(s) at the BoF. [1] https://io500.org/pages/bof-sc24 [2] https://io500.org/submission -- The IO500 Committee

1 0

Issue with Filter in Lifecycle policy
by mahnoosh shahidi 24 Oct '24

24 Oct '24

Hi everyone, I'm trying to set a lifecycle policy to change the storage class of objects based on their size. Here's the rule I'm using: > { > "Rules": [ > { > "ID": "migrating storage class", > "Filter": > { > "ObjectSizeGreaterThan": 10000000 > }, > "Status": "Enabled", > "Transitions": [ > { > "Days": 0, > "StorageClass": "REPLICATED" > } > ] > } > ] > } The issue is that the rule seems to be applying to all objects, regardless of their size. Is the ObjectSizeGreaterThan filter supported in Ceph, or am I missing something in the configuration? Thanks for your help! Best regards, Mahnoosh

2 1

PG autoscaler taking too long
by AJ_ sunny 24 Oct '24

24 Oct '24

Hello team, I have one small size ceph cluster in production having 53 SSD of 7TB each with 6 node Version:- octopus In last two days we just cleared out ~100 tb of data out of 370tb total size So there is bunch of pg with active+clean+snaptrim & snaptrim_wait state comes into the action after that one of the pool placement group number decrease by autoscaler from 2048 to 512 And autoscaler showing remaining time of 4y which is too long can you guys help me out how to fix this issue root@nvme1:~# ceph -s cluster: id: c30f5720-ca5c-11ec-b19c-f9781f61e8ec health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 34 pgs not deep-scrubbed in time services: mon: 3 daemons, quorum nvme3,nvme2,nvme1 (age 14M) mgr: nvme2.ttfore(active, since 13M), standbys: nvme3.zhnsuf osd: 53 osds: 53 up (since 2h), 53 in (since 11d); 61 remapped pgs flags noscrub,nodeep-scrub task status: data: pools: 7 pools, 2784 pgs objects: 26.37M objects, 69 TiB usage: 177 TiB used, 194 TiB / 370 TiB avail pgs: 1011732/79097316 objects misplaced (1.279%) 2723 active+clean 58 active+remapped+backfill_wait 3 active+remapped+backfilling io: client: 78 KiB/s rd, 3.8 MiB/s wr, 238 op/s rd, 314 op/s wr recovery: 17 MiB/s, 6 objects/s progress: PG autoscaler decreasing pool 8 PGs from 2048 to 512 (1h) [............................] (remaining: 4y) root@nvme1:~# Thanks Arihant Jain

2 2

High number of Cephfs Subvolumes compared to Cephfs persistent volumes in K8S environnement
by Edouard FAZENDA 23 Oct '24

23 Oct '24

Dear Ceph Community, Maybe you can help me with this I have inconsistencies with the number of subvolumes in Cephfs in my Ceph Cluster and the number of Persistent Volumes of StorageClass Cephfs in my K8S cluster I have K8S 1.25 version on my kubernetes cluster and use the Ceph CSI CephFs 3.8.1, my ceph cluster is 16.2.15 pacific release. I have 35 Persistent Volumes of type CephFS $ kubectl get pv -o custom-columns=name:.metadata.name,subvolume:.spec.csi.volumeAttributes.subv olumeName,storageclass:.spec.storageClassName | grep cephfs pvc-0062e845-4a87-4a32-8f19-cbfff3b2789d csi-vol-919a2302-f964-11ed-b5e0-b615b8ae8847 csi-cephfs-sc pvc-0298fc13-bc5f-489a-b88e-64d7cdc61f1e csi-vol-7ce2f152-d96c-4f54-b335-9ede672ce320 csi-cephfs-sc pvc-09991b1e-b9f6-4a7c-b710-25131433c6e9 csi-vol-ef2e7be0-8044-11ec-b79c-fa4d817fb9f0 csi-cephfs-sc pvc-19b8f683-b3e8-4476-abc3-9d7d7e4d8941 csi-vol-5a9ce4b8-1edb-11ed-8f9a-e6ec00c2f5e5 csi-cephfs-sc pvc-2fc90e06-22b4-4e80-b33a-c4c5dd6a0baa csi-vol-760f34d1-4e26-11ed-af16-d67e0f01e63a csi-cephfs-sc pvc-316aa495-3d52-45a5-aac3-4f64bf598aa3 csi-vol-9d135b18-8b43-402b-a606-a2c426527b20 csi-cephfs-sc pvc-342b66f9-f6d5-413c-8634-e51a25d638a8 csi-vol-7e3aca09-8d76-11eb-95b9-ee71e85dde57 csi-cephfs-sc pvc-48c664a6-073e-4498-9370-d4a718f11729 csi-vol-72d71655-9518-11ee-a81d-b2c17fca41c9 csi-cephfs-sc pvc-5349d4ec-7e62-48b7-bd8c-8f718a13906b csi-vol-9748018f-4df9-11ed-af16-d67e0f01e63a csi-cephfs-sc pvc-5db458a4-2dfb-46d7-8b23-2bde45a9bf46 csi-vol-bd87eade-e726-4a5d-993c-56dbece42420 csi-cephfs-sc pvc-5fd1f3a5-d120-4283-8e2c-b6e1e3e4cbd1 csi-vol-e7ec3159-9d1c-11eb-82b9-aaa0cbaeebfa csi-cephfs-sc pvc-669c35f9-9927-4e44-94d2-a9ddd0eda914 csi-vol-ef9539f6-9c5a-40d8-a89b-087ad7c6e9b3 csi-cephfs-sc pvc-6909652a-2da6-4a13-a111-59a21ed685b2 csi-vol-a9a1fff7-550a-43a1-aa52-4f6710e1ca9d csi-cephfs-sc pvc-72c2f70e-2976-4188-8d1a-f2928745097b csi-vol-51045501-f17b-4f4d-af08-100ea3db3db3 csi-cephfs-sc pvc-73b1d3f3-f861-44d7-bbd0-43fc16f9e7a9 csi-vol-439769d7-9b94-4350-bfa2-fe5e9a182650 csi-cephfs-sc pvc-76b04010-1e00-4fea-82b7-855ff83ab820 csi-vol-28852a9b-79c0-4eb7-afcf-b443ff9f867d csi-cephfs-sc pvc-7e0b08ea-9d31-45a7-a098-588563c18f19 csi-vol-4a5086d9-62ad-11ee-80a7-323a25995550 csi-cephfs-sc pvc-83853b0b-ab55-46be-ae3a-78f8d431e780 csi-vol-ef2e919c-8044-11ec-b79c-fa4d817fb9f0 csi-cephfs-sc pvc-89c85899-c26a-4efd-91dd-cc891d303ad6 csi-vol-e8b89f50-fdf9-11ec-8f9a-e6ec00c2f5e5 csi-cephfs-sc pvc-8a4c12b1-b7d8-4337-b218-7de2bd6eadde csi-vol-718d31a6-9b4a-11ee-a81d-b2c17fca41c9 csi-cephfs-sc pvc-8b56ae01-1809-4552-9192-930bd1db95f8 csi-vol-a72f2d4c-1f8b-4a71-a846-2b7a6febbf82 csi-cephfs-sc pvc-97b80840-fd60-436a-8223-5cf41bfe56ef csi-vol-0289ed46-4df6-11ed-af16-d67e0f01e63a csi-cephfs-sc pvc-984eadce-c908-4e21-a558-70d12bcff3c2 csi-vol-ef2e8a6b-8044-11ec-b79c-fa4d817fb9f0 csi-cephfs-sc pvc-9de21d9d-80ae-49f8-bea0-2a265945cfa6 csi-vol-c7b4f302-4d71-4777-b6c0-82720ebe8950 csi-cephfs-sc pvc-9f5651b5-0cb2-47d2-889e-69b09d96fe36 csi-vol-9ba650df-3fa5-43f0-abea-5231643fb1bf csi-cephfs-sc pvc-a2607a68-62a9-47a0-8f15-871359460556 csi-vol-028a40c4-4df6-11ed-af16-d67e0f01e63a csi-cephfs-sc pvc-a84158f8-1d21-475a-9987-a0e8a3347c42 csi-vol-22835b9b-c31c-46c6-aec2-67b554c683d3 csi-cephfs-sc pvc-b3d81d90-eaec-4d8a-8065-790d9c0e6ea7 csi-vol-c908a216-cf78-11eb-a202-52b7fddc54da csi-cephfs-sc pvc-bcc5979b-f9cb-4c10-ab64-24424749856b csi-vol-e8dd361e-b6fb-4a19-94af-7cae07d4702c csi-cephfs-sc pvc-bd09916e-a47d-429f-9371-ac779d3959b3 csi-vol-cc9573bc-6733-11ed-af16-d67e0f01e63a csi-cephfs-sc pvc-dac9a949-cd28-4448-9c66-7bab365c3d25 csi-vol-21b0d822-6fc3-11ec-96cb-dec0a2aa44f8 csi-cephfs-sc pvc-e0783e40-f49c-4d7c-a404-4554ce35341c csi-vol-0289e975-4df6-11ed-af16-d67e0f01e63a csi-cephfs-sc pvc-f660eb28-2af4-45de-b00b-72f73e08745d csi-vol-4b6e4fd4-9e4e-4b0d-a022-305b8e1d8e84 csi-cephfs-sc pvc-fd994df6-9ca5-47d5-af64-afec037fd5d1 csi-vol-ef2e42e3-8044-11ec-b79c-fa4d817fb9f0 csi-cephfs-sc pvc-fe72f0a2-c260-4a40-acca-50f47d31c11a csi-vol-6c0efe87-d02c-11eb-a202-52b7fddc54da csi-cephfs-sc I have 985 subvolumes in Cephfs csi volume , i have no idea on why i have this difference. $ ceph fs subvolume ls cephfs csi | awk '{if ($2 != "") print $2}' | sed 's/"//g' | wc -l 1020 If i check the number of Volume Snapshots of StorageClass Cephfs via the command "kubectl get volumesnapshot -A | grep cephfs | wc -l" i have 177 If i check from the subvolumes of Cephfs that are mapped to the K8S PV i have 178 via the command "for subvolume in $(cat list.txt | xargs);do ceph fs subvolume snapshot ls cephfs $subvolume csi | awk '{if ($2 != "") print $2}' | sed 's/"//g' ;done | wc -l" I have 178 Which seems to be correct and consistent. Are my checks valid to identifies subvolumes that are not anymore used by my K8S workload ? Can i safely delete the subvolumes as they seems not associated with a persistent volume in Kubernetes ? Thanks in advance for the help Have a nice day Best Regards, Edouard Fazenda. Edouard FAZENDA Technical Support <https://www.csti.ch/> www.csti.ch

2 1

Issue with Recovery Throughput Not Visible in Ceph Dashboard After Upgrade to 19.2.0 (Squid)
by Sanjay Mohan 23 Oct '24

23 Oct '24

Dear Ceph Users, I hope this message finds you well. I recently performed an upgrade of my Ceph cluster, moving through the following versions: * 17.2.7 >> 18.2.0 >> 18.2.2 >> 18.2.4 >> 19.2.0 (Squid) After successfully upgrading to Ceph 19.2.0, I noticed an issue where the recovery throughput is no longer visible in the new Ceph dashboard. However, Old dashboard metrics and features seem to be working as expected. It is important to note that the recovery throughput was displayed properly in the previous version of the Ceph dashboard. I am using Cephadm for the installation, not Rook. Current behavior: * Recovery throughput metrics are not displayed in the new dashboard after upgrading to 19.2.0. Expected behavior: * The recovery throughput should be visible, as it was in previous versions of the Ceph dashboard. I am reaching out to inquire if there are any known issues, workarounds, or upcoming fixes for this. Your assistance in this matter would be greatly appreciated. Thank you for your time and support. I look forward to hearing from you soon. Best regards, Sanjay Mohan Software Defined Storage Engineer sanjaymohan(a)am.amrita.edu [Amrita University] Disclaimer: The information transmitted in this email, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. Any views expressed in any message are those of the individual sender and may not necessarily reflect the views of Amrita Vishwa Vidyapeetham. If you received this in error, please contact the sender and destroy any copies of this information.

5 9