- ceph-users - lists.ceph.io

Cephfs mirroring --
by Devender Singh 20 Sep '24

20 Sep '24

Hello all I tried configuring cephfs mirroring between cluster as per doc https://docs.ceph.com/en/reef/dev/cephfs-mirroring/ using bootstrap. But My replication is not working. Using two mirror daemons on each cluster. [Enabled module/mirroring/snap mirroring/aded directory/allow_new_snaps] ceph fs snapshot mirror peer_bootstrap create backup_fs client.mirror site-remote # on destination cluster ceph fs snapshot mirror peer_bootstrap import sourcefs <token> # token from above. I see the details on source cluster using “ceph fs snapshot mirror daemon status | jq” but it’s not copying nay data. I am using /repo directory under source filesystem with snapshot and schedule. “ceph fs snapshot mirror add $SOURCEFS /repo" When snap schedule starts, it showing one or two dir on destination but doesn’t grow more than few kb or mb. would be agree help if someone has done that. Regards Dev

1 0

scrubing
by Albert Shih 20 Sep '24

20 Sep '24

Hi everyone. Few time ago I add a new node to my cluster with some HDD. Currently the cluster does the remapping and backfill. I now got a warning about HEALTH_WARN 1 pgs not deep-scrubbed in time So I check and find something a litle weird. root@cthulhu1:~# ceph config get osd osd_deep_scrub_interval 604800.000000 so that's one week. If I check the LAST DEEP SCRUB TIMESTAMP I got root@cthulhu1:~# ceph pg dump pgs | awk '{print $1" "$24}' | grep -v 2024-09-[1-2][0-9] dumped pgs PG_STAT DEEP_SCRUB_STAMP 4.63 2024-09-09T19:00:57.739975+0000 4.5a 2024-09-09T08:17:15.124704+0000 4.56 2024-09-09T21:51:07.478651+0000 4.51 2024-09-08T00:10:30.552347+0000 4.4c 2024-09-09T10:35:02.048445+0000 4.4b 2024-09-09T19:53:19.839341+0000 4.14 2024-09-08T18:36:12.025455+0000 4.c 2024-09-09T16:00:59.047968+0000 4.4 2024-09-09T00:19:07.554153+0000 4.8 2024-09-09T22:19:15.280310+0000 4.25 2024-09-09T06:45:37.258306+0000 4.30 2024-09-09T16:56:21.472410+0000 4.82 2024-09-09T21:14:09.802303+0000 4.c9 2024-09-08T17:10:56.133363+0000 4.f7 2024-09-09T08:25:40.011924+0000 If I check the status of those PG it's or active+clean+scrubbing+deep and deep scrubbing for Xs or queued for deep scrub So my questions are : Why ceph tell me «1» pg not been scrub when I see 15 ? Is they are any way to find which pg ceph status are talking about. Is they are any way to see the progress or scrubbing/remapping/backfill ? Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: ven. 20 sept. 2024 09:35:43 CEST

2 4

CPU requirements
by Laszlo Budai 20 Sep '24

20 Sep '24

Hello everyone, I'm trying to understand the CPU requirements for recent versions of CEPH. Reading the documentation (https://docs.ceph.com/en/latest/start/hardware-recommendations/) I cannot get any conclusion about how to plan CPUs for ceph. There is the following statement: "With earlier releases of Ceph, we would make hardware recommendations based on the number of cores per OSD, but this cores-per-osd metric is no longer as useful a metric as the number of cycles per IOP and the number of IOPS per OSD. For example, with NVMe OSD drives, Ceph can easily utilize five or six cores on real clusters and up to about fourteen cores on single OSDs in isolation. So cores per OSD are no longer as pressing a concern as they were. When selecting hardware, select for IOPS per core." How should I understand this? On one side it's saying "with NVMe OSD drives, Ceph can easily utilize five or six cores on real clusters" and then continues: "and up to about fourteen cores on single OSDs in isolation." so should I cont 5/6 cores per NVMe OSD or 14cores? And after all that there's the next sentence that says: " So cores per OSD are no longer as pressing a concern as they were.", But if a single OSD can consume 5/6 cpu cures (that could go up to 14) then I would assume that cores per OSD is still an important concern. Can anyone explain these CPU requirements? Or point me to some other documents that describes in more details the resources required for Ceph? Thank you, Laszlo

4 5

mclock scheduler kills clients IOs
by Denis Polom 20 Sep '24

20 Sep '24

Hi guys, we have Ceph cluster with Quincy 17.2.7 where we are draining 3 hosts (each one is from one failure domain). Each host has 13 HDDs and there are another 38 hosts with same size in each failure domain. There is a lot of free space. I've set up primary-affinity on drained OSDs to 0 and set OSDs out mclock profile is: osd_mclock_profile = high_client_ops but soon after that I got slow ops on some of the OSDs and clients getting high iowaits until I stop recovering. It's strange to me that client is still trying to use OSDs where primary-affinity is set to 0 Here is the case of osd.15 with primary-affinity 0: REQUESTS 4 homeless 0 8337634 osd15 16.35734e3c 16.e3cs0 [1100,1444,1509,429,540,1106]/1100 [15,1444,403,429,540,1106]/15 e1562924 100087a5278.00000004 0x400024 1 write 8358040 osd15 16.ace50e3c 16.e3cs0 [1100,1444,1509,429,540,1106]/1100 [15,1444,403,429,540,1106]/15 e1562924 100087a5a4e.00000007 0x400024 1 write 8438861 osd15 16.a7b80e3c 16.e3cs0 [1100,1444,1509,429,540,1106]/1100 [15,1444,403,429,540,1106]/15 e1562924 100087a7c6a.0000001f 0x400024 1 write 8443242 osd15 16.de504e3c 16.e3cs0 [1100,1444,1509,429,540,1106]/1100 [15,1444,403,429,540,1106]/15 e1562924 100087a7d76.00000014 0x400024 1 write LINGER REQUESTS BACKOFFS Recovery speed is also not high (about 4GiB) and network is not saturated. I've tried also following custom profile with other custom settings: osd advanced osd_mclock_override_recovery_settings true osd advanced osd_async_recovery_min_cost 100 osd advanced osd_max_backfills 1 osd advanced osd_mclock_profile custom osd advanced osd_mclock_scheduler_background_recovery_res 0.100000 osd advanced osd_mclock_scheduler_client_res 0.900000 osd advanced osd_mclock_scheduler_client_wgt 6 osd advanced osd_recovery_max_active 1 osd advanced osd_recovery_max_chunk 8388608 osd advanced osd_recovery_op_priority 1 but result is same. Immediately clients get high iowaits and can't do any IOs. This didn't happen on this cluster before when we were using wpq scheduler. My question is - are there some other options I need to set? With wpq I was able to set osd_recovery_sleep but this option looks like not possble to change with mclock. Any ideas, please?

5 6

VFS: Busy inodes after unmount of ceph lead to kernel panic (maybe?)
by Christian Kugler 19 Sep '24

19 Sep '24

Hi, we had a problem with a Ceph OSD stuck in snaptrim. We set the Ceph OSD down temporarily to potentially solve the issue. This ended up kernel panicking two of our Kubernetes workers that use this Ceph cluster for RBD and CephFS directly and via CSI driver. The more notable lines were these after "osd25 down" and "osd25 up". Especially VFS busy inodes and the mention of generic_shutdown_super which seems to be dealing with inodes seem suspicious to be to be the culprit. 2024-09-16T16:18:37.687175+02:00 HOSTNAME kernel: [6913636.547576] VFS: Busy inodes after unmount of ceph (ceph) 2024-09-16T16:18:37.687191+02:00 HOSTNAME kernel: [6913636.547588] ------------[ cut here ]------------ 2024-09-16T16:18:37.687196+02:00 HOSTNAME kernel: [6913636.548482] kernel BUG at fs/super.c:503! […snip…] 2024-09-16T16:18:37.731920+02:00 HOSTNAME kernel: [6913636.562214] ? generic_shutdown_super.cold+0x1a/0x1c Log from "libceph (…): osd25 down" to reboot as attachment including some null bytes that snuck in somehow. Has anybody run into this before or has an idea how to prevent this in the future? We are currently running Reef (18.2.2, updating soon). Best, Christian

1 0

Deploy rgw different version using cephadm
by Mahdi Noorbala 19 Sep '24

19 Sep '24

Hello Recently I deployed a ceph cluster (version: reef) in my lab and after that, I deployed RGW using this manifest: service_type: rgw service_id: lab-object-storage placement: label: rgw count_per_host: 1 spec: rgw_frontend_port: 8080 Now I have a rgw container. the docker image is: quay.io/ceph/ceph (latest version) My question is: Is there any way to change rgw image version? How can I downgrade rgw version using cephadm? Thanks.

2 1

Ceph Dashboard TLS
by matthew＠peregrineit.net 19 Sep '24

19 Sep '24

Hi All, I'm running an (experimental) 3-Node Ceph Reef (v18.2.4) Cluster. Each of the 3 nodes runs (amongst other services) the Ceph Dashboard - for fail-over purposes. I can connect to the Ceph Dashboard when not using TLS (ie ceph config set mgr mgr/dashboard/ssl false). I've got a private PKI (Step-CA) and I'd like to use its TLS Certificates for the Ceph Dashboard. The private PKI's CA Cert has been added to my browser, and internal websites, etc, using the internal PKI's Certificates work as expected. I'd like each of the three Ceph Nodes to use their own TLS Certificate. The 3 node's hostnames are ceph01.example.com, ceph02.example.com, and ceph03.example.com. The 3 node's mgr instances are (I think; & this could be where I'm going wrong) are mgr.ceph01, mgr.ceph02, and mgr.ceph03. I've created 3 TLS Certificates with names of cephdash-ceph01.example.com.crt, etc, CNs of cephdash-ceph01.example.com, etc, and SANs of ceph01.example.com, etc I've followed the documentation here: https://docs.ceph.com/en/reef/mgr/dashboard/#ssl-tls-support I have stopped and restarted the Ceph Dashboard service. Commands used (on ceph01.example.com) ceph dashboard set-ssl-certificate ceph01 -i cephdash-ceph01.example.com.crt ceph dashboard set-ssl-certificate-key ceph01 -i cephdash-ceph01.example.com.key ceph mgr module disable dashboard ceph mgr module enable dashboard It's not working (I get an Unable to connect page), and I can't work out why - but I suspect it'll be something bleedingly obvious. :-) My Qs: Can I use EC Certs or must I use RSA Certs? I've tryied both and neither work. Can anyone spot what I'm doing wrong? Thanks in advance Cheers Dulux-Oz

1 0

Re: [EXT] mclock scheduler kills clients IOs
by Kai Stian Olstad 18 Sep '24

18 Sep '24

On Tue, Sep 17, 2024 at 08:48:11PM -0400, Anthony D'Atri wrote: >Were all three in the same failure domain? No they were all in different failure domain. -- Kai Stian Olstad

1 0

Re: High usage (DATA column) on dedicated for OMAP only OSDs
by Александр Руденко 18 Sep '24

18 Sep '24

Hi, Anthony! Thank you, but I checked it twice and carefully. No one other PGs on these OSDs. I did it using ceph pg ls-by-osd XX ср, 18 сент. 2024 г. в 14:47, Anthony D'Atri <anthony.datri(a)gmail.com>: > Dump your CRUSH rules and compare with `ceph osd dump | grep pool` > > I suspect you have a data pool using those SSDs that you don’t realize. > > > On Sep 18, 2024, at 7:32 AM, Александр Руденко <a.rudikk(a)gmail.com> > wrote: > > > > Hi! > > > > We have S3/CephFS cluster with dedicated SSDs for bucket indexes, CephFS > > metadata and for a few small rgw metadata pools. > > We have special crush rules for these pools. > > All of our pools (pool naming is old because it's very old cluster) that > > are placed on SSDs: > > POOL ID PGS STORED OBJECTS USED %USED MAX > > AVAIL > > .rgw.root 1 32 1.3 KiB 5 108 KiB 0 > 690 > > GiB > > .rgw.control 2 32 0 B 8 0 B 0 > 690 > > GiB > > .rgw 3 32 2.0 MiB 8.02k 328 MiB 0.02 > 690 > > GiB > > .rgw.gc 4 64 1.6 GiB 64 4.9 GiB 0.24 > 690 > > GiB > > .users.uid 5 32 1.8 MiB 2.63k 67 MiB 0 > 690 > > GiB > > .users 6 32 275 KiB 7.02k 379 MiB 0.02 > 690 > > GiB > > .usage 7 32 30 MiB 64 30 MiB 0 > 690 > > GiB > > .intent-log 8 64 0 B 0 0 B 0 > 690 > > GiB > > .log 9 64 6.6 GiB 34.29k 21 GiB 1.01 > 690 > > GiB > > .rgw.buckets.index 12 4096 1.9 TiB 248.96k 1.9 TiB 48.02 > 690 > > GiB > > .users.email 13 32 64 KiB 1.61k 72 MiB 0 > 690 > > GiB > > fs1_meta 14 64 467 MiB 35.39k 829 MiB 0.04 > 690 > > GiB > > > > But on all our SSDs we can see high DATA usage, for example: > > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP > META > > AVAIL %USE VAR PGS STATUS TYPE NAME > > 57 ssd 0.00005 1.00000 447 GiB 406 GiB 309 GiB 96 GiB > 1.4 > > GiB 40 GiB 90.96 1.37 138 up osd.57 > > 19 ssd 0.00005 1.00000 447 GiB 402 GiB 309 GiB 92 GiB > 1.4 > > GiB 44 GiB 90.09 1.35 145 up osd.19 > > 10 ssd 0.00005 1.00000 447 GiB 406 GiB 309 GiB 97 GiB > 1.1 > > GiB 40 GiB 91.01 1.37 137 up osd.10 > > 12 ssd 0.00005 1.00000 447 GiB 401 GiB 309 GiB 91 GiB > 1.3 > > GiB 46 GiB 89.77 1.35 134 up osd.12 > > > > All SSDs are bluestore. > > Ceph 16.3.11 and many SSDs were deployed on 16.x. > > As I understand, the majority of these pools contain only "OMAP" which is > > stored in RocksDB and tracked in stats as *OMAP*. > > > > And I don't understand why we have such high *DATA* and we can see that > > DATA usage grows by 1-2 GB/day for the last 90 days (we don't have more > > monitoring data)! > > > > I have checked crush rules and all SSDs for PGs from other pools and > can't > > see any other PGs from "pure" data pools. Rules are correct. > > > > How can I see what is stored in bluestore DATA on some OSD? I can export > > full RocksDB, but not DATA stored in bluestore.. > > What kind of DATA can be stored in these OSDs with these pools? > > We did a lot of offline compactions, 2-5 times for some SSDs in the last > 90 > > days. But i'm not sure if it's related > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

For 750 osd 5 monitor in redhat doc
by Szabo, Istvan (Agoda) 18 Sep '24

18 Sep '24

Hi, I went through some Redhat doc and saw this sentence which I didn't see in Ceph docs: "The storage cluster can run with only one Ceph Monitor; however, to ensure high availability in a production storage cluster, Red Hat will only support deployments with at least three Ceph Monitor nodes. Red Hat recommends deploying a total of 5 Ceph Monitors for storage clusters exceeding 750 Ceph OSDs." here: https://docs.redhat.com/en/documentation/red_hat_ceph_storage/4/html/instal… We will reach this number so wonder do we really need 5 mon? Currently we have 3 collocated mgr/mon on baremetal server with some multiste gw traffic, if needed I'd plan to simply add some mon on vm only. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 2