- ceph-users - lists.ceph.io

How to increment osd_deep_scrub_interval
by Jorge JP 05 Jan '24

05 Jan '24

Hello! I want to increment the interval between deep scrub in all osd. I tryed this but not configured: ceph config set osd.* osd_deep_scrub_interval 1209600 I have 50 osd.. should config every osd? thanks for the support!

2 1

MacOS support for CephFS client
by Fabien Sirjean 04 Jan '24

04 Jan '24

Dear all, Could you please share recent experiences with support for cephfs access from MacOS clients ? I couldn't find any threads on this matter on the list since 2021 (Daniel Persson), where he stated that : > Mac Mini Intel Catalina - Connected and working fine. > Mac Mini M1 BigSur - Can't compile brew cask, no popups for extra > permissions in the GUI. Thanks a lot! Cheers, Fabien

1 0

Unable to find Refresh Interval Option in Ceph Dashboard (Ceph v18.2.1 "reef")- Seeking Assistance
by Alam Mohammad 04 Jan '24

04 Jan '24

Hello, We've been using Ceph for managing our storage infrastructure, and we recently upgraded to the latest version (Ceph v18.2.1 "reef"). However, We've noticed that the "refresh interval" option seems to be missing in the dashboard, and we are facing challenges with monitoring our cluster in real-time. In the earlier version of the Ceph dashboard, there was a useful "refresh interval" option that allowed us to customize the update frequency of the dashboard. This was particularly handy for monitoring changes and responding promptly. However, after the upgrade to Ceph v18.2.1 "reef", We can't seem to find this option anywhere in the dashboard. Additionally, we observed an automatic refresh occurring at every 25 seconds. Seeking guidance locating and tuning the refresh interval settings in the latest version of Ceph to potentially reduce this interval. We've explored the dashboard settings thoroughly and reviewed the release notes for Ceph v18.2.1 "reef", but we couldn't find any mention of the removal of the "refresh interval" option. Any guidance or insights would be greatly appreciated! Thanks, Mohammad Saif Ceph Enthusiast

3 2

Re: Ceph newbee questions
by Anthony D'Atri 04 Jan '24

04 Jan '24

>> >> You can do that for a PoC, but that's a bad idea for any production workload. You'd want at least three nodes with OSDs to use the default RF=3 replication. You can do RF=2, but at the peril of your mortal data. > > I'm not sure I agree - I think size=2, min_size=2 is no worse than > RAID1 for data security. size=2, min_size=2 *is* RAID1. Except that you become unavailable if a single drive is unavailable. >> That isn't even the main risk as I understand it. Of course a double > failure is going to be a problem with size=2, or traditional RAID1, > and I think anybody choosing this configuration accepts this risk. We see people often enough who don’t know that. I’ve seen double failures. ymmv. > As I understand it, the reason min_size=1 is a trap has nothing to do > with double failures per se. It’s one of the concerns. > > The issue is that Ceph OSDs are somewhat prone to flapping during > recovery (OOM, etc). So even if the disk is fine, an OSD can go down > for a short time. If you have size=2, min=1 configured, then when > this happens the PG will become degraded and will continue operating > on the other OSD, and the flapping OSD becomes stale. Then when it > comes back up it recovers. The problem is that if the other OSD has a > permanent failure (disk crash/etc) while the first OSD is flapping, > now you have no good OSDs, because when the flapping OSD comes back up > it is stale, and its PGs have no peer. Indeed, arguably that’s an overlapping failure. I’ve seen this too, and have a pg query to demonstrate it. > I suspect there are ways to re-activate it, though this will result in potential data > inconsistency since writes were allowed to the cluster and will then > get rolled back. Yep. > With only two OSDs I'm guessing that would be the > main impact (well, depending on journaling behavior/etc), but if you > have more OSDs than that then you could have situations where one file > is getting rolled back, and some other file isn't, and so on. But you’d have a voting majority. > > With min_size=2 you're fairly safe from flapping because there will > always be two replicas that have the most recent version of every PG, > and so you can still tolerate a permanent failure of one of them. Exactly. > > size=2, min=2 doesn't suffer this failure mode, because anytime there > is flapping the PG goes inactive and no writes can be made, so when > the other OSD comes back up there is nothing to recover. Of course > this results in IO blocks and downtime, which is obviously > undesirable, but it is likely a more recoverable state than > inconsistent writes. Agreed, the difference between availability and durability. Depends what’s important to you. > > Apologies if I've gotten any of that wrong, but my understanding is > that it is these sorts of failure modes that cause min_size=1 to be a > trap. This isn't the sort of thing that typically happens in a RAID1 > config, or at least that admins don't think about. It’s both.

2 4

CLT Meeting Minutes 2024-01-03
by David Orman 03 Jan '24

03 Jan '24

Happy 2024! Today's CLT meeting covered the following: 1. 2024 brings a focus on performance of Crimson (some information here: https://docs.ceph.com/en/reef/dev/crimson/crimson/ ) 1. Status is available here: https://github.com/ceph/ceph.io/pull/635 2. There will be a new Crimson performance weekly meeting that will be lead by Matan Breizman 1. This does not replace the existing performance weekly, and is focused on Crimson 2. An email will follow with more details about this meeting 2. Ceph Quarterly will be published on/around the 14th of January, 2024. 1. See https://ceph.io/en/community/cq/ for previous issues of CQ 3. A development freeze on Squid is tentatively scheduled for January 31, 2024 4. Upcoming releases 1. 16.2.15 is next (the last Pacific release) 1. Anticipated by the end of January 2. 17.2.8 will follow (Quincy) 3. 18.2.2 will follow this (Reef)

1 0

Ceph Docs: active releases outdated
by Eugen Block 03 Jan '24

03 Jan '24

Hello and a happy new year! I'm wondering if there are some structural changes or something regarding the release page [1]. It still doesn't contain version 18.2.1 (Reef) and the latest two Quincy releases (17.2.6, 17.2.7) are missing as well. And for Pacific it's even worse, the latest entry is for 16.2.11 although we're close to 16.2.5 being released. I wanted to check the changelog for 18.2.1 which can be found in the news blog [2]. I'd understand if the changelogs would be published only in one place (the blog?) but then it should be at least referenced in the docs. Thanks, Eugen [1] https://docs.ceph.com/en/reef/releases/ [2] https://ceph.io/en/news/blog/2023/v18-2-1-reef-released/

3 3

Radosgw replicated -> EC pool
by Jan Kasprzak 02 Jan '24

02 Jan '24

Hello, Ceph users, what is the best way how to change the storage layout of all buckets in radosgw? I have default.rgw.buckets.data pool as replicated, and I want to use an erasure-coded layout instead. One way is to use cache tiering as described here: https://cephnotes.ksperis.com/blog/2015/04/15/ceph-pool-migration/ Could this be done under the running radosgw? If I read this correctly, it should be done, because radosgw is just another RADOS client. Another possible approach would be to create a new erasure-coded pool, a new zone placement, and set it as default. But how can I migrate the existing data? If I understand it correctly, the default placement applies only to new buckets. Something like this: ceph osd erasure-code-profile set k5m2 k=5 m=2 ceph osd pool create default.rgw.buckets.ecdata erasure k5m2 ceph osd pool application enable default.rgw.buckets.ecdata radosgw radosgw-admin zonegroup placement add --rgw-zonegroup default --placement-id ecdata-placement radosgw-admin zone placement add --rgw-zone default --placement-id ecdata-placement --data-pool default.rgw.buckets.ecdata --index-pool default.rgw.buckets.index --data-extra-pool default.rgw.buckets.non-ec radosgw-admin zonegroup placement default --rgw-zonegroup default --placement-id ecdata-placement How to continue from this point? And a secondary question: what purpose does the data-extra-pool serve? Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

2 1

ValueError: invalid literal for int() with base 10: '443 ssl_certificate=c
by Владимир Клеусов 02 Jan '24

02 Jan '24

Hi, When trying to log in to rgb via the dashboard, an error appears in the logs ValueError: invalid literal for int() with base 10: '443 ssl_certificate=config://rgw/cert/rgw.test' RGW with SSL If rgw is without ssl, everything works fine ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) Please tell me how to solve this?

2 2

ceph-iscsi on RL9
by duluxoz 02 Jan '24

02 Jan '24

Hi All, Just successfully(?) completed a "live" update of the first node of a Ceph Quincy cluster from RL8 to RL9. Everything "seems" to be working - EXCEPT the iSCSI Gateway on that box. During the update the ceph-iscsi package was removed (ie `ceph-iscsi-3.6-2.g97f5b02.el8.noarch.rpm` - this is the latest package available from the Ceph Repos). So, obviously, I reinstalled the package. However, `dnf` is throwing errors (unsurprisingly, as that package is an el8 package and this box is now running el9): that package requires python 3.6 and el9 runs with python 3.8 (I believe). So my question(s) is: Can I simply "downgrade" python to 3.6, or is there an el9-compatible version of `ceph-iscsi` somewhere, and/or is there some process I need to follow to get the iSCSI Gateway back up and running? Some further info: The next step in my "happy-happy-fun-time-holiday-ICT-maintenance" was to upgrade the current Ceph Cluster to use `cephadm` and to go from Ceph-Quincy to Ceph-Reef - is this my ultimate upgrade path to get the iSCSI G/W back? BTW the Ceph Cluster is used *only* to provide iSCSI LUNS to an oVirt (KVM) Cluster front-end. Because it is the holidays I can take the entire network down (ie shutdown all the VMs) to facilitate this update process, which also means that I can use some other way (ie a non-iSCSI way - I think) to connect the Ceph SAN Cluster to the oVirt VM-Hosting Cluster - if *this* is the solution (ie no iSCSI) does someone have any experience in running oVirt off of Ceph in a non-iSCSI way - and could you be so kind as to provide some pointers/documentation/help? And before anyone says it, let me: "I broke, now I own it" :-) Thanks in advance, and everyone have a Merry Christmas, Heavenly Hanukkah, Quality Kwanzaa, Really-good (upcoming) Ramadan, and/or a Happy Holidays. Cheers Dulux-Oz

2 2

What is the maximum number of Rados gateway objects in one cluster using the bucket index and in one bucket?
by steve jung 02 Jan '24

02 Jan '24

Hello. We are using Ceph storage to test whether we can run the service by uploading and saving more than 40 billion files. So I'd like to check the contents below. 1) Maximum number of Rados gateway objects that can be stored in one cluster using the bucket index 2) Maximum number of Rados gateway objects that can be stored in one bucket Although we have referred to the limitations on the number of Rados gateway objects mentioned in existing documents, it seems theoretically unlimited If you have operated the number of files at the level we think in actual services or products, we would appreciate it if you could share them. Below are related documents and related settings values. > Related documents - https://documentation.suse.com/ses/5.5/html/ses-all/cha-ceph-gw.html - https://www.ibm.com/docs/en/storage-ceph/6?topic=resharding-limitations-buc… - https://docs.ceph.com/en/latest/dev/radosgw/bucket_index/ > Related config - rgw_dynamic_resharding: true - rgw_max_objs_per_shard: 100000 - rgw_max_dynamic_shards : 65521

2 1