- ceph-users - lists.ceph.io

Re: Insecure keys allowed (aes)
by Eugen Block 25 Aug '26

25 Aug '26

Hi, you can disable the old cipher by setting auth_allowed_ciphers to aes256k only: ceph mon set auth_allowed_ciphers aes256k See step 10 in the docs: https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/index.h… Regards, Eugen Zitat von "GLE, Vivien" <ceph-users(a)ceph.io>: > Hi, > > > I just upgrade (from tentacle 20.2.1 to 20.2.4) and rolled CephX > keys for a Cephadm cluster but there is still an error concerning > the use of insecure keys > > > #ceph config get mon mon_auth_allow_insecure_key > false > > > #ceph health detail > > HEALTH_WARN Monitors are configured to allow auth using insecure key types > [WRN] AUTH_INSECURE_KEYS_ALLOWED: Monitors are configured to allow > auth using insecure key types > insecure cipher aes allowed for auth > > > Didn't found any keys named like this in the ceph configuration but > in the ceph mon dump > > #ceph mon dump > > auth_epoch 0 > auth_service_cipher aes256k > auth_allowed_ciphers aes, aes256k <= = > auth_preferred_cipher aes256k > dumped monmap epoch 15 > > Is the only way to get rid of this error is to manipulate the mon > map ? How can I do it safely ? > > Thanks > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Insecure keys allowed (aes)
by Patrick Donnelly 25 Aug '26

25 Aug '26

Hello, On Tue, Aug 25, 2026 at 8:32 AM GLE, Vivien <ceph-users(a)ceph.io> wrote: > > Hi, > > > I just upgrade (from tentacle 20.2.1 to 20.2.4) and rolled CephX keys for a Cephadm cluster but there is still an error concerning the use of insecure keys > > > #ceph config get mon mon_auth_allow_insecure_key > false > > > #ceph health detail > > HEALTH_WARN Monitors are configured to allow auth using insecure key types > [WRN] AUTH_INSECURE_KEYS_ALLOWED: Monitors are configured to allow auth using insecure key types > insecure cipher aes allowed for auth > > > Didn't found any keys named like this in the ceph configuration but in the ceph mon dump Please read the documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks/#auth-insecu… -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 1

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Eugen Block 25 Aug '26

25 Aug '26

Yes, unmanaged OSDs are of course also rotated (excerpt from an upgrade of unmanaged OSDs): 2026-08-25T12:35:03.728+0000 7f9d451b7640 0 [cephadm INFO cephadm.upgrade] Redeploying osd.5 with new keyring The unmanaged flag is basically to prevent automatic deployment of OSDs only. Zitat von Chris Palmer <ceph-users(a)ceph.io>: > Can anyone confirm whether OSDs that were adopted into cephadm but > remain unmanaged will be automatically rotated? (Currently on 20.2.3). > > On 20/08/2026 14:45, Adam King wrote: >> That command got removed for the time being as it used an old way of doing >> the key rotation that wasn't built to work with the changes for the cve and >> seemed to be broken for a number of daemon types. Since the name would make >> people want to use it it was decided to just temporarily disables it so we >> can fix it and reintroduce it in a new version as an easier way to rotate >> the keys. For any keys tied to a cephadm deployed daemon that aren't >> automatically rotated (cephadm handles mgr, mon, osd, and mds keyrings >> during upgrade) it's a matter of `ceph auth rotate <entity-name> --key-type >> aes256k` then `ceph orch daemon redeploy <daemon-name>`. For anyone using >> nfs exports managed through the nfs module the export's keyrings also need >> to be rotated. `ceph auth rotate <export-keyring-entity> --key-type >> aes256k`, `ceph nfs export info <nfs-cluster-name> <pseudo-path> > >> export_file`, `ceph nfs export apply <nfs-cluster-name> -i export_file`. >> Between rotating the keyrings for the nfs cluster/daemons and rotating the >> keyring for the individual export(s) those export(s) are unavailable iirc. >> >> On Thu, Aug 20, 2026 at 7:11 AM Pedro Macedo <ceph-users(a)ceph.io> wrote: >> >>> On 2026-08-20 12:57 PM, Maximilian Stinsky-Damke wrote: >>>> Hi, >>>> >>>> I upgraded a test cluster to Ceph Squid 19.2.6 via cephadm. The >>> following cephadm managed client keys were not rotated automatically: >>>> client.bootstrap-* (bootstrap-rgw, bootstrap-osd, ...) >>>> client.ceph-exporter. >>>> client.crash. >>>> client.osd-lockbox.. >>>> client.rgw. >>>> >>>> The Squid docs mention ceph orch daemon rotate-key <name>, but in 19.2.* >>> this command does not exist. >>> >>> Also seeing this on the 20.2.4 image via cephadm - rotate-key is missing >>> there, but present on 20.2.3. >>> >>> https://docs.ceph.com/en/squid/cephadm/services/#deploying-a-daemon-on-a-ho… >>> https://github.com/ceph/ceph/blob/v19.2.6/src/pybind/mgr/cephadm/module.py#… >>>> What is the correct way to rotate these client keys via cephadm as part >>> of the cve fix? >>>> Thanks for any guidance. >>>> Best Regards >>>> Max >>>> >>>> ________________________________ >>>> From: Patrick Donnelly <ceph-users(a)ceph.io> >>>> Sent: 19 August 2026 18:08 >>>> To: ceph-users <ceph-users(a)ceph.io> >>>> Subject: [ceph-users] [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 >>> released >>>> Warning: This email is from an unusual correspondent. >>>> Warning: Make sure this is someone you trust. >>>> >>>> Today, August 19th, 2026, the Ceph project has released Tentacle >>>> 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several >>>> components. >>>> >>>> This is the sixth backport release in the Squid series. >>>> >>>> This is the fourth backport release in the Tentacle series. >>>> >>>> Additional details will be posted on the Ceph website/blog: >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fceph.io%2Fen%2Fnews%2Fblog%2F… >>>> We strongly recommend that all Ceph operators upgrade to one of these >>>> releases as soon as possible. >>>> >>>> >>>> ## Critical Upgrade Steps >>>> >>>> The fix for CVE-2025-30156 introduces a new CephX key type, aes256k. >>>> This is the first time Ceph has introduced a new key type for CephX >>>> credentials. Therefore, a new procedure exists for upgrading and >>>> rotating Ceph daemon keys as part of this CVE. Before upgrading a Ceph >>>> cluster, the operator should familiarize themselves with the procedure >>>> documented at: >>>> >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> These steps are for package-based deployments of Ceph. In other >>>> sections, the document also includes new information about key >>>> rotation procedures, monitor authentication controls, and emergency >>>> recovery procedures. >>>> >>>> Deployments using cephadm will automate the process except for client >>>> keys. You may notice that Cephadm spends more time than normal on the >>>> upgrade after updating all daemon images. This is due to a new process >>>> rotating the OSD and MDS keys. Please be patient. There are plans to >>>> improve this in the future. >>>> >>>> Deployments using Rook will also automate rotation of some client keys >>>> with some exclusions. Look to the Rook project’s announcement for more >>>> details. >>>> >>>> Client and kernel upgrades are recommended to support aes256k. Note >>>> that upstream Linux kernel client support began in kernel 7.0 and has >>>> been backported to CentOS Stream 9 and 10. Check with your >>>> distribution vendor for backported key support before rotating client >>>> keys used by the kernel. >>>> >>>> As part of upgrading a Ceph cluster, six new health warnings and >>>> errors will be generated. This is normal. As part of the process to >>>> upgrade/rotate entity keys, you will address the warnings and errors >>>> as you go. These are documented beginning here: >>>> >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> The CephX documentation has been brought up to date and vetted for >>>> accuracy. Those changes are live here: >>>> >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> Regarding the fix for CVE-2026-54330, RGW will now reject Sigv4 >>>> requests with `host` and `x-amz-` headers not included in the signed >>>> subset. Unfortunately, the REST client used in multisite was >>>> generating such improperly signed requests. >>>> >>>> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >>>> option to true before you begin to upgrade. After all clusters are >>>> upgraded, set the option to ``false`` again. >>>> >>>> Regarding the fix for CVE-2026-50152, formal guidance on rotating all >>>> secrets stored in the Monitor config-key store will be forthcoming. In >>>> the meantime, operators of cephadm-managed clusters are strongly >>>> encouraged to rotate the cephadm SSH key, as an established process >>>> exists: >>>> >>>> See: >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> Rotating other secrets is advised but the process is not yet >>>> established. Operators should assess their cluster's potential >>>> exposure to determine if immediate Monitor config-key secret rotation >>>> is required for their environment. >>>> >>>> >>>> ## CVE Fixes >>>> >>>> * CVE-2025-30156 is an authentication bypass in CephX caused by misuse >>>> of AES-CBC. >>>> >>>> See: >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> * CVE-2026-39944 shares the unauthenticated-encryption root cause of >>>> CVE-2025-30156, but applies it to RGW's STS session tokens resulting >>>> in improper verification of a cryptographic signature. >>>> >>>> See: >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> * CVE-2026-50152 is an improper authorization flaw in the Ceph Monitor >>>> subscription handler. >>>> >>>> See: >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> * CVE-2026-54330 is a flaw in RGW not properly verifying its SigV4 >>>> cryptographic signatures in RGW's SigV4 verifier. >>>> >>>> See: >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> ## Other Media >>>> >>>> A Cephalocon 2025 talk covers some of the details surrounding the new >>>> CephX key upgrade process. >>>> >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fcephalocon2025.sched.com%2Fev… >>>> ## Getting Ceph >>>> ------------ >>>> * Source at >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fgithub.com%2Fceph%2Fceph&e=50… >>>> * v19.2.6 Tarball at >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… >>>> * v20.2.4 Tarball at >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdownload.ceph.com%2Ftarballs%… >>>> * Containers at >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fquay.io%2Frepository%2Fceph%2… >>>> * For packages, see >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> * For upgrade with cephadm, see: >>>> >>> https://urlsand.esvalabs.com/?u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Flatest%2… >>>> * v19.2.6 Release git sha1: f9fd95b4335bad6a26d7a74468f55d269e8dbef4 >>>> * v20.2.4 Release git sha1: 7f793731f1b39eb4f465e960113d2363c311b964 >>>> >>>> -- >>>> Patrick Donnelly, Ph.D. >>>> He / Him / His >>>> Red Hat Partner Engineer >>>> IBM, Inc. >>>> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>>> -- >>>> This message has been checked by Libraesva ESG and is found to be clean. >>>> Follow this link to report as spam/bad: >>>> https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/report-as-bad >>>> Follow this link to blocklist sender: >>>> https://mx10.wiit.cloud/action/4hQBLz2RJPzTh0J/blocklist >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Insecure keys allowed (aes)
by GLE, Vivien 25 Aug '26

25 Aug '26

Hi, I just upgrade (from tentacle 20.2.1 to 20.2.4) and rolled CephX keys for a Cephadm cluster but there is still an error concerning the use of insecure keys #ceph config get mon mon_auth_allow_insecure_key false #ceph health detail HEALTH_WARN Monitors are configured to allow auth using insecure key types [WRN] AUTH_INSECURE_KEYS_ALLOWED: Monitors are configured to allow auth using insecure key types insecure cipher aes allowed for auth Didn't found any keys named like this in the ceph configuration but in the ceph mon dump #ceph mon dump auth_epoch 0 auth_service_cipher aes256k auth_allowed_ciphers aes, aes256k <= = auth_preferred_cipher aes256k dumped monmap epoch 15 Is the only way to get rid of this error is to manipulate the mon map ? How can I do it safely ? Thanks

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Eugen Block 25 Aug '26

25 Aug '26

Hi, you're correct, the option doesn't exist in older clusters (I only checked one Reef and one Squid version < 19.2.6). I assume it's recommended to perform a staggered upgrade [0], and before you upgrade the RGWs, you set the config option. I guess that could have been made a bit clearer in the docs. The staggered upgrade works in this test cluster (I don't have multisite enabled): reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.6 --daemon-types mgr,mon,crash,osd,mds wait a bit... reef1:~ # ceph versions { "mon": { "ceph version 19.2.6 (f9fd95b4335bad6a26d7a74468f55d269e8dbef4) squid (stable)": 3 }, "mgr": { "ceph version 19.2.6 (f9fd95b4335bad6a26d7a74468f55d269e8dbef4) squid (stable)": 2 }, "osd": { "ceph version 19.2.6 (f9fd95b4335bad6a26d7a74468f55d269e8dbef4) squid (stable)": 5 }, "rgw": { "ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable)": 2 }, "overall": { "ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable)": 2, "ceph version 19.2.6 (f9fd95b4335bad6a26d7a74468f55d269e8dbef4) squid (stable)": 10 } } reef1:~ # ceph config set client.rgw rgw_sigv4_insecure true reef1:~ # reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.6 --daemon-types rgw And then upgrade the rest (omit daemon-types). Hope this helps! Eugen [0] https://docs.ceph.com/en/latest/cephadm/upgrade/#staggered-upgrade Zitat von Matthew Vernon <ceph-users(a)ceph.io>: > Hi, > > On 19/08/2026 17:09, Patrick Donnelly wrote: > >> If you are running multisite, you must set the ``rgw_sigv4_insecure`` >> option to true before you begin to upgrade. After all clusters are >> upgraded, set the option to ``false`` again. > > Stupid question - how/where does one set this, and will this be > understood by older versions of Ceph? > > I ask because I have a Reef multisite setup, which should be > supported to upgrade to Tentacle. Which I fear I'm now going to have > to do in a hurry! > > AFAICT this is a new config setting, so I'd expect older versions of > Ceph won't understand it... [and searching the reef docs doesn't > find it] > > Thanks, > > Matthew > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Boris 25 Aug '26

25 Aug '26

Hi Matthew, we've set it with: ceph config set client.rgw rgw_sigv4_insecure true --force Best Boris Am Di., 25. Aug. 2026 um 09:41 Uhr schrieb Matthew Vernon <ceph-users(a)ceph.io>: > > Hi, > > On 19/08/2026 17:09, Patrick Donnelly wrote: > > > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > > option to true before you begin to upgrade. After all clusters are > > upgraded, set the option to ``false`` again. > > Stupid question - how/where does one set this, and will this be > understood by older versions of Ceph? > > I ask because I have a Reef multisite setup, which should be supported > to upgrade to Tentacle. Which I fear I'm now going to have to do in a hurry! > > AFAICT this is a new config setting, so I'd expect older versions of > Ceph won't understand it... [and searching the reef docs doesn't find it] > > Thanks, > > Matthew > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

Re: [CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released
by Matthew Vernon 25 Aug '26

25 Aug '26

Hi, On 19/08/2026 17:09, Patrick Donnelly wrote: > If you are running multisite, you must set the ``rgw_sigv4_insecure`` > option to true before you begin to upgrade. After all clusters are > upgraded, set the option to ``false`` again. Stupid question - how/where does one set this, and will this be understood by older versions of Ceph? I ask because I have a Reef multisite setup, which should be supported to upgrade to Tentacle. Which I fear I'm now going to have to do in a hurry! AFAICT this is a new config setting, so I'd expect older versions of Ceph won't understand it... [and searching the reef docs doesn't find it] Thanks, Matthew

1 0

Re: Crush OSD mapping updates when OSD marked out
by Sam Goyal 24 Aug '26

24 Aug '26

> So clearly some modification to the crush map is needed or some post processing of the crush mappings are in order to account for OSDs marked out. Someone can correct me if I am wrong, but after a quick look at the code, the crushmap is not touched when an OSD is marked out. The new weight is stored in the OSDMap. When the new epoch of this map is sent to the OSDs, CRUSH calculates the new PG mappings, skipping the OSD with weight 0 (passed down from the new map). The new mappings differ from the old ones in this case, which triggers peering. > I.e. how to simulate a reweight (to 0 for being out, for any other number other than 1.0 - same problem). I.e. I have a crush map, I'd like to understand how to determine which PGs get remapped to what OSDs when I reweight an OSD from 1.0 to something lower. I think you can use osdmaptool with --test-map-pgs-dump" and "--mark-out" flag to simulate an out OSD. ( https://docs.ceph.com/en/latest/man/8/osdmaptool/#cmdoption-osdmaptool-mark… ) I don't think just the crushmap alone can achieve this. I'll let someone else confirm this. You can also simulate changing the crush weights using the "--adjust-crush-weight" flag. Thanks, Sam On Mon, 24 Aug 2026 at 15:33, Andras Pataki <ceph-users(a)ceph.io> wrote: > I'm trying to understand how 'reweighting' the OSD affects the placement > of the PGs. I.e. how to simulate a reweight (to 0 for being out, for > any other number other than 1.0 - same problem). > I.e. I have a crush map, I'd like to understand how to determine which > PGs get remapped to what OSDs when I reweight an OSD from 1.0 to > something lower. > > Andras > > > On 8/24/26 5:47 PM, Joshua Blanch wrote: > >> The "reweight" field that is set from 1.0 to 0.0 > > Have you tried reweighting to a non-zero, near-zero crush weight, e.g., > > 0.0001? That should leave it in the crush map and simulate it being > 'out'. > > > > - Joshua > > > > On Mon, 24 Aug 2026 at 14:12, Andras Pataki <ceph-users(a)ceph.io> wrote: > > > >> Hi friends of ceph, > >> > >> I'd like to better understand how ceph moves PGs off of OSDs that are > >> marked out for the purposes of simulating potential setup changes. > >> When all OSDs are in (and no upmaps are present), I can use crushtool to > >> find the PG to OSDs mappings of all PGs in the cluster given a crush map > >> (for example: "crushtool -i crushmap.compiled --test --show-mappings > >> --num-rep 3 --min-x 0 --max-x 511 --rule 3 --pool-id 3" would do this > >> for one of our pools). If I change any weights in the crush map, I can > >> recompile it, rerun crushtool to get an updated mapping, and I have > >> found this to perfectly match what ceph does when the crush map is > >> loaded into the cluster. > >> > >> What I am a bit unsure about is how to get the PG to OSD mappings when > >> an OSD is marked out. The "reweight" field that is set from 1.0 to 0.0 > >> when an OSD is marked out is not in the crush map, it is in the osdmap > >> only. So clearly some modification to the crush map is needed or some > >> post processing of the crush mappings are in order to account for OSDs > >> marked out. I originally thought that changing the OSD weight to 0.0 in > >> the crush map without modifying the parent bucket weights would do the > >> trick. While this approach does remap the correct PGs (the ones that > >> originally contain the OSD that is marked out), the new mappings are not > >> the right ones (i.e. the new mappings do not match what live ceph does). > >> > >> So my question if someone can enlighten me or point me to the right > >> direction: starting with a crush map, what do I need to do to simulate > >> the mappings for an OSD that is marked out? I.e. what changes are > >> needed beyond what crushtool --show-mappings would give and how to > >> calculate those offline (without touching a live cluster)? I've tried > >> looking at the code (OSDMap.cc) - but couldn't quite understand where > >> the adjustments are made for OSDs marked out. Any > >> pointers/hints/insights would be appreciated. > >> > >> Thanks, > >> > >> Andras > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Crush OSD mapping updates when OSD marked out
by Andras Pataki 24 Aug '26

24 Aug '26

I'm trying to understand how 'reweighting' the OSD affects the placement of the PGs. I.e. how to simulate a reweight (to 0 for being out, for any other number other than 1.0 - same problem). I.e. I have a crush map, I'd like to understand how to determine which PGs get remapped to what OSDs when I reweight an OSD from 1.0 to something lower. Andras On 8/24/26 5:47 PM, Joshua Blanch wrote: >> The "reweight" field that is set from 1.0 to 0.0 > Have you tried reweighting to a non-zero, near-zero crush weight, e.g., > 0.0001? That should leave it in the crush map and simulate it being 'out'. > > - Joshua > > On Mon, 24 Aug 2026 at 14:12, Andras Pataki <ceph-users(a)ceph.io> wrote: > >> Hi friends of ceph, >> >> I'd like to better understand how ceph moves PGs off of OSDs that are >> marked out for the purposes of simulating potential setup changes. >> When all OSDs are in (and no upmaps are present), I can use crushtool to >> find the PG to OSDs mappings of all PGs in the cluster given a crush map >> (for example: "crushtool -i crushmap.compiled --test --show-mappings >> --num-rep 3 --min-x 0 --max-x 511 --rule 3 --pool-id 3" would do this >> for one of our pools). If I change any weights in the crush map, I can >> recompile it, rerun crushtool to get an updated mapping, and I have >> found this to perfectly match what ceph does when the crush map is >> loaded into the cluster. >> >> What I am a bit unsure about is how to get the PG to OSD mappings when >> an OSD is marked out. The "reweight" field that is set from 1.0 to 0.0 >> when an OSD is marked out is not in the crush map, it is in the osdmap >> only. So clearly some modification to the crush map is needed or some >> post processing of the crush mappings are in order to account for OSDs >> marked out. I originally thought that changing the OSD weight to 0.0 in >> the crush map without modifying the parent bucket weights would do the >> trick. While this approach does remap the correct PGs (the ones that >> originally contain the OSD that is marked out), the new mappings are not >> the right ones (i.e. the new mappings do not match what live ceph does). >> >> So my question if someone can enlighten me or point me to the right >> direction: starting with a crush map, what do I need to do to simulate >> the mappings for an OSD that is marked out? I.e. what changes are >> needed beyond what crushtool --show-mappings would give and how to >> calculate those offline (without touching a live cluster)? I've tried >> looking at the code (OSDMap.cc) - but couldn't quite understand where >> the adjustments are made for OSDs marked out. Any >> pointers/hints/insights would be appreciated. >> >> Thanks, >> >> Andras >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Crush OSD mapping updates when OSD marked out
by Joshua Blanch 24 Aug '26

24 Aug '26

> The "reweight" field that is set from 1.0 to 0.0 Have you tried reweighting to a non-zero, near-zero crush weight, e.g., 0.0001? That should leave it in the crush map and simulate it being 'out'. - Joshua On Mon, 24 Aug 2026 at 14:12, Andras Pataki <ceph-users(a)ceph.io> wrote: > Hi friends of ceph, > > I'd like to better understand how ceph moves PGs off of OSDs that are > marked out for the purposes of simulating potential setup changes. > When all OSDs are in (and no upmaps are present), I can use crushtool to > find the PG to OSDs mappings of all PGs in the cluster given a crush map > (for example: "crushtool -i crushmap.compiled --test --show-mappings > --num-rep 3 --min-x 0 --max-x 511 --rule 3 --pool-id 3" would do this > for one of our pools). If I change any weights in the crush map, I can > recompile it, rerun crushtool to get an updated mapping, and I have > found this to perfectly match what ceph does when the crush map is > loaded into the cluster. > > What I am a bit unsure about is how to get the PG to OSD mappings when > an OSD is marked out. The "reweight" field that is set from 1.0 to 0.0 > when an OSD is marked out is not in the crush map, it is in the osdmap > only. So clearly some modification to the crush map is needed or some > post processing of the crush mappings are in order to account for OSDs > marked out. I originally thought that changing the OSD weight to 0.0 in > the crush map without modifying the parent bucket weights would do the > trick. While this approach does remap the correct PGs (the ones that > originally contain the OSD that is marked out), the new mappings are not > the right ones (i.e. the new mappings do not match what live ceph does). > > So my question if someone can enlighten me or point me to the right > direction: starting with a crush map, what do I need to do to simulate > the mappings for an OSD that is marked out? I.e. what changes are > needed beyond what crushtool --show-mappings would give and how to > calculate those offline (without touching a live cluster)? I've tried > looking at the code (OSDMap.cc) - but couldn't quite understand where > the adjustments are made for OSDs marked out. Any > pointers/hints/insights would be appreciated. > > Thanks, > > Andras > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0