January 2026 - ceph-users

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

cephadm without containers
by Adrian Sevcenco 10 Apr '26

10 Apr '26

Hi! Is there any possibility to use cephadm without the container based tooling? (so to use bare-metal, with the packages directly installed on the machine) If not, is there something technical that prevents it to happen or just that there is a policy that container based deployment is the only best way to do it therefore cephadm will only use it this way? I ask this in the context of finding out that ceph-ansible is deprecated and that it should not be used, but i still need to use ansible for management and i think (so without factual experience) that bare metal services would be easier to manage after doing by hand deployment with cephadm Thank you! Adrian

11 20

CephFS: no MDS does join the filesystem
by Robert Sander 24 Mar '26

24 Mar '26

Hi, we are having an issue at a customer site where a 3PB CephFS is in failed state. The cluster itself is unhealthy and awaits replacements disks: # ceph -s cluster: id: 28ca2bfa-d87e-11ed-83a3-1070fddda30f health: HEALTH_ERR 4 failed cephadm daemon(s) There are daemons running an older version of ceph 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged 8 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 46 pgs backfill_toofull Possible data damage: 4 pgs inconsistent Degraded data redundancy: 6427646/15858772167 objects degraded (0.041%), 8 pgs degraded, 8 pgs undersized 6 pool(s) nearfull (muted: OSDMAP_FLAGS OSD_SCRUB_ERRORS(2d) PG_NOT_DEEP_SCRUBBED PG_NOT_SCRUBBED) services: mon: 3 daemons, quorum sn01,sn03,sn02 (age 3w) mgr: sn03.crlpzh(active, since 33h), standbys: sn01.tegfya, sn02.mzvgcr mds: 18/19 daemons up, 1 standby osd: 181 osds: 174 up (since 4d), 172 in (since 4d); 206 remapped pgs flags nodeep-scrub data: volumes: 2/3 healthy, 1 recovering; 1 damaged pools: 12 pools, 3585 pgs objects: 1.93G objects, 1.3 PiB usage: 2.5 PiB used, 501 TiB / 3.0 PiB avail pgs: 6427646/15858772167 objects degraded (0.041%) 293845758/15858772167 objects misplaced (1.853%) 2844 active+clean 532 active+clean+scrubbing 147 active+remapped+backfill_wait 28 active+remapped+backfill_toofull 11 active+remapped+backfill_wait+backfill_toofull 10 active+remapped+backfilling 6 active+undersized+degraded+remapped+backfill_toofull 2 active+clean+inconsistent 1 active+clean+scrubbing+deep+inconsistent+repair 1 active+undersized+remapped+backfilling 1 active+undersized+degraded+remapped+backfilling 1 active+recovering+degraded+remapped 1 active+remapped+inconsistent+backfill_toofull io: recovery: 183 MiB/s, 312 objects/s The CephFS metadata pool is not affected by the inconsistent PGs. The MDSs have this line in their logfile: "Monitors have assigned me to become a standby." The filesystem is joinable: # ceph fs lsflags storage_cluster joinable allow_snaps allow_multimds_snaps refuse_client_session But no MDS joins: # ceph fs status storage_cluster - 0 clients =============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed POOL TYPE USED AVAIL cephfs_metadata metadata 490G 12.9T cephfs_data data 970T 54.1T shared_data data 1351T 22.5T STANDBY MDS storage_cluster.sn04.cbvzzu MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) Why? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

6 13

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

reef 18.2.8 QE validation status
by Yuri Weinstein 13 Feb '26

13 Feb '26

Note: this is an attempt to test and publish post EOL reef point release Details of this release are summarized here: https://tracker.ceph.com/issues/73906#note-1 Release Notes - TBD LRC upgrade - N/A distro => jammy focal centos9 windows bookworm Seeking approvals/reviews for: smoke - Laura approved? rados - Radek, Laura approved? Travis? Nizamudeen? Adam King approved? (rados is also running on build 2 == build 1 + https://github.com/ceph/ceph/pull/66056 per Igor's request) rgw - Adam E approved? fs - Venky approved? orch - Adam King approved? rbd - Ilya approved? krbd - Ilya approved? upgrade/pacific-x (reef) - can this be deprecated? Josh? Neha? (same as in 18.2.5) upgrade/quincy-x (reef) - Laura, Prashant please take a look. (same as in 18.2.5) powercycle - Brad approved? ceph-volume - see orch, Guillaume approved? Notably upgrades suites show same results as for 18.2.5 (tested in March 2025) To compare, see 18.2.5 tests https://tracker.ceph.com/issues/70563#note-1 TIA

4 5

RGW period map issues and zone name collision
by Greg O'Neill 10 Feb '26

10 Feb '26

Hey folks, I'm having an issue with name collision of rgw zones across different realms in three multi-site clusters. Clusters are setup with bi-directional multi-site peering in addition to a local realm in each cluster for container images and thanos metrics. All clusters are running Reef v18.2.2 The local realm's `default` zone from the primary cluster's multi-site realm `cloud` was added to the period map by mistake and it propogated to the replica clusters. The zone name collided with the existing `default` zone in the `default` realm. I see the default zone in both pcloud and default realms with the same UUID. When I run `radosgw-admin period get --rgw-realm default` on the replica clusters, it returns with no zones or zonegroups when it should have one each named `default`. In a test cluster, I tried renaming or deleting the `default` zone from the multi-site realm but it knocks out any zones named `default` regardless of realm. ### Primary cluster (west coast) goneill@us-west-2-ceph-mon1:~$ sudo radosgw-admin period get --rgw-realm default | jq -r '.period_map.zonegroups[].zones[] | [.id, .name, .endpoints[]] | @tsv' 37929bb2-2eb3-4827-952a-6b5ca05627a6 default http://us-west-2-ceph-mon1:8080 goneill@us-west-2-ceph-mon1:~$ sudo radosgw-admin period get --rgw-realm savagebucket | jq -r '.period_map.zonegroups[].zones[] | [.id, .name, .endpoints[]] | @tsv' 250a2a20-5e11-486b-bae4-e29b173d2556 thanos 107d29a0-b732-4bf1-a26e-1f64f820e839 us-east-1-prod http://us-east-1-ceph-rgw1:8080 1e27bf9c-3a2f-4845-85b6-33a24bbe1c04 us-west-2-corp http://us-west-2-ceph-rgw1:8080 331d3f1e-1b72-4c56-bb5a-d1d0fcf6d0b8 us-west-1-prod http://us-west-1-ceph-rgw1:8080 37929bb2-2eb3-4827-952a-6b5ca05627a6 default ### Replica cluster (west coast) goneill@us-west-1-ceph-mon1:~$ sudo radosgw-admin period get --rgw-realm default | jq -r '.period_map.zonegroups[].zones[] | [.id, .name, .endpoints[]] | @tsv' goneill@us-west-1-ceph-mon1:~$ sudo radosgw-admin period get --rgw-realm savagebucket | jq -r '.period_map.zonegroups[].zones[] | [.id, .name, .endpoints[]] | @tsv' 250a2a20-5e11-486b-bae4-e29b173d2556 thanos 107d29a0-b732-4bf1-a26e-1f64f820e839 us-east-1-prod http://us-east-1-ceph-rgw1:8080 1e27bf9c-3a2f-4845-85b6-33a24bbe1c04 us-west-2-corp http://us-west-2-ceph-rgw1:8080 331d3f1e-1b72-4c56-bb5a-d1d0fcf6d0b8 us-west-1-prod http://us-west-1-ceph-rgw1:8080 37929bb2-2eb3-4827-952a-6b5ca05627a6 default ### Replica cluster (east coast) goneill@us-east-1-ceph-mon1:~$ sudo radosgw-admin period get --rgw-realm default | jq -r '.period_map.zonegroups[].zones[] | [.id, .name, .endpoints[]] | @tsv' goneill@us-east-1-ceph-mon1:~$ sudo radosgw-admin period get --rgw-realm savagebucket | jq -r '.period_map.zonegroups[].zones[] | [.id, .name, .endpoints[]] | @tsv' 250a2a20-5e11-486b-bae4-e29b173d2556 thanos 107d29a0-b732-4bf1-a26e-1f64f820e839 us-east-1-prod http://us-east-1-ceph-rgw1:8080 1e27bf9c-3a2f-4845-85b6-33a24bbe1c04 us-west-2-corp http://us-west-2-ceph-rgw1:8080 331d3f1e-1b72-4c56-bb5a-d1d0fcf6d0b8 us-west-1-prod http://us-west-1-ceph-rgw1:8080 37929bb2-2eb3-4827-952a-6b5ca05627a6 default

1 1

Debian Packages for Trixie
by Andrew 09 Feb '26

09 Feb '26

Hello Team, I can see that there are official packages for Debian Old-Stable (Bookworm) available on the official site: https://download.ceph.com/debian-19.2.3/dists/ Does anyone know if there is an approximate (or likely) ETA for the Debian Packages to be released for current stable version (Trixie)? Many thanks, Andrew

6 6

Re: Urgent help needed.
by gagan tiwari 31 Jan '26

31 Jan '26

Hi Boris, Its 7 osds nodes out of which 2 are acting as mon also. 1 mgr + mon node and 2 dedicated mon nodes Thanks, Gagan On Sat, Jan 31, 2026 at 6:45 PM Boris via ceph-users <ceph-users(a)ceph.io> wrote: > Hi Gagan, > do I understand correctly that you have dedicated 5 Mon, 1 Mgr and 7 OSD > nodes? 13 nodes i total? > > And you only need to wipe 3 Mon and the mgr nodes? > > If this is the case, you will not have problems with the data on the OSDs. > You can just move the mgr to one of the good mon nodes and remove the mon > nodes from quorum. > > The cluster will be fine. You might need to adjust the ceph.conf on the > clients, to not ise the mons you removed from the quorum. > > Alternatively you can just wipe and reinstall the systems. Ceph should > handle one missing node fine. > > You also rarely need a dedicated node for mgr. you can run them together > with the mons. They will failover of the acting mgr is not available > anymore. > > Cheers > Boris > > > Am 31.01.2026 um 13:02 schrieb gagan tiwari via ceph-users < > ceph-users(a)ceph.io>: > > > > HI Guys, > > Our ceph cluster seems to be compromised by rootkit > > attack. > > > > It has 5 mon including one mgr and 7 osd nodes. > > > > We will need to reformat and reinstall OS on ceph mgr node and 2 ceph mon > > nodes. > > > > 2 mon nodes and 7 osd nodes are safe. > > > > Ceph health status is showing ok. And the cluster is still working but > very > > slow. > > > > Please let me know how to proceed to have minimal disruption to the users > > and without any data loss > > > > Thanks, > > Gagan > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: Urgent help needed.
by William Edwards 31 Jan '26

31 Jan '26

William David Edwards > Op 31 jan 2026 om 13:02 heeft gagan tiwari via ceph-users <ceph-users(a)ceph.io> het volgende geschreven: > > HI Guys, > Our ceph cluster seems to be compromised by rootkit > attack. > How did you determine that? > It has 5 mon including one mgr and 7 osd nodes. > > We will need to reformat and reinstall OS on ceph mgr node and 2 ceph mon > nodes. > > 2 mon nodes and 7 osd nodes are safe. > > Ceph health status is showing ok. And the cluster is still working but very > slow. Why slow? > > Please let me know how to proceed to have minimal disruption to the users > and without any data loss What have you tried? What’s your plan so far? > > Thanks, > Gagan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Urgent help needed.
by Boris 31 Jan '26

31 Jan '26

Hi Gagan, do I understand correctly that you have dedicated 5 Mon, 1 Mgr and 7 OSD nodes? 13 nodes i total? And you only need to wipe 3 Mon and the mgr nodes? If this is the case, you will not have problems with the data on the OSDs. You can just move the mgr to one of the good mon nodes and remove the mon nodes from quorum. The cluster will be fine. You might need to adjust the ceph.conf on the clients, to not ise the mons you removed from the quorum. Alternatively you can just wipe and reinstall the systems. Ceph should handle one missing node fine. You also rarely need a dedicated node for mgr. you can run them together with the mons. They will failover of the acting mgr is not available anymore. Cheers Boris > Am 31.01.2026 um 13:02 schrieb gagan tiwari via ceph-users <ceph-users(a)ceph.io>: > > HI Guys, > Our ceph cluster seems to be compromised by rootkit > attack. > > It has 5 mon including one mgr and 7 osd nodes. > > We will need to reformat and reinstall OS on ceph mgr node and 2 ceph mon > nodes. > > 2 mon nodes and 7 osd nodes are safe. > > Ceph health status is showing ok. And the cluster is still working but very > slow. > > Please let me know how to proceed to have minimal disruption to the users > and without any data loss > > Thanks, > Gagan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Urgent help needed.
by gagan tiwari 31 Jan '26

31 Jan '26

HI Guys, Our ceph cluster seems to be compromised by rootkit attack. It has 5 mon including one mgr and 7 osd nodes. We will need to reformat and reinstall OS on ceph mgr node and 2 ceph mon nodes. 2 mon nodes and 7 osd nodes are safe. Ceph health status is showing ok. And the cluster is still working but very slow. Please let me know how to proceed to have minimal disruption to the users and without any data loss Thanks, Gagan

1 0

[ceph-user][ceph-ansible] Quincy: Module 'devicehealth' has failed: unknown operation
by Danish Khan 31 Jan '26

31 Jan '26

Dear Team, I am keep getting below error after every few days: Error : Module 'devicehealth' has failed: unknown operation Actual error from MGR log: 2026-01-30T05:50:59.338+0530 7f314da26640 0 [devicehealth ERROR root] Caught fatal database error: unknown operation 2026-01-30T05:50:59.338+0530 7f314da26640 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'devicehealth' while running on mgr.controller3: unknown operation 2026-01-30T05:50:59.338+0530 7f314da26640 -1 devicehealth.serve: 2026-01-30T05:50:59.338+0530 7f314da26640 -1 Traceback (most recent call last): File "/usr/share/ceph/mgr/devicehealth/module.py", line 408, in serve self._do_serve() File "/usr/share/ceph/mgr/mgr_module.py", line 513, in check return func(self, *args, **kwargs) File "/usr/share/ceph/mgr/devicehealth/module.py", line 399, in _do_serve self.scrape_all() File "/usr/share/ceph/mgr/devicehealth/module.py", line 449, in scrape_all self.put_device_metrics(device, data) File "/usr/share/ceph/mgr/devicehealth/module.py", line 525, in put_device_metrics self._create_device(devid) File "/usr/share/ceph/mgr/devicehealth/module.py", line 511, in _create_device cursor = self.db.execute(SQL, (devid,)) sqlite3.InternalError: unknown operation 2026-01-30T05:50:59.366+0530 7f3155335640 0 log_channel(cluster) log [DBG] : pgmap v116150: 6977 pgs: 92 peering, 40 stale+active+clean, 191 active+clean+laggy, 30 active+undersized, 1026 active+undersized+degraded, 5598 active+clean; 48 TiB data, 153 TiB used, 1.2 PiB / 1.4 PiB avail; 44 MiB/s wr, 1.00k op/s; 8043806/154077270 objects degraded (5.221%); 4.4 GiB/s, 8 keys/s, 530 objects/s recovering And I get one warning for mgr daemon crash but all MGR Service actually doesn't get failed or restarted automatically. I guess it is something related to sqlite3 which checks ceph health internally. Although this is not causing any issue and error disappears after restarting ceph-mgr service or doing a failover of MGR to another controller. Does anyone know how we can resolve this issue permanently? Similar issue: https://github.com/rook/rook/issues/12349 Regards, Danish

1 0

Ceph Community Updates for January
by Anthony Middleton 30 Jan '26

30 Jan '26

Hey Ceph Friends, How is it still January? It feels like this month has lasted eight weeks, but we keep making our way through it, so congrats to all of us! Here are some updates and events happening in the Ceph ecosystem. - A reminder that the Ceph Days Raleigh CFP is still open! Submit your proposal before February 16: https://docs.google.com/forms/d/e/1FAIpQLSfHUyLahd9EFAIJCF0niveMhszt_F5lgTU… - Check out the recording of this month's Ceph User + Dev Meetup. This month's session ran longer than expected, so we will continue this conversation and cover the remaining agenda topics in April. - Watch the recording here: https://youtu.be/r4c2SMtsVLI. <https://youtu.be/r4c2SMtsVLI> - We held our first Ceph Tech Talk of 2026 on "Running Teuthology Outside of the Sepia Lab." Here is the recording: https://youtu.be/6ovdJ79AqbM - *If you would like to lead a Tech Talk in the upcoming months, please submit your proposal here: https://airtable.com/apphc2dbSP8GuCdor/shrm3t1Fwue5YfAlf. <https://airtable.com/apphc2dbSP8GuCdor/shrm3t1Fwue5YfAlf>* That's all for now. If you have questions or concerns about anything Ceph-related, feel free to reach out to me via email at anthony(a)amicmid.com or on Slack <https://join.slack.com/t/ceph-storage/shared_invite/zt-3jlvf8f6e-45tyKGpqkk…>. Enjoy your weekend! Best, -- Anthony Middleton Community Manager, Ceph Foundation community-manager(a)ceph.io LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: Changing compression algorithm
by Anthony D'Atri 30 Jan '26

30 Jan '26

That is my understanding. You can turn the mode on and off too. There is talk of background recompression but it’s as yet a sparkle in someone’s eye. > On Jan 30, 2026, at 3:41 AM, Torkil Svensgaard via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi > > Am I correct in assuming that we can change the compression algorithm for rbd pools, and everything will keep on running, data already written will just keep whatever compression they have and new data will get the new algorithm? > > Mvh. > > Torkil > -- > Torkil Svensgaard > Sysadmin > MR-Forskningssektionen, afs. 714 > DRCMR, Danish Research Centre for Magnetic Resonance > Hvidovre Hospital > Kettegård Allé 30 > DK-2650 Hvidovre > Denmark > Tel: +45 386 22828 > E-mail: torkil(a)drcmr.dk > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Changing compression algorithm
by Torkil Svensgaard 30 Jan '26

30 Jan '26

Hi Am I correct in assuming that we can change the compression algorithm for rbd pools, and everything will keep on running, data already written will just keep whatever compression they have and new data will get the new algorithm? Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

1 0

Re: OSDs won't start after upgrading reef (18.2.7) to squid (19.2.3) with orchestrator
by Reto Gysi 29 Jan '26

29 Jan '26

Hi Boris, I had a similiar problem in the past (I don't remember which ceph release it was) that the block.db links in /var/lib/ceph/<UUID>/osd.N weren't set correctly by the ceph-volume activate script call in the containers unit.run script. The problem was that I had the block.db on a lvm raid1 mirror on SSDs (so a failure of a single SSD doesn't take down multiple OSDs), and the "ceph-volume activate" script call couldn't properly handle this case. It took one of the entries that had the lv_role private instead of public For example: LV Devices Role ceph-db-osd1 ceph-db-osd1_rimage_0(0),ceph-db-osd1_rimage_1(0) public [ceph-db-osd1_rimage_0] ceph-db-osd1_rimage_0_iorig(0) private,raid,image [ceph-db-osd1_rimage_0_imeta] /dev/sdg(8314) private,integrity,metadata [ceph-db-osd1_rimage_0_iorig] /dev/sdg(9216) private,integrity,origin,integrityorigin [ceph-db-osd1_rimage_0_iorig] /dev/sdg(82518) private,integrity,origin,integrityorigin [ceph-db-osd1_rimage_0_iorig] /dev/sdg(55297) private,integrity,origin,integrityorigin [ceph-db-osd1_rimage_0_iorig] /dev/sdg(59888) private,integrity,origin,integrityorigin [ceph-db-osd1_rimage_0_iorig] /dev/sdg(62448) private,integrity,origin,integrityorigin [ceph-db-osd1_rimage_1] ceph-db-osd1_rimage_1_iorig(0) private,raid,image [ceph-db-osd1_rimage_1_imeta] /dev/sdh(91281) private,integrity,metadata [ceph-db-osd1_rimage_1_iorig] /dev/sdh(1) private,integrity,origin,integrityorigin [ceph-db-osd1_rimage_1_iorig] /dev/sdh(84486) private,integrity,origin,integrityorigin [ceph-db-osd1_rimage_1_iorig] /dev/sdh(89094) private,integrity,origin,integrityorigin [ceph-db-osd1_rmeta_0] /dev/sdg(46080) private,raid,metadata [ceph-db-osd1_rmeta_1] /dev/sdh(0) private,raid,metadata I think I fixed it by manually changing the ceph-volume activate call in the unit.run to ceph-volume lvm activate ceph-volume lvm activate <ID> <FSID>. Luckily my cluster contains not that many OSDs. With ceph tentacle release I no longer had to manually change the unit.run script call. Cheers, Reto Am Di., 27. Jan. 2026 um 11:55 Uhr schrieb Boris via ceph-users < ceph-users(a)ceph.io>: > Hi, > we are currently facing an issue, that suddenly none of the OSDs will start > after the container started with the new versions. > > This seems to be an issue with some hosts/OSDs. The first 30 OSDs worked, > but took really long (like 5 hours) and then every single OSD after that > needed a host reboot to bring the disk back up and continue the update. > > We've stopped after 6 tries. > > And one disk never came back up. We removed and zapped the OSD. The > orchestrator picked the available disk and recreated it. It came up within > seconds. > > We have around 90 clusters and this happened only on a single one. All > others updates within two hours without any issues. > > The cluster uses HDDs (8TB) with the block.db on SSD (5 block.db per SSD). > The file /var/log/ceph/UUID/ceph-volume.log get hammered with a lot of > output from udevadm, lsblk and nsenter > The activation container (ceph-UUID-osd-N-activate) gets killed after a > couple of minutes. > It also looks like the block and block.db links > in /var/lib/ceph/UUID/osd.N/ are not correctly set. > When we restart the daemons that needed a host restart, the OSD doesn't > come up and needs a host restart. > > All OSDs are encrypted. > > Does anyone got some ideas how to debug further? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Untrimmed purged_snaps impacts osd startup, possibly oom killer
by Eugen Block 29 Jan '26

29 Jan '26

Thanks for chiming in, Anthony. I'm already in contact with Adam who helped unblocking my address from receiving emails. I do receive all mails to the list in my inbox(es), but my responses from the first subscribed address don't land in my inbox. I was wondering why my responses seemed to be ignored until I realized this. But I hope that Adam can help me out again. And yes, this happens to newly created OSDs. I'm not sure if commenting on the open (but resolved) tracker is the right place, I wonder if I should create a separate tracker for this. Since it's production relevant, I would like to get as much attention as possible on that matter. Am Do., 29. Jan. 2026 um 17:12 Uhr schrieb Anthony D'Atri via ceph-users < ceph-users(a)ceph.io>: > > > > Hi, it appears that my mails to the lists are only visible on > lists.ceph.io > > but are not sent as emails, so I'm resending from a different account. > > The ongoing relocation of the Sepia lab seems to have disrupted some > member settings. There’s one for whether a given member address receives > list email, you might auth to the list interface and check yours. I am > subscribed with two addresses because sometimes I goof and send from an > address that I’ve been slowly deprecating, but I don’t want duplicate > delivery, so I have delivery turned off for the latter. If an audit of > settings doesn’t improve your experience, let me know and I’ll forward to > the community manager. > > > Each newly created OSD process grows in RAM usage to around 140 GB > within a > > few minutes, > > Yikes! > > > Anyway, the only way to safely create OSDs is one by one, maybe two at > once > > depending on the host's RAM capacity. An automated (unattended) OSD > > deployment is currently not possible. > > This happens to *new* OSDs? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Untrimmed purged_snaps impacts osd startup, possibly oom killer
by Anthony D'Atri 29 Jan '26

29 Jan '26

> Hi, it appears that my mails to the lists are only visible on lists.ceph.io > but are not sent as emails, so I'm resending from a different account. The ongoing relocation of the Sepia lab seems to have disrupted some member settings. There’s one for whether a given member address receives list email, you might auth to the list interface and check yours. I am subscribed with two addresses because sometimes I goof and send from an address that I’ve been slowly deprecating, but I don’t want duplicate delivery, so I have delivery turned off for the latter. If an audit of settings doesn’t improve your experience, let me know and I’ll forward to the community manager. > Each newly created OSD process grows in RAM usage to around 140 GB within a > few minutes, Yikes! > Anyway, the only way to safely create OSDs is one by one, maybe two at once > depending on the host's RAM capacity. An automated (unattended) OSD > deployment is currently not possible. This happens to *new* OSDs?

1 0

Re: OSDs won't start after upgrading reef (18.2.7) to squid (19.2.3) with orchestrator
by Eugen Block 29 Jan '26

29 Jan '26

I was thinking about the same bug you commented, Boris: https://tracker.ceph.com/issues/73107 <https://tracker.ceph.com/issues/73107#change-331393> I am also subscribed to that bug because we upgraded to 19.2.3 a couple of months ago. But since we don’t have that many OSDs per host, we weren’t facing any impact as described in the tracker. Boris via ceph-users <ceph-users(a)ceph.io> schrieb am Mi. 28. Jan. 2026 um 22:19: > Hi Malte, > we just upped the timeout in the service file to 720 (cronjob is going to > set it every minute). > Starting the OSDs takes around 4 minutes. We still think this is an issue > with the ceph-volume activate, because this is taking ages. As soon as the > cryptsetup open the luks devices the rest is going as normal. > > ceph-volume log seems to reflect that: > root@s3db15:~# ls -alh > /var/log/ceph/dca79fff-ffd0-58f4-1cff-82a2feea05f4/ceph-volume.log* > -rw-r--r-- 1 root root 2.4G Jan 27 15:59 > /var/log/ceph/dca79fff-ffd0-58f4-1cff-82a2feea05f4/ceph-volume.log > -rw-r--r-- 1 root root 680M Jan 27 00:00 > /var/log/ceph/dca79fff-ffd0-58f4-1cff-82a2feea05f4/ceph-volume.log.1.gz > -rw-r--r-- 1 root root 5.0M Jan 25 23:46 > /var/log/ceph/dca79fff-ffd0-58f4-1cff-82a2feea05f4/ceph-volume.log.2.gz > -rw-r--r-- 1 root root 4.8M Jan 24 23:35 > /var/log/ceph/dca79fff-ffd0-58f4-1cff-82a2feea05f4/ceph-volume.log.3.gz > -rw-r--r-- 1 root root 4.8M Jan 23 23:45 > /var/log/ceph/dca79fff-ffd0-58f4-1cff-82a2feea05f4/ceph-volume.log.4.gz > -rw-r--r-- 1 root root 4.8M Jan 22 23:38 > /var/log/ceph/dca79fff-ffd0-58f4-1cff-82a2feea05f4/ceph-volume.log.5.gz > -rw-r--r-- 1 root root 4.8M Jan 21 23:39 > /var/log/ceph/dca79fff-ffd0-58f4-1cff-82a2feea05f4/ceph-volume.log.6.gz > -rw-r--r-- 1 root root 4.8M Jan 20 23:38 > /var/log/ceph/dca79fff-ffd0-58f4-1cff-82a2feea05f4/ceph-volume.log.7.gz > --- > # time systemctl restart ceph-dca79fff-ffd0-58f4-1cff-82a2feea05f4(a)osd.135 > > real 3m46.251s > user 0m0.060s > sys 0m0.063s > --- > I will prepare an output of the ceph-collect from 42on. But I need to > remove the sensitive stuff like the rgw config and so. But I don't think > this will give the right clues, as the output won't show what is happening > during the start. > > > Am Di., 27. Jan. 2026 um 13:51 Uhr schrieb Malte Stroem < > malte.stroem(a)gmail.com>: > > > We do not know a lot about you cluster, so it's hard to help. > > > > Give us alle the information one needs. > > > > ceph -s, ceph orch host ls, ceph health detail, all the good things to > > get an overview. > > > > On 1/27/26 12:43, Boris via ceph-users wrote: > > > All disks are 8TB HDD. We have some 16TB HDDs but those are all in the > > > newest host, that updated just fine. > > > > > > Am Di., 27. Jan. 2026 um 12:39 Uhr schrieb Malte Stroem < > > > malte.stroem(a)gmail.com>: > > > > > >> From what I can see, the OSD is running. It's starting up and still > > >> needs time. > > >> > > >> How big is the disk? > > >> > > >> On 1/27/26 12:31, Boris via ceph-users wrote: > > >>> Sure: https://pastebin.com/9RLzyUQs > > >>> > > >>> I've trimmed the log a little bit (removed peering, epoch, trim and > so > > >> on). > > >>> This is the last OSD that we tried that did not work. > > >>> > > >>> We tried another host, where the upgrade just went through. But this > > Host > > >>> also got the newest hardware. > > >>> But we don't think it is a hardware issue, because the first 30 OSDs > > were > > >>> on the two oldest hosts and the first one that failed was on the same > > >> host > > >>> as the last OSD that did not fail. > > >>> > > >>> > > >>> > > >>> Am Di., 27. Jan. 2026 um 12:05 Uhr schrieb Malte Stroem < > > >>> malte.stroem(a)gmail.com>: > > >>> > > >>>> Could be the kind of hardware you are using. Is it different from > the > > >>>> other clusters' hardware? > > >>>> > > >>>> Send us logs, so we can help you out. > > >>>> > > >>>> Example: > > >>>> > > >>>> journalctl -eu ceph-fsid(a)osd.0 > > >>>> > > >>>> Best, > > >>>> Malte > > >>>> > > >>>> On 1/27/26 11:55, Boris via ceph-users wrote: > > >>>>> Hi, > > >>>>> we are currently facing an issue, that suddenly none of the OSDs > will > > >>>> start > > >>>>> after the container started with the new versions. > > >>>>> > > >>>>> This seems to be an issue with some hosts/OSDs. The first 30 OSDs > > >> worked, > > >>>>> but took really long (like 5 hours) and then every single OSD after > > >> that > > >>>>> needed a host reboot to bring the disk back up and continue the > > update. > > >>>>> > > >>>>> We've stopped after 6 tries. > > >>>>> > > >>>>> And one disk never came back up. We removed and zapped the OSD. The > > >>>>> orchestrator picked the available disk and recreated it. It came up > > >>>> within > > >>>>> seconds. > > >>>>> > > >>>>> We have around 90 clusters and this happened only on a single one. > > All > > >>>>> others updates within two hours without any issues. > > >>>>> > > >>>>> The cluster uses HDDs (8TB) with the block.db on SSD (5 block.db > per > > >>>> SSD). > > >>>>> The file /var/log/ceph/UUID/ceph-volume.log get hammered with a lot > > of > > >>>>> output from udevadm, lsblk and nsenter > > >>>>> The activation container (ceph-UUID-osd-N-activate) gets killed > > after a > > >>>>> couple of minutes. > > >>>>> It also looks like the block and block.db links > > >>>>> in /var/lib/ceph/UUID/osd.N/ are not correctly set. > > >>>>> When we restart the daemons that needed a host restart, the OSD > > doesn't > > >>>>> come up and needs a host restart. > > >>>>> > > >>>>> All OSDs are encrypted. > > >>>>> > > >>>>> Does anyone got some ideas how to debug further? > > >>>>> _______________________________________________ > > >>>>> ceph-users mailing list -- ceph-users(a)ceph.io > > >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >>>> > > >>>> > > >>> > > >> > > >> > > > > > > > > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im > groÃƒ¼en Saal. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: Untrimmed purged_snaps impacts osd startup, possibly oom killer
by Eugen Block 29 Jan '26

29 Jan '26

Hi, it appears that my mails to the lists are only visible on lists.ceph.io but are not sent as emails, so I'm resending from a different account. This is kind of a follow-up on a two years old thread [0] and I wanted to raise some awareness for the corresponding tracker [1]. Back then we managed to limit the impact on the mon store performance with some paxos configs, but now the OSDs are impacted as well when there are new ones created: Each newly created OSD process grows in RAM usage to around 140 GB within a few minutes, easily causing oom killers on hosts if multiple OSDs are created at once. The residual RAM usage drops to the memory target after it successfully booted. The reason are the purged_snaps that are loaded during OSD boot (snap_mapper.record_purged_snaps purged_snaps), two years ago the customer had more than 42 million purged_snap entries. I don't know how many there are today, I don't have access myself, but I'll try to get a current number. Anyway, the only way to safely create OSDs is one by one, maybe two at once depending on the host's RAM capacity. An automated (unattended) OSD deployment is currently not possible. Unfortunately, the new command [2] doesn't seem to work as expected, at least in my test cluster it didn't have any impact on the number of purged_snaps in the mon store. That's why we haven't tried it on the customer cluster(s) yet. How do other operators/admins/users deal with this kind of scenario? Having many snapshots can't be a corner case, but I can't remember having read anything like this on the list(s). I'd appreciate any comments Thanks! Eugen

1 0

[MDS crash / ceph_assert failure] CephFS MDS crash on Reef 18.2.2 (Filer::_probed assertion)
by Андрей Муханов 29 Jan '26

29 Jan '26

1 0

Re: Adding new osds on new host
by Paul Mezzanini 28 Jan '26

28 Jan '26

I disagree with your blanket statement of "don't do that". Do what fits your needs, skills, and environment. Use the right tools for the job, and if that is a containerized environment then cephadm all the way. If you have the tooling in place to run a package based install and want to be "closer" to the hardware, then do that instead. -- Paul Mezzanini Platform Engineer III Research Computing Rochester Institute of Technology ________________________________________ From: Malte Stroem via ceph-users <ceph-users(a)ceph.io> Sent: Wednesday, January 28, 2026 9:46 AM To: Anthony D'Atri; Malte Stroem Cc: Rok Jaklič; ceph-users Subject: [ceph-users] Re: Adding new osds on new host Yes, don't do that. Use cephadm. On 1/28/26 14:26, Anthony D'Atri via ceph-users wrote: > The OP may be using a package-based install. > >> On Jan 28, 2026, at 3:14 AM, Malte Stroem via ceph-users <ceph-users(a)ceph.io> wrote: >> >> Use cephadm, "ceph orch" commands to do this. >> >>> On 1/27/26 20:51, Rok Jaklič via ceph-users wrote: >>> Hi, >>> i am adding new OSDs from a new host with something like: >>> <PROD> [root@server1 ~]# ceph-volume lvm create --data /dev/mapper/mpatha >>> Running command: /usr/bin/ceph-authtool --gen-print-key >>> Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd >>> --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new >>> dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> Running command: vgcreate --force --yes >>> ceph-11881314-f5e5-480f-9ac4-d25739e0dd74 /dev/mapper/mpatha >>> stdout: Physical volume "/dev/mapper/mpatha" successfully created. >>> stdout: Volume group "ceph-11881314-f5e5-480f-9ac4-d25739e0dd74" >>> successfully created >>> Running command: lvcreate --yes -l 3814911 -n >>> osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> ceph-11881314-f5e5-480f-9ac4-d25739e0dd74 >>> stdout: Logical volume "osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809" >>> created. >>> Running command: /usr/bin/ceph-authtool --gen-print-key >>> Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-233 >>> Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-233 >>> Running command: /usr/bin/chown -h ceph:ceph >>> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-30 >>> Running command: /usr/bin/ln -s >>> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> /var/lib/ceph/osd/ceph-233/block >>> Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd >>> --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o >>> /var/lib/ceph/osd/ceph-233/activate.monmap >>> stderr: 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 auth: unable to find >>> a keyring on >>> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: >>> (2) No such file or directory >>> 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 AuthRegistry(0x7f9a380648f8) >>> no keyring found at >>> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, >>> disabling cephx >>> stderr: got monmap epoch 6 >>> --> Creating keyring file for osd.233 >>> Running command: /usr/bin/chown -R ceph:ceph >>> /var/lib/ceph/osd/ceph-233/keyring >>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233/ >>> Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore >>> bluestore --mkfs -i 233 --monmap /var/lib/ceph/osd/ceph-233/activate.monmap >>> --keyfile - --osd-data /var/lib/ceph/osd/ceph-233/ --osd-uuid >>> dbb58525-c4f6-4a1e-b03a-ac5966f09809 --setuser ceph --setgroup ceph >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >>> decode label at offset 102: void >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >>> decode label at offset 102: void >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >>> decode label at offset 102: void >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233/) _read_fsid unparsable uuid >>> --> ceph-volume lvm prepare successful for: /dev/mapper/mpatha >>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233 >>> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir >>> --dev >>> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> --path /var/lib/ceph/osd/ceph-233 --no-mon-config >>> Running command: /usr/bin/ln -snf >>> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> /var/lib/ceph/osd/ceph-233/block >>> Running command: /usr/bin/chown -h ceph:ceph >>> /var/lib/ceph/osd/ceph-233/block >>> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-30 >>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233 >>> Running command: /usr/bin/systemctl enable >>> ceph-volume@lvm-233-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> stderr: Created symlink >>> /etc/systemd/system/multi-user.target.wants/ceph-volume(a)lvm-233-dbb58525-c4f6-4a1e-b03a-ac5966f09809.service >>> → /usr/lib/systemd/system/ceph-volume@.service. >>> Running command: /usr/bin/systemctl enable --runtime ceph-osd@233 >>> stderr: Created symlink >>> /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)233.service → >>> /usr/lib/systemd/system/ceph-osd@.service. >>> Running command: /usr/bin/systemctl start ceph-osd@233 >>> --> ceph-volume lvm activate successful for osd ID: 233 >>> --> ceph-volume lvm create successful for: /dev/mapper/mpatha >>> --- >>> I get few errors in output like: >>> stderr: 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 auth: unable to find >>> a keyring on >>> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: >>> (2) No such file or directory >>> 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 AuthRegistry(0x7f9a380648f8) >>> no keyring found at >>> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, >>> disabling cephx >>> ... and then ... >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >>> decode label at offset 102: void >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >>> decode label at offset 102: void >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233/) _read_fsid unparsable uuid >>> ... but cluster seems to add and work on new OSDs anyway. >>> Anyone had a similar issue? Or can I ignore this error? >>> Kind regards, >>> Rok >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Adding new osds on new host
by Malte Stroem 28 Jan '26

28 Jan '26

Yes, don't do that. Use cephadm. On 1/28/26 14:26, Anthony D'Atri via ceph-users wrote: > The OP may be using a package-based install. > >> On Jan 28, 2026, at 3:14 AM, Malte Stroem via ceph-users <ceph-users(a)ceph.io> wrote: >> >> Use cephadm, "ceph orch" commands to do this. >> >>> On 1/27/26 20:51, Rok Jaklič via ceph-users wrote: >>> Hi, >>> i am adding new OSDs from a new host with something like: >>> <PROD> [root@server1 ~]# ceph-volume lvm create --data /dev/mapper/mpatha >>> Running command: /usr/bin/ceph-authtool --gen-print-key >>> Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd >>> --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new >>> dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> Running command: vgcreate --force --yes >>> ceph-11881314-f5e5-480f-9ac4-d25739e0dd74 /dev/mapper/mpatha >>> stdout: Physical volume "/dev/mapper/mpatha" successfully created. >>> stdout: Volume group "ceph-11881314-f5e5-480f-9ac4-d25739e0dd74" >>> successfully created >>> Running command: lvcreate --yes -l 3814911 -n >>> osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> ceph-11881314-f5e5-480f-9ac4-d25739e0dd74 >>> stdout: Logical volume "osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809" >>> created. >>> Running command: /usr/bin/ceph-authtool --gen-print-key >>> Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-233 >>> Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-233 >>> Running command: /usr/bin/chown -h ceph:ceph >>> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-30 >>> Running command: /usr/bin/ln -s >>> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> /var/lib/ceph/osd/ceph-233/block >>> Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd >>> --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o >>> /var/lib/ceph/osd/ceph-233/activate.monmap >>> stderr: 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 auth: unable to find >>> a keyring on >>> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: >>> (2) No such file or directory >>> 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 AuthRegistry(0x7f9a380648f8) >>> no keyring found at >>> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, >>> disabling cephx >>> stderr: got monmap epoch 6 >>> --> Creating keyring file for osd.233 >>> Running command: /usr/bin/chown -R ceph:ceph >>> /var/lib/ceph/osd/ceph-233/keyring >>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233/ >>> Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore >>> bluestore --mkfs -i 233 --monmap /var/lib/ceph/osd/ceph-233/activate.monmap >>> --keyfile - --osd-data /var/lib/ceph/osd/ceph-233/ --osd-uuid >>> dbb58525-c4f6-4a1e-b03a-ac5966f09809 --setuser ceph --setgroup ceph >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >>> decode label at offset 102: void >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >>> decode label at offset 102: void >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >>> decode label at offset 102: void >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233/) _read_fsid unparsable uuid >>> --> ceph-volume lvm prepare successful for: /dev/mapper/mpatha >>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233 >>> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir >>> --dev >>> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> --path /var/lib/ceph/osd/ceph-233 --no-mon-config >>> Running command: /usr/bin/ln -snf >>> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> /var/lib/ceph/osd/ceph-233/block >>> Running command: /usr/bin/chown -h ceph:ceph >>> /var/lib/ceph/osd/ceph-233/block >>> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-30 >>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233 >>> Running command: /usr/bin/systemctl enable >>> ceph-volume@lvm-233-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >>> stderr: Created symlink >>> /etc/systemd/system/multi-user.target.wants/ceph-volume(a)lvm-233-dbb58525-c4f6-4a1e-b03a-ac5966f09809.service >>> → /usr/lib/systemd/system/ceph-volume@.service. >>> Running command: /usr/bin/systemctl enable --runtime ceph-osd@233 >>> stderr: Created symlink >>> /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)233.service → >>> /usr/lib/systemd/system/ceph-osd@.service. >>> Running command: /usr/bin/systemctl start ceph-osd@233 >>> --> ceph-volume lvm activate successful for osd ID: 233 >>> --> ceph-volume lvm create successful for: /dev/mapper/mpatha >>> --- >>> I get few errors in output like: >>> stderr: 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 auth: unable to find >>> a keyring on >>> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: >>> (2) No such file or directory >>> 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 AuthRegistry(0x7f9a380648f8) >>> no keyring found at >>> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, >>> disabling cephx >>> ... and then ... >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >>> decode label at offset 102: void >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >>> decode label at offset 102: void >>> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >>> decode past end of struct encoding: Malformed input [buffer:3] >>> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >>> bluestore(/var/lib/ceph/osd/ceph-233/) _read_fsid unparsable uuid >>> ... but cluster seems to add and work on new OSDs anyway. >>> Anyone had a similar issue? Or can I ignore this error? >>> Kind regards, >>> Rok >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Adding new osds on new host
by Anthony D'Atri 28 Jan '26

28 Jan '26

The OP may be using a package-based install. > On Jan 28, 2026, at 3:14 AM, Malte Stroem via ceph-users <ceph-users(a)ceph.io> wrote: > > Use cephadm, "ceph orch" commands to do this. > >> On 1/27/26 20:51, Rok Jaklič via ceph-users wrote: >> Hi, >> i am adding new OSDs from a new host with something like: >> <PROD> [root@server1 ~]# ceph-volume lvm create --data /dev/mapper/mpatha >> Running command: /usr/bin/ceph-authtool --gen-print-key >> Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd >> --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new >> dbb58525-c4f6-4a1e-b03a-ac5966f09809 >> Running command: vgcreate --force --yes >> ceph-11881314-f5e5-480f-9ac4-d25739e0dd74 /dev/mapper/mpatha >> stdout: Physical volume "/dev/mapper/mpatha" successfully created. >> stdout: Volume group "ceph-11881314-f5e5-480f-9ac4-d25739e0dd74" >> successfully created >> Running command: lvcreate --yes -l 3814911 -n >> osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >> ceph-11881314-f5e5-480f-9ac4-d25739e0dd74 >> stdout: Logical volume "osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809" >> created. >> Running command: /usr/bin/ceph-authtool --gen-print-key >> Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-233 >> Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-233 >> Running command: /usr/bin/chown -h ceph:ceph >> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-30 >> Running command: /usr/bin/ln -s >> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >> /var/lib/ceph/osd/ceph-233/block >> Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd >> --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o >> /var/lib/ceph/osd/ceph-233/activate.monmap >> stderr: 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 auth: unable to find >> a keyring on >> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: >> (2) No such file or directory >> 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 AuthRegistry(0x7f9a380648f8) >> no keyring found at >> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, >> disabling cephx >> stderr: got monmap epoch 6 >> --> Creating keyring file for osd.233 >> Running command: /usr/bin/chown -R ceph:ceph >> /var/lib/ceph/osd/ceph-233/keyring >> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233/ >> Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore >> bluestore --mkfs -i 233 --monmap /var/lib/ceph/osd/ceph-233/activate.monmap >> --keyfile - --osd-data /var/lib/ceph/osd/ceph-233/ --osd-uuid >> dbb58525-c4f6-4a1e-b03a-ac5966f09809 --setuser ceph --setgroup ceph >> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >> decode label at offset 102: void >> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >> decode past end of struct encoding: Malformed input [buffer:3] >> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >> decode label at offset 102: void >> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >> decode past end of struct encoding: Malformed input [buffer:3] >> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >> decode label at offset 102: void >> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >> decode past end of struct encoding: Malformed input [buffer:3] >> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >> bluestore(/var/lib/ceph/osd/ceph-233/) _read_fsid unparsable uuid >> --> ceph-volume lvm prepare successful for: /dev/mapper/mpatha >> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233 >> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir >> --dev >> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >> --path /var/lib/ceph/osd/ceph-233 --no-mon-config >> Running command: /usr/bin/ln -snf >> /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >> /var/lib/ceph/osd/ceph-233/block >> Running command: /usr/bin/chown -h ceph:ceph >> /var/lib/ceph/osd/ceph-233/block >> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-30 >> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233 >> Running command: /usr/bin/systemctl enable >> ceph-volume@lvm-233-dbb58525-c4f6-4a1e-b03a-ac5966f09809 >> stderr: Created symlink >> /etc/systemd/system/multi-user.target.wants/ceph-volume(a)lvm-233-dbb58525-c4f6-4a1e-b03a-ac5966f09809.service >> → /usr/lib/systemd/system/ceph-volume@.service. >> Running command: /usr/bin/systemctl enable --runtime ceph-osd@233 >> stderr: Created symlink >> /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)233.service → >> /usr/lib/systemd/system/ceph-osd@.service. >> Running command: /usr/bin/systemctl start ceph-osd@233 >> --> ceph-volume lvm activate successful for osd ID: 233 >> --> ceph-volume lvm create successful for: /dev/mapper/mpatha >> --- >> I get few errors in output like: >> stderr: 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 auth: unable to find >> a keyring on >> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: >> (2) No such file or directory >> 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 AuthRegistry(0x7f9a380648f8) >> no keyring found at >> /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, >> disabling cephx >> ... and then ... >> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >> decode past end of struct encoding: Malformed input [buffer:3] >> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >> decode label at offset 102: void >> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >> decode past end of struct encoding: Malformed input [buffer:3] >> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >> bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to >> decode label at offset 102: void >> bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) >> decode past end of struct encoding: Malformed input [buffer:3] >> stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 >> bluestore(/var/lib/ceph/osd/ceph-233/) _read_fsid unparsable uuid >> ... but cluster seems to add and work on new OSDs anyway. >> Anyone had a similar issue? Or can I ignore this error? >> Kind regards, >> Rok >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: upmap-read balancer with CephFS kernel clients
by Janek Bevendorff 28 Jan '26

28 Jan '26

All right, I found a way by using rbd-nbd. However, for that I need to recreate all my Kubernetes StorageClasses, because the parameter fields are immutable. Not ideal. On 28/01/2026 09:17, Janek Bevendorff via ceph-users wrote: > I just noticed that this applies to RBD clients as well. How is this > supposed to work? Can you use only radosgw and ceph-fuse if you want > to use upmap-read? It seems strange to me that the most central Ceph > systems don't work with these new features. > > > On 26/01/2026 18:55, Janek Bevendorff via ceph-users wrote: >> Hi, >> >> Is there a way to use the new upmap-read balancer mode when I have >> CephFS kernel clients connected? Our cluster runs Squid and all >> connected clients are on kernel 6.11-6.14, so they should be new >> enough. However, they identify themselves as luminous (I could not >> find clear documentation about this, but this seems to be hard-coded >> and by design), so I cannot run ceph osd >> set-require-min-compat-client reef, which is required for the >> balancer module to work. >> >> I have 500 clients connected and migrating them all to FUSE would be >> a headache. Is there a way around that? Some of my pools have >> extremely high read balance scores and we're also struggling with >> performance issues as a result. >> >> Janek >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Bauhaus-Universität Weimar Bauhausstr. 9a, R308 99423 Weimar, Germany Phone: +49 3643 58 3577 www.webis.de

1 0

Re: upmap-read balancer with CephFS kernel clients
by Janek Bevendorff 28 Jan '26

28 Jan '26

I just noticed that this applies to RBD clients as well. How is this supposed to work? Can you use only radosgw and ceph-fuse if you want to use upmap-read? It seems strange to me that the most central Ceph systems don't work with these new features. On 26/01/2026 18:55, Janek Bevendorff via ceph-users wrote: > Hi, > > Is there a way to use the new upmap-read balancer mode when I have > CephFS kernel clients connected? Our cluster runs Squid and all > connected clients are on kernel 6.11-6.14, so they should be new > enough. However, they identify themselves as luminous (I could not > find clear documentation about this, but this seems to be hard-coded > and by design), so I cannot run ceph osd set-require-min-compat-client > reef, which is required for the balancer module to work. > > I have 500 clients connected and migrating them all to FUSE would be a > headache. Is there a way around that? Some of my pools have extremely > high read balance scores and we're also struggling with performance > issues as a result. > > Janek > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Bauhaus-Universität Weimar Bauhausstr. 9a, R308 99423 Weimar, Germany Phone: +49 3643 58 3577 www.webis.de

1 0

Re: Adding new osds on new host
by Malte Stroem 28 Jan '26

28 Jan '26

Use cephadm, "ceph orch" commands to do this. On 1/27/26 20:51, Rok Jaklič via ceph-users wrote: > Hi, > > i am adding new OSDs from a new host with something like: > <PROD> [root@server1 ~]# ceph-volume lvm create --data /dev/mapper/mpatha > Running command: /usr/bin/ceph-authtool --gen-print-key > Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd > --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new > dbb58525-c4f6-4a1e-b03a-ac5966f09809 > Running command: vgcreate --force --yes > ceph-11881314-f5e5-480f-9ac4-d25739e0dd74 /dev/mapper/mpatha > stdout: Physical volume "/dev/mapper/mpatha" successfully created. > stdout: Volume group "ceph-11881314-f5e5-480f-9ac4-d25739e0dd74" > successfully created > Running command: lvcreate --yes -l 3814911 -n > osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 > ceph-11881314-f5e5-480f-9ac4-d25739e0dd74 > stdout: Logical volume "osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809" > created. > Running command: /usr/bin/ceph-authtool --gen-print-key > Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-233 > Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-233 > Running command: /usr/bin/chown -h ceph:ceph > /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 > Running command: /usr/bin/chown -R ceph:ceph /dev/dm-30 > Running command: /usr/bin/ln -s > /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 > /var/lib/ceph/osd/ceph-233/block > Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd > --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o > /var/lib/ceph/osd/ceph-233/activate.monmap > stderr: 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 auth: unable to find > a keyring on > /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: > (2) No such file or directory > 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 AuthRegistry(0x7f9a380648f8) > no keyring found at > /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, > disabling cephx > stderr: got monmap epoch 6 > --> Creating keyring file for osd.233 > Running command: /usr/bin/chown -R ceph:ceph > /var/lib/ceph/osd/ceph-233/keyring > Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233/ > Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore > bluestore --mkfs -i 233 --monmap /var/lib/ceph/osd/ceph-233/activate.monmap > --keyfile - --osd-data /var/lib/ceph/osd/ceph-233/ --osd-uuid > dbb58525-c4f6-4a1e-b03a-ac5966f09809 --setuser ceph --setgroup ceph > stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 > bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to > decode label at offset 102: void > bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) > decode past end of struct encoding: Malformed input [buffer:3] > stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 > bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to > decode label at offset 102: void > bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) > decode past end of struct encoding: Malformed input [buffer:3] > stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 > bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to > decode label at offset 102: void > bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) > decode past end of struct encoding: Malformed input [buffer:3] > stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 > bluestore(/var/lib/ceph/osd/ceph-233/) _read_fsid unparsable uuid > --> ceph-volume lvm prepare successful for: /dev/mapper/mpatha > Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233 > Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir > --dev > /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 > --path /var/lib/ceph/osd/ceph-233 --no-mon-config > Running command: /usr/bin/ln -snf > /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 > /var/lib/ceph/osd/ceph-233/block > Running command: /usr/bin/chown -h ceph:ceph > /var/lib/ceph/osd/ceph-233/block > Running command: /usr/bin/chown -R ceph:ceph /dev/dm-30 > Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233 > Running command: /usr/bin/systemctl enable > ceph-volume@lvm-233-dbb58525-c4f6-4a1e-b03a-ac5966f09809 > stderr: Created symlink > /etc/systemd/system/multi-user.target.wants/ceph-volume(a)lvm-233-dbb58525-c4f6-4a1e-b03a-ac5966f09809.service > → /usr/lib/systemd/system/ceph-volume@.service. > Running command: /usr/bin/systemctl enable --runtime ceph-osd@233 > stderr: Created symlink > /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)233.service → > /usr/lib/systemd/system/ceph-osd@.service. > Running command: /usr/bin/systemctl start ceph-osd@233 > --> ceph-volume lvm activate successful for osd ID: 233 > --> ceph-volume lvm create successful for: /dev/mapper/mpatha > > --- > > I get few errors in output like: > stderr: 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 auth: unable to find > a keyring on > /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: > (2) No such file or directory > 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 AuthRegistry(0x7f9a380648f8) > no keyring found at > /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, > disabling cephx > > ... and then ... > > bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) > decode past end of struct encoding: Malformed input [buffer:3] > stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 > bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to > decode label at offset 102: void > bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) > decode past end of struct encoding: Malformed input [buffer:3] > stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 > bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to > decode label at offset 102: void > bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) > decode past end of struct encoding: Malformed input [buffer:3] > stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 > bluestore(/var/lib/ceph/osd/ceph-233/) _read_fsid unparsable uuid > > ... but cluster seems to add and work on new OSDs anyway. > > Anyone had a similar issue? Or can I ignore this error? > > Kind regards, > Rok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: OSDs won't start after upgrading reef (18.2.7) to squid (19.2.3) with orchestrator
by Eugen Block 27 Jan '26

27 Jan '26

Hi, did you by any chance restart the mgr (or simply fail)? The OSD status is not always correctly reported. From your log output you can see that OSD.200 actually booted successfully after around 3 minutes (as Malte also already pointed out): -- Boot 032bb082802d4003a96b9b2248f0e2eb -- Jan 27 10:08:07 host14 systemd[1]: Starting Ceph osd.200 for dca79fff-ffd0-58f4-1cff-82a2feea05f4... ... Jan 27 10:11:31 host14 ceph-osd[100997]: osd.200 1389134 load_pgs opened 135 pgs Jan 27 10:11:31 host14 ceph-osd[100997]: osd.200 1389134 log_to_monitors true Jan 27 10:11:31 host14 conmon[100991]: 2026-01-27T10:11:31.277+0000 7f541f5dd740 -1 osd.200 1389134 log_to_monitors true Jan 27 10:11:32 host14 ceph-osd[100997]: osd.200 1389134 done with init, starting boot process Jan 27 10:11:32 host14 ceph-osd[100997]: osd.200 1389134 start_boot ... Jan 27 10:11:33 host14 ceph-osd[100997]: osd.200 1389172 state: booting -> active Is there anything missing from the logs? Or is this not the right OSD log? Zitat von Boris via ceph-users <ceph-users(a)ceph.io>: > Sure: https://pastebin.com/9RLzyUQs > > I've trimmed the log a little bit (removed peering, epoch, trim and so on). > This is the last OSD that we tried that did not work. > > We tried another host, where the upgrade just went through. But this Host > also got the newest hardware. > But we don't think it is a hardware issue, because the first 30 OSDs were > on the two oldest hosts and the first one that failed was on the same host > as the last OSD that did not fail. > > > > Am Di., 27. Jan. 2026 um 12:05 Uhr schrieb Malte Stroem < > malte.stroem(a)gmail.com>: > >> Could be the kind of hardware you are using. Is it different from the >> other clusters' hardware? >> >> Send us logs, so we can help you out. >> >> Example: >> >> journalctl -eu ceph-fsid(a)osd.0 >> >> Best, >> Malte >> >> On 1/27/26 11:55, Boris via ceph-users wrote: >> > Hi, >> > we are currently facing an issue, that suddenly none of the OSDs will >> start >> > after the container started with the new versions. >> > >> > This seems to be an issue with some hosts/OSDs. The first 30 OSDs worked, >> > but took really long (like 5 hours) and then every single OSD after that >> > needed a host reboot to bring the disk back up and continue the update. >> > >> > We've stopped after 6 tries. >> > >> > And one disk never came back up. We removed and zapped the OSD. The >> > orchestrator picked the available disk and recreated it. It came up >> within >> > seconds. >> > >> > We have around 90 clusters and this happened only on a single one. All >> > others updates within two hours without any issues. >> > >> > The cluster uses HDDs (8TB) with the block.db on SSD (5 block.db per >> SSD). >> > The file /var/log/ceph/UUID/ceph-volume.log get hammered with a lot of >> > output from udevadm, lsblk and nsenter >> > The activation container (ceph-UUID-osd-N-activate) gets killed after a >> > couple of minutes. >> > It also looks like the block and block.db links >> > in /var/lib/ceph/UUID/osd.N/ are not correctly set. >> > When we restart the daemons that needed a host restart, the OSD doesn't >> > come up and needs a host restart. >> > >> > All OSDs are encrypted. >> > >> > Does anyone got some ideas how to debug further? >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im > groÃƒ¼en Saal. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Adding new osds on new host
by Rok Jaklič 27 Jan '26

27 Jan '26

Hi, i am adding new OSDs from a new host with something like: <PROD> [root@server1 ~]# ceph-volume lvm create --data /dev/mapper/mpatha Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new dbb58525-c4f6-4a1e-b03a-ac5966f09809 Running command: vgcreate --force --yes ceph-11881314-f5e5-480f-9ac4-d25739e0dd74 /dev/mapper/mpatha stdout: Physical volume "/dev/mapper/mpatha" successfully created. stdout: Volume group "ceph-11881314-f5e5-480f-9ac4-d25739e0dd74" successfully created Running command: lvcreate --yes -l 3814911 -n osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 ceph-11881314-f5e5-480f-9ac4-d25739e0dd74 stdout: Logical volume "osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809" created. Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-233 Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-233 Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-30 Running command: /usr/bin/ln -s /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 /var/lib/ceph/osd/ceph-233/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-233/activate.monmap stderr: 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 AuthRegistry(0x7f9a380648f8) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx stderr: got monmap epoch 6 --> Creating keyring file for osd.233 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233/keyring Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233/ Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 233 --monmap /var/lib/ceph/osd/ceph-233/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-233/ --osd-uuid dbb58525-c4f6-4a1e-b03a-ac5966f09809 --setuser ceph --setgroup ceph stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 bluestore(/var/lib/ceph/osd/ceph-233/) _read_fsid unparsable uuid --> ceph-volume lvm prepare successful for: /dev/mapper/mpatha Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 --path /var/lib/ceph/osd/ceph-233 --no-mon-config Running command: /usr/bin/ln -snf /dev/ceph-11881314-f5e5-480f-9ac4-d25739e0dd74/osd-block-dbb58525-c4f6-4a1e-b03a-ac5966f09809 /var/lib/ceph/osd/ceph-233/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-233/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-30 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-233 Running command: /usr/bin/systemctl enable ceph-volume@lvm-233-dbb58525-c4f6-4a1e-b03a-ac5966f09809 stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume(a)lvm-233-dbb58525-c4f6-4a1e-b03a-ac5966f09809.service → /usr/lib/systemd/system/ceph-volume@.service. Running command: /usr/bin/systemctl enable --runtime ceph-osd@233 stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)233.service → /usr/lib/systemd/system/ceph-osd@.service. Running command: /usr/bin/systemctl start ceph-osd@233 --> ceph-volume lvm activate successful for osd ID: 233 --> ceph-volume lvm create successful for: /dev/mapper/mpatha --- I get few errors in output like: stderr: 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2026-01-27T20:22:26.212+0100 7f9a406c6640 -1 AuthRegistry(0x7f9a380648f8) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx ... and then ... bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 bluestore(/var/lib/ceph/osd/ceph-233//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2026-01-27T20:22:26.421+0100 7fe148736780 -1 bluestore(/var/lib/ceph/osd/ceph-233/) _read_fsid unparsable uuid ... but cluster seems to add and work on new OSDs anyway. Anyone had a similar issue? Or can I ignore this error? Kind regards, Rok

1 0

Device prediction life unknown
by Info 27 Jan '26

27 Jan '26

Hello users When checking disk life every time getting unknown, it’s 16.2 with add’s. Ceph device predict-life-expectancy osd.84 unknown Any idea? Regards Dev

1 0

Re: OSDs won't start after upgrading reef (18.2.7) to squid (19.2.3) with orchestrator
by Malte Stroem 27 Jan '26

27 Jan '26

We do not know a lot about you cluster, so it's hard to help. Give us alle the information one needs. ceph -s, ceph orch host ls, ceph health detail, all the good things to get an overview. On 1/27/26 12:43, Boris via ceph-users wrote: > All disks are 8TB HDD. We have some 16TB HDDs but those are all in the > newest host, that updated just fine. > > Am Di., 27. Jan. 2026 um 12:39 Uhr schrieb Malte Stroem < > malte.stroem(a)gmail.com>: > >> From what I can see, the OSD is running. It's starting up and still >> needs time. >> >> How big is the disk? >> >> On 1/27/26 12:31, Boris via ceph-users wrote: >>> Sure: https://pastebin.com/9RLzyUQs >>> >>> I've trimmed the log a little bit (removed peering, epoch, trim and so >> on). >>> This is the last OSD that we tried that did not work. >>> >>> We tried another host, where the upgrade just went through. But this Host >>> also got the newest hardware. >>> But we don't think it is a hardware issue, because the first 30 OSDs were >>> on the two oldest hosts and the first one that failed was on the same >> host >>> as the last OSD that did not fail. >>> >>> >>> >>> Am Di., 27. Jan. 2026 um 12:05 Uhr schrieb Malte Stroem < >>> malte.stroem(a)gmail.com>: >>> >>>> Could be the kind of hardware you are using. Is it different from the >>>> other clusters' hardware? >>>> >>>> Send us logs, so we can help you out. >>>> >>>> Example: >>>> >>>> journalctl -eu ceph-fsid(a)osd.0 >>>> >>>> Best, >>>> Malte >>>> >>>> On 1/27/26 11:55, Boris via ceph-users wrote: >>>>> Hi, >>>>> we are currently facing an issue, that suddenly none of the OSDs will >>>> start >>>>> after the container started with the new versions. >>>>> >>>>> This seems to be an issue with some hosts/OSDs. The first 30 OSDs >> worked, >>>>> but took really long (like 5 hours) and then every single OSD after >> that >>>>> needed a host reboot to bring the disk back up and continue the update. >>>>> >>>>> We've stopped after 6 tries. >>>>> >>>>> And one disk never came back up. We removed and zapped the OSD. The >>>>> orchestrator picked the available disk and recreated it. It came up >>>> within >>>>> seconds. >>>>> >>>>> We have around 90 clusters and this happened only on a single one. All >>>>> others updates within two hours without any issues. >>>>> >>>>> The cluster uses HDDs (8TB) with the block.db on SSD (5 block.db per >>>> SSD). >>>>> The file /var/log/ceph/UUID/ceph-volume.log get hammered with a lot of >>>>> output from udevadm, lsblk and nsenter >>>>> The activation container (ceph-UUID-osd-N-activate) gets killed after a >>>>> couple of minutes. >>>>> It also looks like the block and block.db links >>>>> in /var/lib/ceph/UUID/osd.N/ are not correctly set. >>>>> When we restart the daemons that needed a host restart, the OSD doesn't >>>>> come up and needs a host restart. >>>>> >>>>> All OSDs are encrypted. >>>>> >>>>> Does anyone got some ideas how to debug further? >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>>> >>> >> >> >

2 1

Re: OSDs won't start after upgrading reef (18.2.7) to squid (19.2.3) with orchestrator
by Malte Stroem 27 Jan '26

27 Jan '26

From what I can see, the OSD is running. It's starting up and still needs time. How big is the disk? On 1/27/26 12:31, Boris via ceph-users wrote: > Sure: https://pastebin.com/9RLzyUQs > > I've trimmed the log a little bit (removed peering, epoch, trim and so on). > This is the last OSD that we tried that did not work. > > We tried another host, where the upgrade just went through. But this Host > also got the newest hardware. > But we don't think it is a hardware issue, because the first 30 OSDs were > on the two oldest hosts and the first one that failed was on the same host > as the last OSD that did not fail. > > > > Am Di., 27. Jan. 2026 um 12:05 Uhr schrieb Malte Stroem < > malte.stroem(a)gmail.com>: > >> Could be the kind of hardware you are using. Is it different from the >> other clusters' hardware? >> >> Send us logs, so we can help you out. >> >> Example: >> >> journalctl -eu ceph-fsid(a)osd.0 >> >> Best, >> Malte >> >> On 1/27/26 11:55, Boris via ceph-users wrote: >>> Hi, >>> we are currently facing an issue, that suddenly none of the OSDs will >> start >>> after the container started with the new versions. >>> >>> This seems to be an issue with some hosts/OSDs. The first 30 OSDs worked, >>> but took really long (like 5 hours) and then every single OSD after that >>> needed a host reboot to bring the disk back up and continue the update. >>> >>> We've stopped after 6 tries. >>> >>> And one disk never came back up. We removed and zapped the OSD. The >>> orchestrator picked the available disk and recreated it. It came up >> within >>> seconds. >>> >>> We have around 90 clusters and this happened only on a single one. All >>> others updates within two hours without any issues. >>> >>> The cluster uses HDDs (8TB) with the block.db on SSD (5 block.db per >> SSD). >>> The file /var/log/ceph/UUID/ceph-volume.log get hammered with a lot of >>> output from udevadm, lsblk and nsenter >>> The activation container (ceph-UUID-osd-N-activate) gets killed after a >>> couple of minutes. >>> It also looks like the block and block.db links >>> in /var/lib/ceph/UUID/osd.N/ are not correctly set. >>> When we restart the daemons that needed a host restart, the OSD doesn't >>> come up and needs a host restart. >>> >>> All OSDs are encrypted. >>> >>> Does anyone got some ideas how to debug further? >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >

2 1

Re: OSDs won't start after upgrading reef (18.2.7) to squid (19.2.3) with orchestrator
by Malte Stroem 27 Jan '26

27 Jan '26

Could be the kind of hardware you are using. Is it different from the other clusters' hardware? Send us logs, so we can help you out. Example: journalctl -eu ceph-fsid(a)osd.0 Best, Malte On 1/27/26 11:55, Boris via ceph-users wrote: > Hi, > we are currently facing an issue, that suddenly none of the OSDs will start > after the container started with the new versions. > > This seems to be an issue with some hosts/OSDs. The first 30 OSDs worked, > but took really long (like 5 hours) and then every single OSD after that > needed a host reboot to bring the disk back up and continue the update. > > We've stopped after 6 tries. > > And one disk never came back up. We removed and zapped the OSD. The > orchestrator picked the available disk and recreated it. It came up within > seconds. > > We have around 90 clusters and this happened only on a single one. All > others updates within two hours without any issues. > > The cluster uses HDDs (8TB) with the block.db on SSD (5 block.db per SSD). > The file /var/log/ceph/UUID/ceph-volume.log get hammered with a lot of > output from udevadm, lsblk and nsenter > The activation container (ceph-UUID-osd-N-activate) gets killed after a > couple of minutes. > It also looks like the block and block.db links > in /var/lib/ceph/UUID/osd.N/ are not correctly set. > When we restart the daemons that needed a host restart, the OSD doesn't > come up and needs a host restart. > > All OSDs are encrypted. > > Does anyone got some ideas how to debug further? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

OSDs won't start after upgrading reef (18.2.7) to squid (19.2.3) with orchestrator
by Boris 27 Jan '26

27 Jan '26

Hi, we are currently facing an issue, that suddenly none of the OSDs will start after the container started with the new versions. This seems to be an issue with some hosts/OSDs. The first 30 OSDs worked, but took really long (like 5 hours) and then every single OSD after that needed a host reboot to bring the disk back up and continue the update. We've stopped after 6 tries. And one disk never came back up. We removed and zapped the OSD. The orchestrator picked the available disk and recreated it. It came up within seconds. We have around 90 clusters and this happened only on a single one. All others updates within two hours without any issues. The cluster uses HDDs (8TB) with the block.db on SSD (5 block.db per SSD). The file /var/log/ceph/UUID/ceph-volume.log get hammered with a lot of output from udevadm, lsblk and nsenter The activation container (ceph-UUID-osd-N-activate) gets killed after a couple of minutes. It also looks like the block and block.db links in /var/lib/ceph/UUID/osd.N/ are not correctly set. When we restart the daemons that needed a host restart, the OSD doesn't come up and needs a host restart. All OSDs are encrypted. Does anyone got some ideas how to debug further?

1 0

upmap-read balancer with CephFS kernel clients
by Janek Bevendorff 26 Jan '26

26 Jan '26

Hi, Is there a way to use the new upmap-read balancer mode when I have CephFS kernel clients connected? Our cluster runs Squid and all connected clients are on kernel 6.11-6.14, so they should be new enough. However, they identify themselves as luminous (I could not find clear documentation about this, but this seems to be hard-coded and by design), so I cannot run ceph osd set-require-min-compat-client reef, which is required for the balancer module to work. I have 500 clients connected and migrating them all to FUSE would be a headache. Is there a way around that? Some of my pools have extremely high read balance scores and we're also struggling with performance issues as a result. Janek

1 0

January Ceph Science User Group Virtual Meeting
by Kevin Hrpcek 23 Jan '26

23 Jan '26

Hey folks, We will be having a Ceph science/research/big cluster call on Tuesday January 27th. This is an informal open call of community members mostly from hpc/htc/research/big cluster environments, though anyone is welcome, where we discuss whatever is on our minds regarding Ceph. Updates, outages, features, maintenance, etc...there is no single set presenter for the length of the meeting, but it is somewhat moderated to keep the conversation flowing. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments, you are most welcome to contact me. We usually start each meeting with a brief 10-15 minutes talk given by a community member, but currently don't have one scheduled for this meeting.If you are interested in signing up to give a short talk for a meeting please let me know! It is a very informal talk. Virtual event details: January 27th, 2026 9am Central US 15:00 UTC 4pm Central European Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_20260127 Pad Index: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone https://meet.jit.si/ceph-science-wg Ceph Community Calendar -- https://ceph.io/en/community/meetups/ See you there! Kevin

1 0

Re: rgw - migrate data from replicated to ec pool
by Anthony D'Atri 22 Jan '26

22 Jan '26

Thanks, Casey, I was very excited to learn of this at Cephalocon. At the time it seemed like a proposal, hearing that work has begun is fantastic. This should AFAICT allow retrofitting a first SSD data pool into an existing CephFS deployment, though I’ll be curious if the pool indirection overhead erodes the benefits of doing so. > On Jan 22, 2026, at 9:04 AM, Casey Bodley via ceph-users <ceph-users(a)ceph.io> wrote: > > storage classes and/or lifecycle are not ideal for this because, while > they can add a new pool, they don't provide a way to retire the old > one. removing/replacing the default STANDARD storage class is > especially difficult because, as Janne suggested, the STANDARD storage > class pool stores all 'head objects' which act like symlinks for > object data in other storage class pools. lifecycle transitions will > only move object data around, not their head objects > > luckily, the rados team has started work on a "pool migration" > feature. sharing links to the design doc and early draft pr: > https://docs.ceph.com/en/latest/dev/pool-migration-design/ > https://github.com/ceph/ceph/pull/65746 > > On Thu, Jan 22, 2026 at 4:42 AM Konstantin Shalygin via ceph-users > <ceph-users(a)ceph.io> wrote: >> >> >>> On 22 Jan 2026, at 11:38, Janne Johansson via ceph-users <ceph-users(a)ceph.io> wrote: >>> >>> Please correct me if I am wrong here (because I might totally be), but >>> doesn't this leave some kind of "softlink" object in the old pool and >>> "only" move the content over to the new pool? >> >> Depends on client (s5cmd/aws cli program), mostly. What exactly this command actually do - copy data over client or not >> >> Also, your clients may use LifeCycle Policy to move objects to another class. In your case this should work, cause since implementation Ceph can't move objects between classes in shared pool, see #50974 [1] >> In this ticket you can grab copy-paste XML for what I'm talking about >> >> >> >> [1] https://tracker.ceph.com/issues/50974 >> k >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: rgw - migrate data from replicated to ec pool
by Anthony D'Atri 22 Jan '26

22 Jan '26

> ceph-users <ceph-users(a)ceph.io>: >>> On 21 Jan 2026, at 16:58, Boris via ceph-users <ceph-users(a)ceph.io> wrote: >>> >>> we want to migrate the data from a replicated to an EC pool. >>> Is there a way to do it without downtime? >> >> Your clients can move objects from one storage class to another, example >> s5cmd mv s3://bucket s3://bucket2 --storage-class="GLACIER" > > Please correct me if I am wrong here (because I might totally be), but > doesn't this leave some kind of "softlink" object in the old pool and > "only" move the content over to the new pool? Not to my knowledge. You can however accomplish the same thing with an LC policy, which would likely be way more efficient. I must note, though, that storage class names are arbitrary to RGW, but not to some clients / libraries, which expect specific server behavior from GLACIER, or won’t work at all for an SC named, say, ETHELMERMAN. It’s best to stick to AWS names as documented here: https://docs.ceph.com/en/latest/radosgw/placement/#using-storage-classes Today, you can use rbd-mirror or live migration to move volumes between pools, but at some point the client attachment I think needs to be refreshed. With CephFS you attach the new data pool, setfattr a directory for the file layout to use the pool, and cp (not mv) each file onto it. There are scripts available online. Since this is a client operation that copies all the data then deletes the old, it is resource-intensive and not transparent to clients, at least not until such a time that the directory names are swapped, and not for any open files. As with RGW head objects, which are always in the default storage class, CephFS benefits from the first data pool being on fast SSDs for backtraces, especially when there are a zillion tiny files. This cannot AFAIK be retrofitted (at least not before Umbrella), but one could create a new CephFS with new pools and migrate files onto it. > there was some kind of situation where you still need to retain the old pool. If transparent RADOS pool migration makes it into Umbrella, that will be true. But likely one will be able to pare down pg_num for that pool and it won’t be a big deal. > Perhaps this is related to layering ceph pools > (also one way in the past to move data from pool A to B while changing > EC values or repl, not recommended anymore) which is why I am so > unsure. If anyone wants to test/verify I am happy to be proven wrong. > > If I am wrong, then https://docs.ceph.com/en/latest/radosgw/placement/ > has some good info on how to make more than one ceph pool available > for rgw usage and set another one as the new default for users/buckets > created after the change and so forth. Thanks! > > -- > May the most significant bit of your life be positive. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: rgw - migrate data from replicated to ec pool
by Boris 22 Jan '26

22 Jan '26

Hi Casey, that is exactly what I was looking for. I had all the ideas that were mentioned here, but I wanted it to be transparent for the customer without them noticing. With the pool migration feature, this would be really great. Thanks all for your input :) Am Do., 22. Jan. 2026 um 15:06 Uhr schrieb Casey Bodley via ceph-users < ceph-users(a)ceph.io>: > storage classes and/or lifecycle are not ideal for this because, while > they can add a new pool, they don't provide a way to retire the old > one. removing/replacing the default STANDARD storage class is > especially difficult because, as Janne suggested, the STANDARD storage > class pool stores all 'head objects' which act like symlinks for > object data in other storage class pools. lifecycle transitions will > only move object data around, not their head objects > > luckily, the rados team has started work on a "pool migration" > feature. sharing links to the design doc and early draft pr: > https://docs.ceph.com/en/latest/dev/pool-migration-design/ > https://github.com/ceph/ceph/pull/65746 > > On Thu, Jan 22, 2026 at 4:42 AM Konstantin Shalygin via ceph-users > <ceph-users(a)ceph.io> wrote: > > > > > > > On 22 Jan 2026, at 11:38, Janne Johansson via ceph-users < > ceph-users(a)ceph.io> wrote: > > > > > > Please correct me if I am wrong here (because I might totally be), but > > > doesn't this leave some kind of "softlink" object in the old pool and > > > "only" move the content over to the new pool? > > > > Depends on client (s5cmd/aws cli program), mostly. What exactly this > command actually do - copy data over client or not > > > > Also, your clients may use LifeCycle Policy to move objects to another > class. In your case this should work, cause since implementation Ceph can't > move objects between classes in shared pool, see #50974 [1] > > In this ticket you can grab copy-paste XML for what I'm talking about > > > > > > > > [1] https://tracker.ceph.com/issues/50974 > > k > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

Re: rgw - migrate data from replicated to ec pool
by Casey Bodley 22 Jan '26

22 Jan '26

storage classes and/or lifecycle are not ideal for this because, while they can add a new pool, they don't provide a way to retire the old one. removing/replacing the default STANDARD storage class is especially difficult because, as Janne suggested, the STANDARD storage class pool stores all 'head objects' which act like symlinks for object data in other storage class pools. lifecycle transitions will only move object data around, not their head objects luckily, the rados team has started work on a "pool migration" feature. sharing links to the design doc and early draft pr: https://docs.ceph.com/en/latest/dev/pool-migration-design/ https://github.com/ceph/ceph/pull/65746 On Thu, Jan 22, 2026 at 4:42 AM Konstantin Shalygin via ceph-users <ceph-users(a)ceph.io> wrote: > > > > On 22 Jan 2026, at 11:38, Janne Johansson via ceph-users <ceph-users(a)ceph.io> wrote: > > > > Please correct me if I am wrong here (because I might totally be), but > > doesn't this leave some kind of "softlink" object in the old pool and > > "only" move the content over to the new pool? > > Depends on client (s5cmd/aws cli program), mostly. What exactly this command actually do - copy data over client or not > > Also, your clients may use LifeCycle Policy to move objects to another class. In your case this should work, cause since implementation Ceph can't move objects between classes in shared pool, see #50974 [1] > In this ticket you can grab copy-paste XML for what I'm talking about > > > > [1] https://tracker.ceph.com/issues/50974 > k > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: rgw - migrate data from replicated to ec pool
by Konstantin Shalygin 22 Jan '26

22 Jan '26

> On 22 Jan 2026, at 11:38, Janne Johansson via ceph-users <ceph-users(a)ceph.io> wrote: > > Please correct me if I am wrong here (because I might totally be), but > doesn't this leave some kind of "softlink" object in the old pool and > "only" move the content over to the new pool? Depends on client (s5cmd/aws cli program), mostly. What exactly this command actually do - copy data over client or not Also, your clients may use LifeCycle Policy to move objects to another class. In your case this should work, cause since implementation Ceph can't move objects between classes in shared pool, see #50974 [1] In this ticket you can grab copy-paste XML for what I'm talking about [1] https://tracker.ceph.com/issues/50974 k

1 0

Re: rgw - migrate data from replicated to ec pool
by Janne Johansson 22 Jan '26

22 Jan '26

Den tors 22 jan. 2026 kl 09:26 skrev Konstantin Shalygin via ceph-users <ceph-users(a)ceph.io>: > > On 21 Jan 2026, at 16:58, Boris via ceph-users <ceph-users(a)ceph.io> wrote: > > > > we want to migrate the data from a replicated to an EC pool. > > Is there a way to do it without downtime? > > Your clients can move objects from one storage class to another, example > s5cmd mv s3://bucket s3://bucket2 --storage-class="GLACIER" Please correct me if I am wrong here (because I might totally be), but doesn't this leave some kind of "softlink" object in the old pool and "only" move the content over to the new pool? I seem to recall something like this when you move objects between storage pools/placements in ceph s3, and it could totally be about something else so don't take my word for it, but there is a nagging feeling in my mind there was some kind of situation where you still need to retain the old pool. Perhaps this is related to layering ceph pools (also one way in the past to move data from pool A to B while changing EC values or repl, not recommended anymore) which is why I am so unsure. If anyone wants to test/verify I am happy to be proven wrong. If I am wrong, then https://docs.ceph.com/en/latest/radosgw/placement/ has some good info on how to make more than one ceph pool available for rgw usage and set another one as the new default for users/buckets created after the change and so forth. -- May the most significant bit of your life be positive.

1 0

Re: rgw - migrate data from replicated to ec pool
by Konstantin Shalygin 22 Jan '26

22 Jan '26

Hi, > On 21 Jan 2026, at 16:58, Boris via ceph-users <ceph-users(a)ceph.io> wrote: > > we want to migrate the data from a replicated to an EC pool. > Is there a way to do it without downtime? Your clients can move objects from one storage class to another, example s5cmd mv s3://bucket s3://bucket2 --storage-class="GLACIER" k

1 0

Re: Fwd: On your post: "[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag"
by Alfonso Escribano Merino 22 Jan '26

22 Jan '26

1 0

Re: Fwd: On your post: "[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag"
by Joshua Blanch 21 Jan '26

21 Jan '26

We ran into that allow_ec_optimization on a legacy pool and scrubbing issue recently on 20.2.0. Wrote down some notes here: https://docs.clyso.com/blog/2026/04/13/tentacle-v20.2.0-ec-bug/ Specifically it is these trackers which will be in the v20.2.1 release allow_ec issue: https://tracker.ceph.com/issues/71642 scrubbing issue: https://tracker.ceph.com/issues/73184 https://tracker.ceph.com/issues/73260 On Wed, 21 Jan 2026 at 11:17, Eric Dold via ceph-users <ceph-users(a)ceph.io> wrote: > Hi Reto, hi Bill > > I found this PR that was merged after v20.2.0 was tagged. > https://github.com/ceph/ceph/pull/66021 > > I'm running Tentacle on Ubuntu Jammy without cephadm, with EC 6+2 and > FastEC enabled. > > I'm getting inconsistent PGs with errors like these: > 3.336s0 shard 25(5) soid 3:6cc82c0a:::10002107475.0000002c:head : size > 700416 != size 696320 from auth oi > 3:6cc82c0a:::10002107475.0000002c:head(2830345'2540087 > osd.41.2825431:3886213 dirty|data_digest s 4194304 uv 17841689 dd 8f22c13e > alloc_hint [0 0 0]) > 3.336s0 shard 13(4) soid 3:6cc1c47e:::10002585665.0000045e:10279 : size > 700416 != size 696320 from auth oi > 3:6cc1c47e:::10002585665.0000045e:10279(2905787'3123318 > osd.41.2896446:3729885 dirty|data_digest s 4194304 uv 23445132 dd 365ee4af > alloc_hint [0 0 0] shard_versions={1: 2904402'3115959, 2: 2904402'3115959, > 3: 2904402'3115959, 4: 2904402'3115959, 5: 2904402'3115959}) > 3.336s0 shard 13(4) soid 3:6cc1f637:::20000007ea9.000008fb:bd : size 700416 > != size 696320 from auth oi > 3:6cc1f637:::20000007ea9.000008fb:bd(45314'680679 client.124476.0:17244371 > dirty|data_digest s 4194304 uv 274244 dd 9f754009 alloc_hint [0 0 0]) > 3.336 shard 13(4) soid 3:6cc82c71:::1000098dda0.00000237:head : candidate > size 700416 info size 696320 mismatch > 3.336 shard 60(1) soid 3:6cde2c17:::20001b3c880.00000000:head : candidate > size 4096 info size 0 mismatch > 3.336 shard 13(4) soid 3:6cde2eb0:::100000d68b9.0000000b:head : candidate > size 622592 info size 618496 mismatch > > All these errors have something to do with size. > > When I restart the OSDs the cluster is healthy until some scrub picks this > up again. > The data seems ok. > > It seems like the fix is not in the tentacle release. > Is there a future release with fixes for this planned? > > Regards > Eric > > > On Fri, 5 Dec 2025 at 16:34, Reto Gysi via ceph-users <ceph-users(a)ceph.io> > wrote: > > > Hi Bill > > > > Am Fr., 5. Dez. 2025 um 10:02 Uhr schrieb Bill Scales < > > bill_scales(a)uk.ibm.com>: > > > > > Hi Reto, > > > > > > Sorry to hear about your problems with turning on ec optimizations. > I've > > > led the team that developed this feature, so we are keen to help > > understand > > > what happened to your system. > > > > > > Your configuration looks fine as far as being supported with ec > > > optimizations. The daemons (mons, osds and mgr) need to be running > > tentacle > > > code to use this feature, there is no requirement to update any of your > > > clients. > > > > > > Do you have any more examples of the inconsistent objects logs that you > > > can share with me? > > > > > > There was a bug that we fixed late in the development cycle which was > > > scrubbing incorrectly reporting a size mismatch for objects written > prior > > > to turning on optimizations. This is because prior to turning on > > > optimizations objects are padded to a multiple of the stripe_width, > > > afterwards objects are no longer padded. The scrub code was > occasionally > > > getting confused and incorrectly reporting an inconsistency. In this > case > > > the scrub was reporting false positive errors - there was nothing wrong > > > with the data and no problems accessing the data. I notice the log you > > > shared in the email was for a size mismatch, I'm interested whether all > > the > > > logs were for mismatched sizes. We will do some further work to confirm > > > that the fix is in the 20.2.0 tentacle release and that there are no > > > problems with the back port. > > > > > > > As far I could see the errors where all about the mismatched sizes. > > > > > > > > > > You also mention that you saw some OSD crashes. Do you have any further > > > information about these? > > > > > > > I just could cause another crash with starting up a windows VM, and > doing a > > WIndows file history backup to a drive that is on an RBD Image on pool > > rbd_ecpool where I had the allow_ec_optimization flag enabled. > > > > <disk type="network" device="disk"> > > <driver name="qemu" type="raw" cache="writethrough" io="threads" > > discard="unmap" detect_zeroes="unmap"/> > > <auth username="admin"> > > <secret type="ceph" uuid="878b0bc5-c471-4ec6-a92a-f65282ffbdf6"/> > > </auth> > > <source protocol="rbd" name="rbd/game_windows_backup_drive" index="3"> > > <host name="zephir" port="6789"/> > > </source> > > <target dev="vdd" bus="virtio"/> > > <alias name="virtio-disk3"/> > > <address type="pci" domain="0x0000" bus="0x00" slot="0x02" > > function="0x0"/> > > </disk> > > > > root@zephir:~# ceph -s > > cluster: > > id: 27923302-87a5-11ec-ac5b-976d21a49941 > > health: HEALTH_WARN > > 2 osds down > > Reduced data availability: 49 pgs inactive > > Degraded data redundancy: 15722192/79848340 objects degraded > > (19.690%), 246 pgs degraded, 247 pgs undersized > > > > services: > > mon: 3 daemons, quorum zephir,debian,raspi (age 20h) > [leader: > > zephir] > > mgr: zephir.enywvy(active, since 21h), standbys: > debian.nusuye > > mds: 3/3 daemons up, 3 standby > > osd: 18 osds: 16 up (since 5m), 18 in (since 3d); 8 remapped > > pgs > > cephfs-mirror: 2 daemons active (2 hosts) > > rbd-mirror: 2 daemons active (2 hosts) > > rgw: 1 daemon active (1 hosts, 1 zones) > > tcmu-runner: 5 portals active (2 hosts) > > > > data: > > volumes: 3/3 healthy > > pools: 25 pools, 450 pgs > > objects: 17.52M objects, 62 TiB > > usage: 116 TiB used, 63 TiB / 179 TiB avail > > pgs: 10.889% pgs not active > > 15722192/79848340 objects degraded (19.690%) > > 197 active+undersized+degraded > > 183 active+clean > > 49 undersized+degraded+peered > > 15 active+clean+scrubbing > > 5 active+clean+scrubbing+deep > > 1 active+undersized > > > > io: > > client: 1.7 KiB/s rd, 1023 B/s wr, 1 op/s rd, 0 op/s wr > > > > root@zephir:~# ceph osd status > > ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE > > 0 debian 11.2T 5289G 0 0 0 0 exists,up > > 1 zephir 11.6T 4846G 0 0 0 0 exists > > 2 zephir 10.7T 5827G 0 0 2 4 exists,up > > 3 zephir 11.0T 5497G 0 0 0 0 exists > > 4 zephir 159G 1376G 0 0 2 0 exists,up > > 5 zephir 120G 1415G 0 0 0 0 exists,up > > 6 debian 11.3T 5200G 0 0 1 0 exists,up > > 7 zephir 305G 1230G 0 0 0 0 exists,up > > 8 zephir 10.9T 5594G 0 0 0 0 exists,up > > 9 zephir 11.9T 4616G 0 0 0 0 exists,up > > 10 zephir 235G 1300G 0 0 0 0 exists,up > > 11 zephir 223G 1312G 0 0 0 0 exists,up > > 12 zephir 11.0T 5479G 0 0 0 0 exists,up > > 13 debian 118G 631G 0 0 3 8 exists,up > > 14 debian 345G 1190G 0 0 0 0 exists,up > > 15 debian 11.4T 5079G 0 0 0 0 exists,up > > 16 debian 554G 3029G 0 0 3 8 exists,up > > 17 debian 12.5T 5874G 0 0 1 0 exists,up > > root@zephir:~# > > > > I attached the log of osd.1 to > > https://filebin.net/jwvs6kuqrc7hx8id > > > > Best Regards, > > > > Reto > > > > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Fwd: On your post: "[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag"
by Eric Dold 21 Jan '26

21 Jan '26

Hi Reto, hi Bill I found this PR that was merged after v20.2.0 was tagged. https://github.com/ceph/ceph/pull/66021 I'm running Tentacle on Ubuntu Jammy without cephadm, with EC 6+2 and FastEC enabled. I'm getting inconsistent PGs with errors like these: 3.336s0 shard 25(5) soid 3:6cc82c0a:::10002107475.0000002c:head : size 700416 != size 696320 from auth oi 3:6cc82c0a:::10002107475.0000002c:head(2830345'2540087 osd.41.2825431:3886213 dirty|data_digest s 4194304 uv 17841689 dd 8f22c13e alloc_hint [0 0 0]) 3.336s0 shard 13(4) soid 3:6cc1c47e:::10002585665.0000045e:10279 : size 700416 != size 696320 from auth oi 3:6cc1c47e:::10002585665.0000045e:10279(2905787'3123318 osd.41.2896446:3729885 dirty|data_digest s 4194304 uv 23445132 dd 365ee4af alloc_hint [0 0 0] shard_versions={1: 2904402'3115959, 2: 2904402'3115959, 3: 2904402'3115959, 4: 2904402'3115959, 5: 2904402'3115959}) 3.336s0 shard 13(4) soid 3:6cc1f637:::20000007ea9.000008fb:bd : size 700416 != size 696320 from auth oi 3:6cc1f637:::20000007ea9.000008fb:bd(45314'680679 client.124476.0:17244371 dirty|data_digest s 4194304 uv 274244 dd 9f754009 alloc_hint [0 0 0]) 3.336 shard 13(4) soid 3:6cc82c71:::1000098dda0.00000237:head : candidate size 700416 info size 696320 mismatch 3.336 shard 60(1) soid 3:6cde2c17:::20001b3c880.00000000:head : candidate size 4096 info size 0 mismatch 3.336 shard 13(4) soid 3:6cde2eb0:::100000d68b9.0000000b:head : candidate size 622592 info size 618496 mismatch All these errors have something to do with size. When I restart the OSDs the cluster is healthy until some scrub picks this up again. The data seems ok. It seems like the fix is not in the tentacle release. Is there a future release with fixes for this planned? Regards Eric On Fri, 5 Dec 2025 at 16:34, Reto Gysi via ceph-users <ceph-users(a)ceph.io> wrote: > Hi Bill > > Am Fr., 5. Dez. 2025 um 10:02 Uhr schrieb Bill Scales < > bill_scales(a)uk.ibm.com>: > > > Hi Reto, > > > > Sorry to hear about your problems with turning on ec optimizations. I've > > led the team that developed this feature, so we are keen to help > understand > > what happened to your system. > > > > Your configuration looks fine as far as being supported with ec > > optimizations. The daemons (mons, osds and mgr) need to be running > tentacle > > code to use this feature, there is no requirement to update any of your > > clients. > > > > Do you have any more examples of the inconsistent objects logs that you > > can share with me? > > > > There was a bug that we fixed late in the development cycle which was > > scrubbing incorrectly reporting a size mismatch for objects written prior > > to turning on optimizations. This is because prior to turning on > > optimizations objects are padded to a multiple of the stripe_width, > > afterwards objects are no longer padded. The scrub code was occasionally > > getting confused and incorrectly reporting an inconsistency. In this case > > the scrub was reporting false positive errors - there was nothing wrong > > with the data and no problems accessing the data. I notice the log you > > shared in the email was for a size mismatch, I'm interested whether all > the > > logs were for mismatched sizes. We will do some further work to confirm > > that the fix is in the 20.2.0 tentacle release and that there are no > > problems with the back port. > > > > As far I could see the errors where all about the mismatched sizes. > > > > > > You also mention that you saw some OSD crashes. Do you have any further > > information about these? > > > > I just could cause another crash with starting up a windows VM, and doing a > WIndows file history backup to a drive that is on an RBD Image on pool > rbd_ecpool where I had the allow_ec_optimization flag enabled. > > <disk type="network" device="disk"> > <driver name="qemu" type="raw" cache="writethrough" io="threads" > discard="unmap" detect_zeroes="unmap"/> > <auth username="admin"> > <secret type="ceph" uuid="878b0bc5-c471-4ec6-a92a-f65282ffbdf6"/> > </auth> > <source protocol="rbd" name="rbd/game_windows_backup_drive" index="3"> > <host name="zephir" port="6789"/> > </source> > <target dev="vdd" bus="virtio"/> > <alias name="virtio-disk3"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x02" > function="0x0"/> > </disk> > > root@zephir:~# ceph -s > cluster: > id: 27923302-87a5-11ec-ac5b-976d21a49941 > health: HEALTH_WARN > 2 osds down > Reduced data availability: 49 pgs inactive > Degraded data redundancy: 15722192/79848340 objects degraded > (19.690%), 246 pgs degraded, 247 pgs undersized > > services: > mon: 3 daemons, quorum zephir,debian,raspi (age 20h) [leader: > zephir] > mgr: zephir.enywvy(active, since 21h), standbys: debian.nusuye > mds: 3/3 daemons up, 3 standby > osd: 18 osds: 16 up (since 5m), 18 in (since 3d); 8 remapped > pgs > cephfs-mirror: 2 daemons active (2 hosts) > rbd-mirror: 2 daemons active (2 hosts) > rgw: 1 daemon active (1 hosts, 1 zones) > tcmu-runner: 5 portals active (2 hosts) > > data: > volumes: 3/3 healthy > pools: 25 pools, 450 pgs > objects: 17.52M objects, 62 TiB > usage: 116 TiB used, 63 TiB / 179 TiB avail > pgs: 10.889% pgs not active > 15722192/79848340 objects degraded (19.690%) > 197 active+undersized+degraded > 183 active+clean > 49 undersized+degraded+peered > 15 active+clean+scrubbing > 5 active+clean+scrubbing+deep > 1 active+undersized > > io: > client: 1.7 KiB/s rd, 1023 B/s wr, 1 op/s rd, 0 op/s wr > > root@zephir:~# ceph osd status > ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE > 0 debian 11.2T 5289G 0 0 0 0 exists,up > 1 zephir 11.6T 4846G 0 0 0 0 exists > 2 zephir 10.7T 5827G 0 0 2 4 exists,up > 3 zephir 11.0T 5497G 0 0 0 0 exists > 4 zephir 159G 1376G 0 0 2 0 exists,up > 5 zephir 120G 1415G 0 0 0 0 exists,up > 6 debian 11.3T 5200G 0 0 1 0 exists,up > 7 zephir 305G 1230G 0 0 0 0 exists,up > 8 zephir 10.9T 5594G 0 0 0 0 exists,up > 9 zephir 11.9T 4616G 0 0 0 0 exists,up > 10 zephir 235G 1300G 0 0 0 0 exists,up > 11 zephir 223G 1312G 0 0 0 0 exists,up > 12 zephir 11.0T 5479G 0 0 0 0 exists,up > 13 debian 118G 631G 0 0 3 8 exists,up > 14 debian 345G 1190G 0 0 0 0 exists,up > 15 debian 11.4T 5079G 0 0 0 0 exists,up > 16 debian 554G 3029G 0 0 3 8 exists,up > 17 debian 12.5T 5874G 0 0 1 0 exists,up > root@zephir:~# > > I attached the log of osd.1 to > https://filebin.net/jwvs6kuqrc7hx8id > > Best Regards, > > Reto > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

rgw - migrate data from replicated to ec pool
by Boris 21 Jan '26

21 Jan '26

Hi, we want to migrate the data from a replicated to an EC pool. Is there a way to do it without downtime? Cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

Re: Prevent rebalancing after setting crush map (NVMe-first crush rule)
by Jan Kasprzak 20 Jan '26

20 Jan '26

Andreas, Anthony, Andreas Eriksson via ceph-users wrote: > I've done exactly this recently. > > Changing crush rule to "step take default class hdd". And with 'nobackfill' > set, I ran "pgremapper cancel-backfill --yes" two or three times > to make the misplaced objects reduce from 95% -> 18%. OK, this helped somewhat. Both pgremapper and upmap-remapped.py failed for me, I had to do this: ceph osd set-require-min-compat-client octopus Then I edited the crush rule for my biggest replicated pool adding "class hdd" to the line "step take default", compiled and set a new crush map, and ran pgremapper. Then there were no remapped pgs. So far good. Then I added "class hdd" to the crush rules of all the other pools as well and set a new crush map. After that, pgremapper printed the following type errors: WARNING: pg 77.1b: conflicting mapping 34->9 found when trying to map 39->34 WARNING: pg 71.12: conflicting mapping 8->21 found when trying to map 6->8 WARNING: pg 68.11: conflicting mapping 25->23 found when trying to map 37->25 WARNING: pg 68.11: conflicting mapping 4->0 found when trying to map 11->4 (many more probably for all PGs of these three pools, with pools 68, 71, and 77 being my erasure-coded pools). ./upmap-remapped.py | sh -x apparently does not print this type of error. Now I have 56 out of ~650 pgs remapped, and about 6 % of objects misplaced. Interestingly enough, most of the remapped pgs are from my main replicated pool, which were not remapped after the first run of pgremapper. But 6 % this is small enough, so I am letting it remap itself now. Thanks! -Yenya > > This was on a Quincy (17.2.x) cluster. > > On 1/19/26 5:48 PM, Anthony D'Atri via ceph-users wrote: > > I’m surprised that this would cause data to move. > > > > > How can I pin the PGs to HDD-based OSDs where they are, without the cluster > > > rebalancing itself? > > You can use upmap-remapped.py, though the balancer will progressively still move them if that’s what CRUSH really wants. > > > > I have an RFE in to at the very least raise a warning if any rules/pools do not specify a device class, ideally setting one by default. > > > > > The whole CRUSH map can be seen here: > > > > > > https://www.fi.muni.cz/~kas/tmp/crushmap.txt > > > > > > Thanks! > > > > > > -Yenya > > > > > > Jan Kasprzak via ceph-users wrote: > > > > Hello, Ceph users, > > > > > > > > how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs, > > > > with all replicas on different hosts? > > > > > > > > One of my clusters has been upgraded to two brand new ~6TB NVMe disks > > > > together with previously used two 12TB HDDs per host. > > > > Right now I have two HDD-based OSDs per host, each with ~200GB partition > > > > on NVMe for metadata. The rest of NVMe is free, so I can now create > > > > two additional ~6TB NVMe OSDs per host. > > > > > > > > My question is how to create a CRUSH rule which would have three replicas > > > > on three different hosts, with one of these replicas on NVMe, and the > > > > remaining two on HDD. > > > > > > > > In the docs here: > > > > https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-ru… > > > > there is an example of a SSD-first rule, which is close to what I would like > > > > to do, but it seems the SSD replica in that rule can end up on the same host > > > > as one of the HDD replicas. How can I avoid that? > > > > > > > > And a noob question -- how can I create a crush rule from the > > > > "rule something { ... }" text in that example? > > > > > > > > Thanks! > > > > > > > > -Yenya > > > > > > > > -- > > > > | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | > > > > | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | > > > > I don't like Python; its lack of inline, anonymous, multi-statement > > > > functions makes me sad. --Eric Wastl > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > -- > > > | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | > > > | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | > > > I don't like Python; its lack of inline, anonymous, multi-statement > > > functions makes me sad. --Eric Wastl > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl

1 0

Re: MDS tuning for large production cluster
by Devin A. Bougie 20 Jan '26

20 Jan '26

Thanks so much, Elia! I greatly appreciate your reply and pointer to the CephFS dashboard! We continue to get a few “MDS_CLIENT_RECALL” "clients failing to respond to cache pressure” messages each day, but it’s much quieter than it used to be and overall everything seems to be working well. I’m afraid our workload is *very* unpredictable and varies greatly between clients and even processes within clients, so maybe a few MDS_CLIENT_RECALL’s are to be expected. I’ll continue to play with things a bit, but again appreciate you taking the time to glance at our tuning and share what you’ve done. Sincerely, Devin > On Jan 16, 2026, at 6:12 AM, Oggian Elia via ceph-users <ceph-users(a)ceph.io> wrote: > > Hello Devin, > > I struggled a lot with MDS tuning on a full NVMe SSD cephfs only cluster. > > To give you an idea of the HW and sizing we have 3 MON nodes (128 cores / 128GB) and 7 OSD nodes (48cores / 256GB RAM ) with 8x 7.68TB NVMe SSD (2x OSDs per device) > > Obviously the MDS tuning and CephFS usage in general heavily depends on the workload and I/O patterns that are using it. > > To provide some context, we have ~150 nodes of Alps (https://www.cscs.ch/computers/alps ) and ~20 nodes of a Kubernetes cluster (running on VMs) mounting cephfs and doing I/O in predictable paths and directory structures. The filesystem is used as scratch file system for jobs, each jobs creates a number of directories in a known path, which makes directory pinning easier. > > With a single MDS it was not able to cope with the load and using this very useful dashboard (https://grafana.com/grafana/dashboards/9340-ceph-cephfs/) I was able to see MDS accumulating log segments and finally crash. One crash even led to FS corruption that I was not able to fix and I had to nuke the whole fs. > > That said, after upgrading to Squid I decided to scale UP MDSs to 6 and to configure ephemeral pinning on the parent directory of the jobs. This resulted in decent spread of load on the MDS servers and good overall behavior of the filesystem, avg reply latencies in the order of a few ms with some spikes up to ~100ms under heavy load, no more log segments accumulation, stable operations. > > This is our current MDS configuration: > > mds_cache_memory_limit 68719476736 > mds_cache_mid 0.700000 > mds_cache_reservation 0.050000 > mds_cache_trim_decay_rate 0.800000 > mds_cache_trim_threshold 524288 > mds_cap_revoke_eviction_timeout 300.000000 > mds_health_cache_threshold 1.500000 > mds_log_max_segments 256 > mds_max_caps_per_client 50000 > mds_recall_global_max_decay_threshold 131072 > mds_recall_max_caps 30000 > mds_recall_max_decay_rate 1.500000 > mds_recall_max_decay_threshold 131072 > mds_recall_warning_decay_rate 60.000000 > mds_recall_warning_threshold 262144 > > mds_beacon_grace 120.000000 > mon_mds_skip_sanity true > mds_client_delegate_inos_pct 0 > mds_session_blocklist_on_evict false > mds_session_blocklist_on_timeout false > > > I don’t consider myself an expert but I hope this can be helpful. > > Best Regards, > > > ELIA OGGIAN > SYSTEM ENGINEER > > CSCS > Centro Svizzero di Calcolo Scientifico > Swiss National Supercomputing Centre > elia.oggian(a)cscs.ch<mailto:elia.oggian@cscs.ch> > > > From: Devin A. Bougie via ceph-users <ceph-users(a)ceph.io> > Date: Thursday, 15 January 2026 at 23:37 > To: ceph-users <ceph-users(a)ceph.io> > Subject: [ceph-users] MDS tuning for large production cluster > > We have a 19.2.3 cluster managed by cephadm with five management nodes and 21 OSD nodes. We have roughly 300 linux clients across several subnets doing cephfs kernel mounts running a wide range of applications. > > Each management node has 760GB of memory. We’re currently using a single active MDS daemon, but have experimented with multiple MDS daemons and directory pinning. > > Each OSD for the cephfs data pool uses hdd (SATA) drives, with the DB and WAL on nvme's internal to the storage node. The cephfs metadata pool is on nvme drives internal to the management nodes. > > After a lot of testing in attempts to quiet down persistent MDS_CLIENT_LATE_RELEASE "clients failing to respond to capability release” and MDS_CLIENT_RECALL "clients failing to respond to cache pressure” warnings, we’ve ended up with the following settings. This seems to be working well. We still get periodic MDS_CLIENT_RECALL warnings, but not nearly as many as we were seeing and they clear relatively quickly. > > I’d greatly appreciate any suggestions for further improvements, or any concerns anyone has with these. > > Many thanks, > Devin > > ——— > cephfs session_timeout 120 > mds mds_cache_memory_limit 549755813888 mds mds_cache_mid 0.700000 mds mds_cache_reservation 0.100000 mds mds_cache_trim_decay_rate 0.900000 mds mds_cache_trim_threshold 524288 mds mds_cap_revoke_eviction_timeout 30.000000 mds mds_health_cache_threshold 2.000000 mds mds_log_max_segments 256 mds mds_max_caps_per_client 5000000 mds mds_recall_global_max_decay_threshold 1048576 mds mds_recall_max_caps 5000 mds mds_recall_max_decay_rate 1.500000 mds mds_recall_max_decay_threshold 524288 mds mds_recall_warning_decay_rate 120.000000 mds mds_recall_warning_threshold 262144 > ——— > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Metrics for multi-site sync delay?
by Casey Bodley 20 Jan '26

20 Jan '26

for the tentacle release, "rgw_sync_delta" perf counters were added to track the "oldest incremental change not applied" in https://github.com/ceph/ceph/pull/61418. i don't believe they've been exposed by the dashboard yet On Mon, Jan 19, 2026 at 11:39 AM Matthew Vernon via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > The RGW Sync Overview dashboard has graphs for replication (objects and > byes/s) from source, polling request latency, and unsuccessful object > replications. But it doesn't have anything like "age of oldest > incremental change not applied". > > i.e. if I do radosgw-admin sync status and replication is not fully > caught up, I get lines like: > > data is behind on 1 shards > oldest incremental change not applied: 2026-01-19T13:51:27.418794+0000 [39] > > but neither is graphed (or maybe even in the output metrics). Is there a > handy way of getting these in machine-readable manner? We're thinking we > might want to e.g. alert on a high number of behind shards and/or the > age of the oldest not applied change. > > We run Reef / Cephadm. > > Thanks, > > Matthew > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Prevent rebalancing after setting crush map (NVMe-first crush rule)
by Andreas Eriksson 20 Jan '26

20 Jan '26

I've done exactly this recently. Changing crush rule to "step take default class hdd". And with 'nobackfill' set, I ran "pgremapper cancel-backfill --yes" two or three times to make the misplaced objects reduce from 95% -> 18%. This was on a Quincy (17.2.x) cluster. On 1/19/26 5:48 PM, Anthony D'Atri via ceph-users wrote: > I’m surprised that this would cause data to move. > >> How can I pin the PGs to HDD-based OSDs where they are, without the cluster >> rebalancing itself? > You can use upmap-remapped.py, though the balancer will progressively still move them if that’s what CRUSH really wants. > > I have an RFE in to at the very least raise a warning if any rules/pools do not specify a device class, ideally setting one by default. > >> The whole CRUSH map can be seen here: >> >> https://www.fi.muni.cz/~kas/tmp/crushmap.txt >> >> Thanks! >> >> -Yenya >> >> Jan Kasprzak via ceph-users wrote: >>> Hello, Ceph users, >>> >>> how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs, >>> with all replicas on different hosts? >>> >>> One of my clusters has been upgraded to two brand new ~6TB NVMe disks >>> together with previously used two 12TB HDDs per host. >>> Right now I have two HDD-based OSDs per host, each with ~200GB partition >>> on NVMe for metadata. The rest of NVMe is free, so I can now create >>> two additional ~6TB NVMe OSDs per host. >>> >>> My question is how to create a CRUSH rule which would have three replicas >>> on three different hosts, with one of these replicas on NVMe, and the >>> remaining two on HDD. >>> >>> In the docs here: >>> https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-ru… >>> there is an example of a SSD-first rule, which is close to what I would like >>> to do, but it seems the SSD replica in that rule can end up on the same host >>> as one of the HDD replicas. How can I avoid that? >>> >>> And a noob question -- how can I create a crush rule from the >>> "rule something { ... }" text in that example? >>> >>> Thanks! >>> >>> -Yenya >>> >>> -- >>> | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | >>> | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | >>> I don't like Python; its lack of inline, anonymous, multi-statement >>> functions makes me sad. --Eric Wastl >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> -- >> | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | >> | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | >> I don't like Python; its lack of inline, anonymous, multi-statement >> functions makes me sad. --Eric Wastl >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Prevent rebalancing after setting crush map (NVMe-first crush rule)
by Anthony D'Atri 19 Jan '26

19 Jan '26

I’m surprised that this would cause data to move. > How can I pin the PGs to HDD-based OSDs where they are, without the cluster > rebalancing itself? You can use upmap-remapped.py, though the balancer will progressively still move them if that’s what CRUSH really wants. I have an RFE in to at the very least raise a warning if any rules/pools do not specify a device class, ideally setting one by default. > > The whole CRUSH map can be seen here: > > https://www.fi.muni.cz/~kas/tmp/crushmap.txt > > Thanks! > > -Yenya > > Jan Kasprzak via ceph-users wrote: >> Hello, Ceph users, >> >> how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs, >> with all replicas on different hosts? >> >> One of my clusters has been upgraded to two brand new ~6TB NVMe disks >> together with previously used two 12TB HDDs per host. >> Right now I have two HDD-based OSDs per host, each with ~200GB partition >> on NVMe for metadata. The rest of NVMe is free, so I can now create >> two additional ~6TB NVMe OSDs per host. >> >> My question is how to create a CRUSH rule which would have three replicas >> on three different hosts, with one of these replicas on NVMe, and the >> remaining two on HDD. >> >> In the docs here: >> https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-ru… >> there is an example of a SSD-first rule, which is close to what I would like >> to do, but it seems the SSD replica in that rule can end up on the same host >> as one of the HDD replicas. How can I avoid that? >> >> And a noob question -- how can I create a crush rule from the >> "rule something { ... }" text in that example? >> >> Thanks! >> >> -Yenya >> >> -- >> | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | >> | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | >> I don't like Python; its lack of inline, anonymous, multi-statement >> functions makes me sad. --Eric Wastl >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | > | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | > I don't like Python; its lack of inline, anonymous, multi-statement > functions makes me sad. --Eric Wastl > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Metrics for multi-site sync delay?
by Matthew Vernon 19 Jan '26

19 Jan '26

Hi, The RGW Sync Overview dashboard has graphs for replication (objects and byes/s) from source, polling request latency, and unsuccessful object replications. But it doesn't have anything like "age of oldest incremental change not applied". i.e. if I do radosgw-admin sync status and replication is not fully caught up, I get lines like: data is behind on 1 shards oldest incremental change not applied: 2026-01-19T13:51:27.418794+0000 [39] but neither is graphed (or maybe even in the output metrics). Is there a handy way of getting these in machine-readable manner? We're thinking we might want to e.g. alert on a high number of behind shards and/or the age of the oldest not applied change. We run Reef / Cephadm. Thanks, Matthew

1 0

Re: Unable to start scrub on PG with unfound object
by Konstantin Shalygin 19 Jan '26

19 Jan '26

Hi, > On 18 Jan 2026, at 21:04, Michal Strnad via ceph-users <ceph-users(a)ceph.io> wrote: > > State of PG: > > pg 133.186 is stuck undersized for 6d, current state active+backfill_unfound+undersized+degraded+remapped+forced_backfill, last acting Seems this is EC 4+2, right? > [NONE,233,768,1202,NONE,423] > > Our ceph version: 18.2.1 (Reef) > > Do you have any ideas on how to get the scrub to run, or on alternative steps we should take to verify that only this single object is lost? You was try's list all PG's objects and read them? k

1 0

Re: Prevent rebalancing after setting crush map (NVMe-first crush rule)
by Eugen Block 19 Jan '26

19 Jan '26

Hi, check out the --reclassify parameter for crushtool, works like a charm. https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/L3XPGQMJ2Y3… Zitat von Jan Kasprzak via ceph-users <ceph-users(a)ceph.io>: > Hello, Ceph users, > > In my Ceph cluster I have only OSDs of class "hdd" now. In preparation of > adding also NVMe-based OSDs I wanted to prevent cluster rebalancing > as soon as I start adding the new OSDs. So I thought adding "class hdd" > to all my CRUSH rules would do what I want - i.e. pin all the PGs where > they are, on HDD-based OSDs. I did the following: > > ceph osd getcrushmap -o /var/tmp/crushmap.bin > crushtool -d /var/tmp/crushmap.bin > /var/tmp/crushmap.txt > vi /var/tmp/crushmap.txt > > edit: > - step take default > + step take default class hdd > > crushtool -c /var/tmp/crushmap.txt -o /var/tmp/crushmap-new.bin > ceph osd setcrushmap -i /var/tmp/crushmap-new.bin > > However, after doing this, my cluster reports 90 % PGs as remapped, > and wants to completely rebalance itself. After reverting the crush map > with > > ceph osd setcrushmap -i /var/tmp/crushmap.bin > > it quickly went back to a steady state. > > How can I pin the PGs to HDD-based OSDs where they are, without the cluster > rebalancing itself? > > The whole CRUSH map can be seen here: > > https://www.fi.muni.cz/~kas/tmp/crushmap.txt > > Thanks! > > -Yenya > > Jan Kasprzak via ceph-users wrote: >> Hello, Ceph users, >> >> how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs, >> with all replicas on different hosts? >> >> One of my clusters has been upgraded to two brand new ~6TB NVMe disks >> together with previously used two 12TB HDDs per host. >> Right now I have two HDD-based OSDs per host, each with ~200GB partition >> on NVMe for metadata. The rest of NVMe is free, so I can now create >> two additional ~6TB NVMe OSDs per host. >> >> My question is how to create a CRUSH rule which would have three replicas >> on three different hosts, with one of these replicas on NVMe, and the >> remaining two on HDD. >> >> In the docs here: >> https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-ru… >> there is an example of a SSD-first rule, which is close to what I would like >> to do, but it seems the SSD replica in that rule can end up on the same host >> as one of the HDD replicas. How can I avoid that? >> >> And a noob question -- how can I create a crush rule from the >> "rule something { ... }" text in that example? >> >> Thanks! >> >> -Yenya >> >> -- >> | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | >> | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | >> I don't like Python; its lack of inline, anonymous, multi-statement >> functions makes me sad. --Eric Wastl >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | > | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | > I don't like Python; its lack of inline, anonymous, multi-statement > functions makes me sad. --Eric Wastl > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Prevent rebalancing after setting crush map (NVMe-first crush rule)
by Jan Kasprzak 19 Jan '26

19 Jan '26

Hello, Ceph users, In my Ceph cluster I have only OSDs of class "hdd" now. In preparation of adding also NVMe-based OSDs I wanted to prevent cluster rebalancing as soon as I start adding the new OSDs. So I thought adding "class hdd" to all my CRUSH rules would do what I want - i.e. pin all the PGs where they are, on HDD-based OSDs. I did the following: ceph osd getcrushmap -o /var/tmp/crushmap.bin crushtool -d /var/tmp/crushmap.bin > /var/tmp/crushmap.txt vi /var/tmp/crushmap.txt edit: - step take default + step take default class hdd crushtool -c /var/tmp/crushmap.txt -o /var/tmp/crushmap-new.bin ceph osd setcrushmap -i /var/tmp/crushmap-new.bin However, after doing this, my cluster reports 90 % PGs as remapped, and wants to completely rebalance itself. After reverting the crush map with ceph osd setcrushmap -i /var/tmp/crushmap.bin it quickly went back to a steady state. How can I pin the PGs to HDD-based OSDs where they are, without the cluster rebalancing itself? The whole CRUSH map can be seen here: https://www.fi.muni.cz/~kas/tmp/crushmap.txt Thanks! -Yenya Jan Kasprzak via ceph-users wrote: > Hello, Ceph users, > > how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs, > with all replicas on different hosts? > > One of my clusters has been upgraded to two brand new ~6TB NVMe disks > together with previously used two 12TB HDDs per host. > Right now I have two HDD-based OSDs per host, each with ~200GB partition > on NVMe for metadata. The rest of NVMe is free, so I can now create > two additional ~6TB NVMe OSDs per host. > > My question is how to create a CRUSH rule which would have three replicas > on three different hosts, with one of these replicas on NVMe, and the > remaining two on HDD. > > In the docs here: > https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-ru… > there is an example of a SSD-first rule, which is close to what I would like > to do, but it seems the SSD replica in that rule can end up on the same host > as one of the HDD replicas. How can I avoid that? > > And a noob question -- how can I create a crush rule from the > "rule something { ... }" text in that example? > > Thanks! > > -Yenya > > -- > | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | > | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | > I don't like Python; its lack of inline, anonymous, multi-statement > functions makes me sad. --Eric Wastl > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl

1 0

Re: Unable to start scrub on PG with unfound object
by Eugen Block 19 Jan '26

19 Jan '26

Hi, is this an EC pool, maybe k4 m2? If it is, have you tried setting min_size to k for this pool? And if you do that, please don't forget to set it back to k + 1. Do you have the 'ceph pg 133.186 query' output for this PG? What's the health status of the rest of the cluster? I'm not entirely sure in which state a PG has to be to be able to deep-scrub, but this one is in forced_backfill state. Maybe if you cancelled that (ceph pg cancel-force-backfill) a scrub could be scheduled? What's the scrub status of that PG anyway ('ceph pg dump' has that information)? And one more thing, according to this: [NONE,233,768,1202,NONE,423] OSD.233 is not the primary OSD but the secondary. Regards, Eugen Zitat von Michal Strnad via ceph-users <ceph-users(a)ceph.io>: > Hi. > > We have a single unfound object. All OSDs are up, so a simple “put > down to up” of an OSD is unlikely to recover it. > > Before we notify the object owner and mark the object as lost, I > would like to scrub the PG to confirm that only this one object is > missing. However, we are unable to force either a normal scrub or a > deep scrub on the affected PG (e.g., ceph pg scrub 133.186 or ceph > pg deep‑scrub 133.186). We even increased osd_max_scrubs from 1 to 3 > on all OSDs (which are participating on PG 133.186): > > > for i in 328 233 768 1202 84 423; do > ceph config set osd.$i osd_max_scrubs 3 > done > > Despite this, the scrub still does not start. We have set > osd_scrub_during_recovery to true. I am attaching the configuration > of the primary OSD (osd.233); the other OSDs have identical settings. > > WHO MASK LEVEL OPTION VALUE RO > global basic device_failure_prediction_mode > local osd advanced osd_max_backfills > 10 osd.233 advanced osd_max_scrubs > 3 osd advanced osd_mclock_override_recovery_settings > false osd advanced osd_op_queue > wpq * > global advanced osd_pool_default_pg_autoscale_mode warn > osd advanced osd_repair_during_recovery true > osd advanced osd_scrub_auto_repair true > osd advanced osd_scrub_during_recovery true > osd.233 advanced osd_scrub_load_threshold > 10.000000 > > > State of PG: > > pg 133.186 is stuck undersized for 6d, current state > active+backfill_unfound+undersized+degraded+remapped+forced_backfill, last > acting [NONE,233,768,1202,NONE,423] > > Our ceph version: 18.2.1 (Reef) > > Do you have any ideas on how to get the scrub to run, or on > alternative steps we should take to verify that only this single > object is lost? > > Best regards, > Michal

1 0

Unable to start scrub on PG with unfound object
by Michal Strnad 18 Jan '26

18 Jan '26

Hi. We have a single unfound object. All OSDs are up, so a simple “put down to up” of an OSD is unlikely to recover it. Before we notify the object owner and mark the object as lost, I would like to scrub the PG to confirm that only this one object is missing. However, we are unable to force either a normal scrub or a deep scrub on the affected PG (e.g., ceph pg scrub 133.186 or ceph pg deep‑scrub 133.186). We even increased osd_max_scrubs from 1 to 3 on all OSDs (which are participating on PG 133.186): for i in 328 233 768 1202 84 423; do ceph config set osd.$i osd_max_scrubs 3 done Despite this, the scrub still does not start. We have set osd_scrub_during_recovery to true. I am attaching the configuration of the primary OSD (osd.233); the other OSDs have identical settings. WHO MASK LEVEL OPTION VALUE RO global basic device_failure_prediction_mode local osd advanced osd_max_backfills 10 osd.233 advanced osd_max_scrubs 3 osd advanced osd_mclock_override_recovery_settings false osd advanced osd_op_queue wpq * global advanced osd_pool_default_pg_autoscale_mode warn osd advanced osd_repair_during_recovery true osd advanced osd_scrub_auto_repair true osd advanced osd_scrub_during_recovery true osd.233 advanced osd_scrub_load_threshold 10.000000 State of PG: pg 133.186 is stuck undersized for 6d, current state active+backfill_unfound+undersized+degraded+remapped+forced_backfill, last acting [NONE,233,768,1202,NONE,423] Our ceph version: 18.2.1 (Reef) Do you have any ideas on how to get the scrub to run, or on alternative steps we should take to verify that only this single object is lost? Best regards, Michal

1 0

mailing list's owners/admins ?
by lejeczek 18 Jan '26

18 Jan '26

Hi guys. Does anybody know if this mailing lists admins are actually humans and if so, then how to contact them? I emailed - ceph-users-owner(a)ceph.io - already, trying to get subscription (not this account) but have not heard back, it's a week. many thanks, L.

1 0

Community Manager Updates Early January
by Anthony Middleton 16 Jan '26

16 Jan '26

Hello, Ceph Friends! I hope the New Year is treating you well! I wanted to share a few updates and ongoing projects with you all before we log off for the weekend. 1. *Call for Proposals (CFPs) *are open for two upcoming Ceph Days: - Ceph Days India: *The deadline has been extended to January 18th at 11:59 PM IST*. More details can be found here: https://ceph.io/en/community/events/2026/ceph-days-india/ - Ceph Days Raleigh: *The deadline is February 16th*. For more information, visit: https://ceph.io/en/community/events/2026/ceph-days-raleigh/ 2. The recording of our latest *Ceph Tech Talk, "Running Teuthology Outside of the Sepia Lab,"* is now live. You can watch it here: https://youtu.be/6ovdJ79AqbM. If you have ideas for future tech talks, please share them by responding to this email or contacting community-manager(a)ceph.io. 3. Good news! The *original Ceph subreddit* has been released from Reddit jail. Just kidding! The group was temporarily banned due to a bot attack and insufficient monitoring. Thanks to the efforts of several community members, the account is now unbanned. Please help us keep this account active and report any suspicious activity. If you notice anything unusual, feel free to reach out to me. *Join here: https://www.reddit.com/r/ceph/ <https://www.reddit.com/r/ceph/>.* 4. I've set up a *Slack channel called #getting-started* for Ceph newcomers. If you're new to Ceph and want a welcoming space to ask questions, please join. Mentors and tech leads are also encouraged to participate. Your input would be valuable as we continue to grow our community. *Ceph Slack: https://join.slack.com/t/ceph-storage/shared_invite/zt-3jlvf8f6e-45tyKGpqkk… <https://join.slack.com/t/ceph-storage/shared_invite/zt-3jlvf8f6e-45tyKGpqkk…>* 5. Enjoy this highlight video from *Cephalocon 2025*: https://www.youtube.com/watch?v=cYfls0m9h7Y. That's all for now. If you need anything, you can always email or message me on Slack. Take care and have a great weekend! Best, -- Anthony Middleton Community Manager, Ceph Foundation community-manager(a)ceph.io LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

RBD: exclusive lock vs. advisory locks vs. fencing
by Christian Theune 16 Jan '26

16 Jan '26

Hey, I’m currently preparing a long overdue move from the traditional advisory locks to the exclusive-lock feature and am a bit perplexed by the situation. I’d love some feedback whether I’m missing something or not. We currently use the advisory locks to determine whether it’s safe to start a VM or not. We use exclusive advisory locks on the RBD images that belong to a VM so that we get guaranteed atomicity: if we get the lock, then we’re good to go. The exclusive locks are incompatible with this as they build on the same infrastructure but with different semantics. As the docs say: exclusive locks are used to protect the RBD-internal objects (so the data on the RADOS layer) from being inconsistently manipulated but they will only serialize RBD-level concurrent access. There are some parts of the tool chain that allow having a single client (map --exclusive) but this isn’t consistently exposed and not useful as it AFAICT will undermine the ability for concurrent internal tasks like mirrorin/migration/… Now, the docs also talk about fencing - however - that seems to be missing the original atomicity promise: I can fence (blocklist) a known other client, but I can not be sure that no other client appeared in between. Without atomicity I can’t safely start using this volume. This issue can be solved on a higher level orchestrator, but at that point one loses the ability for consistent low level access to volumes (e.g. for backups, debugging or other tasks). Am I describing the situation accurately? Is this the way it is on purpose or is there room for improvement here? Thanks, Christian -- Christian Theune · ct(a)flyingcircus.io · +49 345 219401 0 Flying Circus Internet Operations GmbH · https://flyingcircus.io Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick

1 0

Re: MDS tuning for large production cluster
by Oggian Elia 16 Jan '26

16 Jan '26

Hello Devin, I struggled a lot with MDS tuning on a full NVMe SSD cephfs only cluster. To give you an idea of the HW and sizing we have 3 MON nodes (128 cores / 128GB) and 7 OSD nodes (48cores / 256GB RAM ) with 8x 7.68TB NVMe SSD (2x OSDs per device) Obviously the MDS tuning and CephFS usage in general heavily depends on the workload and I/O patterns that are using it. To provide some context, we have ~150 nodes of Alps (https://www.cscs.ch/computers/alps ) and ~20 nodes of a Kubernetes cluster (running on VMs) mounting cephfs and doing I/O in predictable paths and directory structures. The filesystem is used as scratch file system for jobs, each jobs creates a number of directories in a known path, which makes directory pinning easier. With a single MDS it was not able to cope with the load and using this very useful dashboard (https://grafana.com/grafana/dashboards/9340-ceph-cephfs/) I was able to see MDS accumulating log segments and finally crash. One crash even led to FS corruption that I was not able to fix and I had to nuke the whole fs. That said, after upgrading to Squid I decided to scale UP MDSs to 6 and to configure ephemeral pinning on the parent directory of the jobs. This resulted in decent spread of load on the MDS servers and good overall behavior of the filesystem, avg reply latencies in the order of a few ms with some spikes up to ~100ms under heavy load, no more log segments accumulation, stable operations. This is our current MDS configuration: mds_cache_memory_limit 68719476736 mds_cache_mid 0.700000 mds_cache_reservation 0.050000 mds_cache_trim_decay_rate 0.800000 mds_cache_trim_threshold 524288 mds_cap_revoke_eviction_timeout 300.000000 mds_health_cache_threshold 1.500000 mds_log_max_segments 256 mds_max_caps_per_client 50000 mds_recall_global_max_decay_threshold 131072 mds_recall_max_caps 30000 mds_recall_max_decay_rate 1.500000 mds_recall_max_decay_threshold 131072 mds_recall_warning_decay_rate 60.000000 mds_recall_warning_threshold 262144 mds_beacon_grace 120.000000 mon_mds_skip_sanity true mds_client_delegate_inos_pct 0 mds_session_blocklist_on_evict false mds_session_blocklist_on_timeout false I don’t consider myself an expert but I hope this can be helpful. Best Regards, ELIA OGGIAN SYSTEM ENGINEER CSCS Centro Svizzero di Calcolo Scientifico Swiss National Supercomputing Centre elia.oggian(a)cscs.ch<mailto:elia.oggian@cscs.ch> www.cscs.ch<http://www.cscs.ch> From: Devin A. Bougie via ceph-users <ceph-users(a)ceph.io> Date: Thursday, 15 January 2026 at 23:37 To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] MDS tuning for large production cluster We have a 19.2.3 cluster managed by cephadm with five management nodes and 21 OSD nodes. We have roughly 300 linux clients across several subnets doing cephfs kernel mounts running a wide range of applications. Each management node has 760GB of memory. We’re currently using a single active MDS daemon, but have experimented with multiple MDS daemons and directory pinning. Each OSD for the cephfs data pool uses hdd (SATA) drives, with the DB and WAL on nvme's internal to the storage node. The cephfs metadata pool is on nvme drives internal to the management nodes. After a lot of testing in attempts to quiet down persistent MDS_CLIENT_LATE_RELEASE "clients failing to respond to capability release” and MDS_CLIENT_RECALL "clients failing to respond to cache pressure” warnings, we’ve ended up with the following settings. This seems to be working well. We still get periodic MDS_CLIENT_RECALL warnings, but not nearly as many as we were seeing and they clear relatively quickly. I’d greatly appreciate any suggestions for further improvements, or any concerns anyone has with these. Many thanks, Devin ——— cephfs session_timeout 120 mds mds_cache_memory_limit 549755813888 mds mds_cache_mid 0.700000 mds mds_cache_reservation 0.100000 mds mds_cache_trim_decay_rate 0.900000 mds mds_cache_trim_threshold 524288 mds mds_cap_revoke_eviction_timeout 30.000000 mds mds_health_cache_threshold 2.000000 mds mds_log_max_segments 256 mds mds_max_caps_per_client 5000000 mds mds_recall_global_max_decay_threshold 1048576 mds mds_recall_max_caps 5000 mds mds_recall_max_decay_rate 1.500000 mds mds_recall_max_decay_threshold 524288 mds mds_recall_warning_decay_rate 120.000000 mds mds_recall_warning_threshold 262144 ——— _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

MDS tuning for large production cluster
by Devin A. Bougie 15 Jan '26

15 Jan '26

We have a 19.2.3 cluster managed by cephadm with five management nodes and 21 OSD nodes. We have roughly 300 linux clients across several subnets doing cephfs kernel mounts running a wide range of applications. Each management node has 760GB of memory. We’re currently using a single active MDS daemon, but have experimented with multiple MDS daemons and directory pinning. Each OSD for the cephfs data pool uses hdd (SATA) drives, with the DB and WAL on nvme's internal to the storage node. The cephfs metadata pool is on nvme drives internal to the management nodes. After a lot of testing in attempts to quiet down persistent MDS_CLIENT_LATE_RELEASE "clients failing to respond to capability release” and MDS_CLIENT_RECALL "clients failing to respond to cache pressure” warnings, we’ve ended up with the following settings. This seems to be working well. We still get periodic MDS_CLIENT_RECALL warnings, but not nearly as many as we were seeing and they clear relatively quickly. I’d greatly appreciate any suggestions for further improvements, or any concerns anyone has with these. Many thanks, Devin ——— cephfs session_timeout 120 mds mds_cache_memory_limit 549755813888 mds mds_cache_mid 0.700000 mds mds_cache_reservation 0.100000 mds mds_cache_trim_decay_rate 0.900000 mds mds_cache_trim_threshold 524288 mds mds_cap_revoke_eviction_timeout 30.000000 mds mds_health_cache_threshold 2.000000 mds mds_log_max_segments 256 mds mds_max_caps_per_client 5000000 mds mds_recall_global_max_decay_threshold 1048576 mds mds_recall_max_caps 5000 mds mds_recall_max_decay_rate 1.500000 mds mds_recall_max_decay_threshold 524288 mds mds_recall_warning_decay_rate 120.000000 mds mds_recall_warning_threshold 262144 ———

1 0

Re: resize underlying volume - disk size increase ?
by Eugen Block 15 Jan '26

15 Jan '26

Hm, maybe I misremember and the expand command isn't for the main device but only for DB/WAL... I'll need to check again, as I wrote, it's been quite a while since I last played with those commands... Zitat von lejeczek via ceph-users <ceph-users(a)ceph.io>: > is this... > > > -> $ systemctl stop ceph-9f4f9dba-72c7-11f0-8052-525400519d29(a)osd.10.service > > -> $ ceph-bluestore-tool bluefs-bdev-sizes --path > /var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/osd.10/ > inferring bluefs devices from bluestore path > 1 : device size 0x4affc00000(300 GiB) : using 0x36f49bd000(220 GiB) > > -> $ ceph-bluestore-tool bluefs-bdev-expand --path > /var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/osd.10/ > inferring bluefs devices from bluestore path > 1 : device size 0x4affc00000(300 GiB) : using 0x36f49bd000(220 GiB) > Expanding DB/WAL... > 1 : nothing to do, skipped > > not weird(wrong) - would not all agree? > > -> $ ceph osd df > ... > 10 hdd 0.29300 1.00000 300 GiB 219 GiB 218 GiB 567 KiB 632 > MiB 81 GiB 72.91 0.99 165 up > > -> $ ceph-volume lvm list > ... > ====== osd.10 ====== > ... > devices /dev/vdc > ... > -> $ fdisk -l /dev/vdc > Disk /dev/vdc: 400 GiB, 429496729600 bytes, 104857600 sectors > > Something, somewhere Ceph is ignoring, not picking up...? > From what I can find 'ceph-volume' does not offer re-sizing volumes. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 1

Re: Dual stack support in squid?
by Burkhard Linke 15 Jan '26

15 Jan '26

Hi Robert, On 1/15/26 16:57, Robert Sander via ceph-users wrote: > Hi, > > Am 15.01.26 um 3:24 PM schrieb Burkhard Linke via ceph-users: > >> Nonetheless, since most clients are already running with dual stack >> (the k8s cluster will be the last and probably most "interesting" >> setup..), let's adopt the migration and target ipv6 only: >> >> - set ms_bind_ipv4 to false and ms_bind_ipv6 to true >> >> - delete and re-add mons one after another >> >> - process the other components like OSDs etc. > > As a cluster cannot run dual stack there is unfortunately no way to > migrate from IPv4 to IPv6 online. > > You may be able to edit the MON DBs on all MONs manually when the > cluster is completely offline and replace the MON IPs there. > > But I have no experience in doing this. Online migration seems to be possible: - ensure that ipv6 _and_ ipv4 connectivity works for all ceph hosts and ceph clients - set ms_bind_ipv4=false and ms_bind_ipv6=true, remove ipv4 subnets from public_networks - add a new mon for safety - delete and readd the old mons -> should be using ipv6 only now - restart OSDs and other components -> will bind to ipv6, client will be updated with new ipv6 addresses to connect to I'm currently running a test with mon, mds and osds on one host using ipv6. The v6 addresses are correctly populated in monmap, osdmap and mdsmap. Client connections are using v6 (as advertised in the maps), connections between osds are using v4 if the osds are still running with ipv4. So if your hossts are dual stacked, you should be able to online migrate from ipv4 only to ipv6 only. Best regards, Burkhard

1 0

Re: resize underlying volume - disk size increase ?
by lejeczek 15 Jan '26

15 Jan '26

is this... -> $ systemctl stop ceph-9f4f9dba-72c7-11f0-8052-525400519d29(a)osd.10.service -> $ ceph-bluestore-tool bluefs-bdev-sizes --path /var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/osd.10/ inferring bluefs devices from bluestore path 1 : device size 0x4affc00000(300 GiB) : using 0x36f49bd000(220 GiB) -> $ ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/osd.10/ inferring bluefs devices from bluestore path 1 : device size 0x4affc00000(300 GiB) : using 0x36f49bd000(220 GiB) Expanding DB/WAL... 1 : nothing to do, skipped not weird(wrong) - would not all agree? -> $ ceph osd df ... 10 hdd 0.29300 1.00000 300 GiB 219 GiB 218 GiB 567 KiB 632 MiB 81 GiB 72.91 0.99 165 up -> $ ceph-volume lvm list ... ====== osd.10 ====== ... devices /dev/vdc ... -> $ fdisk -l /dev/vdc Disk /dev/vdc: 400 GiB, 429496729600 bytes, 104857600 sectors Something, somewhere Ceph is ignoring, not picking up...? From what I can find 'ceph-volume' does not offer re-sizing volumes.

1 0

Re: Dual stack support in squid?
by Robert Sander 15 Jan '26

15 Jan '26

Hi, Am 15.01.26 um 3:24 PM schrieb Burkhard Linke via ceph-users: > Nonetheless, since most clients are already running with dual stack (the > k8s cluster will be the last and probably most "interesting" setup..), > let's adopt the migration and target ipv6 only: > > - set ms_bind_ipv4 to false and ms_bind_ipv6 to true > > - delete and re-add mons one after another > > - process the other components like OSDs etc. As a cluster cannot run dual stack there is unfortunately no way to migrate from IPv4 to IPv6 online. You may be able to edit the MON DBs on all MONs manually when the cluster is completely offline and replace the MON IPs there. But I have no experience in doing this. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: resize underlying volume - disk size increase ?
by Eugen Block 15 Jan '26

15 Jan '26

First: yes, the OSD has to be down to be able to operate on it. Second: for cephadm managed daemons you'll need to use the cephadm shell, so something like (omit the FSID because the full path is mapped into the container): cephadm shell -- ceph-bluestore-tool bluefs-bdev-sizes --path /var/lib/ceph/osd.9/ Third: there's a bluefs-bdev-expand command for ceph-bluestore-tool. That could help here (IIUC, your main device has more space, no migration of DB/WAL). I haven't done this in a while (never on a production cluster, concurring with Janne's response), but I'm pretty sure it worked. So stop osd (set noout), resize PV (which you already have), run bluestore-tool (within cephadm shell) to expand the device, then start the osd (and unset noout). Zitat von lejeczek via ceph-users <ceph-users(a)ceph.io>: > In my lab ceph is deployed with cephadm - so, containerized env, when I do: > > -> $ ceph-bluestore-tool bluefs-bdev-sizes --path > /var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/osd.9/ > ... > I get a coredump - on each node, each osd - as I thumbed through the > net I read somewhere that an 'osd' must be stopped for > 'ceph-bluestore-tool' to operate on it? > But how would that make sense ? not in my mind, but hey... or I'm > doing something forbidden there. > > I don't think a "migration" - loosely defined - applies to my > scenario as there is nothing to migration onto - disks which Cephs > uses "got" more space - no new/more disks/devices. > > Ceph sees those changes already: > -> $ ceph orch device ls > podster1.mine.priv /dev/vdc hdd 400G No 13m ago > Has a FileSystem, LVM detected > ... > podster3.mine.priv /dev/vdc hdd 300G No 13m ago > Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM > detected > > first above, is easy to guess - got extra 100GB. I did fiddled with > it actually "outside" of ceph, did 'pvresize' (PV which Ceph itself > crerated and used LVM) which did not cause any troubles, apparently. > So I was itching to do the 'lvresize' too but though Ceph's way > should be (existent) preferred - I can imagine even in _production_ > people do use.. > partitions/LVM - as opposed to whole disks/drives.. no? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: resize underlying volume - disk size increase ?
by Janne Johansson 15 Jan '26

15 Jan '26

> So I was itching to do the 'lvresize' too but though Ceph's > way should be (existent) preferred - I can imagine even in > _production_ people do use.. > partitions/LVM - as opposed to whole disks/drives.. no? Well, in my production setups, disks just never "grow", they get replaced, so having ceph work redo a disk with a bad setting by adding another and remaking the OSD is kind of how we do it. I get that it can be interesting to see how to live-grow what basically amounts to a database on a partition, but since we left filestore which was easy to expand in this manner for bluestore, this is a lot harder now, but it was never a real use-case for serious ceph clusters at our place to mess around with partition sizes in this way. Yes, we use whole disks, but ceph-volume makes them into lvm (with cryptsetup on top if asked) anyhow for other reasons than resizing support. -- May the most significant bit of your life be positive.

1 0

Re: resize underlying volume - disk size increase ?
by lejeczek 15 Jan '26

15 Jan '26

In my lab ceph is deployed with cephadm - so, containerized env, when I do: -> $ ceph-bluestore-tool bluefs-bdev-sizes --path /var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/osd.9/ ... I get a coredump - on each node, each osd - as I thumbed through the net I read somewhere that an 'osd' must be stopped for 'ceph-bluestore-tool' to operate on it? But how would that make sense ? not in my mind, but hey... or I'm doing something forbidden there. I don't think a "migration" - loosely defined - applies to my scenario as there is nothing to migration onto - disks which Cephs uses "got" more space - no new/more disks/devices. Ceph sees those changes already: -> $ ceph orch device ls podster1.mine.priv /dev/vdc hdd 400G No 13m ago Has a FileSystem, LVM detected ... podster3.mine.priv /dev/vdc hdd 300G No 13m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected first above, is easy to guess - got extra 100GB. I did fiddled with it actually "outside" of ceph, did 'pvresize' (PV which Ceph itself crerated and used LVM) which did not cause any troubles, apparently. So I was itching to do the 'lvresize' too but though Ceph's way should be (existent) preferred - I can imagine even in _production_ people do use.. partitions/LVM - as opposed to whole disks/drives.. no?

1 0

Re: resize underlying volume - disk size increase ?
by Joachim Kraftmayer 15 Jan '26

15 Jan '26

hope it helps: https://docs.clyso.com/blog/ceph-osd-migrate-db-to-larger-ssd-flash-device/ joachim.kraftmayer(a)clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677 Am Do., 15. Jan. 2026 um 15:26 Uhr schrieb Burkhard Linke via ceph-users < ceph-users(a)ceph.io>: > Hi, > > On 1/15/26 15:12, lejeczek via ceph-users wrote: > > Hi guys. > > > > I do not suppose this happens in often - it's a lab in my case - when > > disks sizes, which Ceph uses increase. > > Is there a procedure - or howto/method if such thing is not > > approved/recommended - to have Ceph make use of such added disk space? > > > You can use the ceph-bluestore-tool to resize OSD _DB_ partitions, not > sure whether it also supports resizing the actual data partition. > > If in doubt the probably better method is draining the OSD, remove it > and recreate it afterwards. > > > Best regards, > > Burkhard > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: resize underlying volume - disk size increase ?
by Burkhard Linke 15 Jan '26

15 Jan '26

Hi, On 1/15/26 15:12, lejeczek via ceph-users wrote: > Hi guys. > > I do not suppose this happens in often - it's a lab in my case - when > disks sizes, which Ceph uses increase. > Is there a procedure - or howto/method if such thing is not > approved/recommended - to have Ceph make use of such added disk space? You can use the ceph-bluestore-tool to resize OSD _DB_ partitions, not sure whether it also supports resizing the actual data partition. If in doubt the probably better method is draining the OSD, remove it and recreate it afterwards. Best regards, Burkhard

1 0

Re: Dual stack support in squid?
by Burkhard Linke 15 Jan '26

15 Jan '26

Hi Adam, On 1/15/26 14:21, Adam Prycki via ceph-users wrote: > Hello, > > ceph doesn't support dual-stack networking. > Cluster components bind to either v4 or v6 address. > > It's the same with multiple subnets. You can have multiple subnets > configured in your cluster, but each component binds only to one of > these subnets. > > > - How does the mon process decide which address to use? > Lookup ms_bind_ipv4 and ms_bind_ipv6. (these options are exclusive, as > I mentioned above) > > Thanks for the information. https://docs.ceph.com/en/squid/rados/configuration/network-config-ref/ does not mention that the ms_bind_ipv4 an ms_bind_ipv6 options are exclusive, so I'm a little bit confused. I would have expected software like ceph to handle dual stack setups in 2026. If I remember correctly this was one of the goals of the MsgrV2 protocol. Nonetheless, since most clients are already running with dual stack (the k8s cluster will be the last and probably most "interesting" setup..), let's adopt the migration and target ipv6 only: - set ms_bind_ipv4 to false and ms_bind_ipv6 to true - delete and re-add mons one after another - process the other components like OSDs etc. Do Ceph components use their public address to bind _outgoing_ connections to a certain interface/IP address? If this is the case I do not see any sane and simple way to migrate to IPv6. Example: mon A: 1.2.3.1/24 (ceph public) and 1234::1/64 mon B: 1.2.3.2/24 and 1234::2/64 (ceph public) If mon A want to send packets to mon B, it can use its ipv6 address as source, which is _not_ the public address of the mon; vice versa for connections from mon B to mon A. in this setup a migration to IPv6 is straightforward, if all clients are capable of IPv6. Best regards, Burkhard

1 0

resize underlying volume - disk size increase ?
by lejeczek 15 Jan '26

15 Jan '26

Hi guys. I do not suppose this happens in often - it's a lab in my case - when disks sizes, which Ceph uses increase. Is there a procedure - or howto/method if such thing is not approved/recommended - to have Ceph make use of such added disk space? many thanks, L.

1 0

Re: Dual stack support in squid?
by Adam Prycki 15 Jan '26

15 Jan '26

Hello, ceph doesn't support dual-stack networking. Cluster components bind to either v4 or v6 address. It's the same with multiple subnets. You can have multiple subnets configured in your cluster, but each component binds only to one of these subnets. > - How does the mon process decide which address to use? Lookup ms_bind_ipv4 and ms_bind_ipv6. (these options are exclusive, as I mentioned above) Best regerds Adam Prycki On 15/01/2026 14:10, Burkhard Linke via ceph-users wrote: > Hi, > > > we are currently migrating our ipv4 infrastructure to ipv6, and maybe > later to ipv6 only. This also include our ceph clusters. > > My naive approach this is: > > 1. add a forth (ipv4) mon as safe guard and remove an existing mon > > 2. set ms_bind_ipv6 to true and add ipv6 networks to public_network > (both in ceph.conf and via ceph config) > > 3. manually re-add the removed mon with current map and key > > 4. repeat with other mons and afterwards restart osds and other services > to bind to ipv6, too > > > The re-added mon in step 3 does not bind to ipv6, only to ipv4. (both > port 3300 and 6789). According to the `ceph tell mon.XYZ config show` > output both the ms_bind_ipv6 and the public_network settings are > recognized by the process. > > > So the questions are: > > - Is dual stack supported for mons? The documentation does not mention > it explicitly... > > - Can a mon bind to multiple addresses at all, e.g. one IPv4 and one IPv6? > > - How does the mon process decide which address to use? > > > Best regards, > > Burkhard > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Dual stack support in squid?
by Burkhard Linke 15 Jan '26

15 Jan '26

Hi, we are currently migrating our ipv4 infrastructure to ipv6, and maybe later to ipv6 only. This also include our ceph clusters. My naive approach this is: 1. add a forth (ipv4) mon as safe guard and remove an existing mon 2. set ms_bind_ipv6 to true and add ipv6 networks to public_network (both in ceph.conf and via ceph config) 3. manually re-add the removed mon with current map and key 4. repeat with other mons and afterwards restart osds and other services to bind to ipv6, too The re-added mon in step 3 does not bind to ipv6, only to ipv4. (both port 3300 and 6789). According to the `ceph tell mon.XYZ config show` output both the ms_bind_ipv6 and the public_network settings are recognized by the process. So the questions are: - Is dual stack supported for mons? The documentation does not mention it explicitly... - Can a mon bind to multiple addresses at all, e.g. one IPv4 and one IPv6? - How does the mon process decide which address to use? Best regards, Burkhard

1 0

Re: Huge (and increasing) list of removed_snaps
by Eugen Block 15 Jan '26

15 Jan '26

Hi, do you maybe have some history about snapshot usage etc? Did someone create a pool snapshot on rados level (which would make those unmanaged snaps) but isn't allow to delete it? At least that's what I interpret here [0]: if (!is_unmanaged_snap_op_permitted(cct, mon->key_server, session->entity_name, session->caps, session->get_peer_socket_addr(), pool_name)) { dout(0) << "got unmanaged-snap pool op from entity with insufficient " << "privileges. message: " << *m << std::endl << "caps: " << session->caps << dendl; _pool_op_reply(op, -EPERM, osdmap.get_epoch()); return true; } And reading that, is your log excerpt missing some more output? If you have, please add more details. Anyway, I expect the log to contain the entity's name and the relevant caps. Those could help to understand better what exactly is going on here. Regards, Eugen [0] https://github.com/ceph/ceph/blob/v14.2.22/src/mon/OSDMonitor.cc#L13017 Zitat von Work Ceph via ceph-users <ceph-users(a)ceph.io>: > Hello guys, > > We are using Ceph Nautilus, and we are facing an unexpected situation. We > see a huge list of removed_snaps, and they do not seem to be cleaned. We > also see increasing use of CPUs in Ceph monitors (we have thousands of > these elements). However, we are not sure if this situation is related. > > In the logs, we noticed the following message, which seems to be related to > this situation: > > ``` > got unmanaged-snap pool op from entity with insufficient privileges. > message: pool_op > ``` > > Has anyone faced similar situations with Ceph and RBD snapshots management? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Cannot adopt osd (on squid)
by Eugen Block 15 Jan '26

15 Jan '26

Since SUSE is building Ceph packages again, I tested the adoption of a Squid package based test cluster (no data in it), and it worked like a charm. Of course, the MDS container image part came up again, that should definitely be in the docs to override the default before creating MDS containers. I'm gonna file a tracker for that. Zitat von Chris Palmer via ceph-users <ceph-users(a)ceph.io>: > I have a squid package-installed test cluster that I am trying to > adopt into cephadm. > > Monitors & managers have been adopted, and the orchestrator is functioning. > > When I try to adopt an OSD though (e.g. osd.0), cephadm creates the > new directory (<fsid>/osd.0) and populates the files from the legacy > directory. It then crashes complaining that the directory osd/ceph-0 > is not empty so cannot be removed. In some cases none of the files > have been removed, and in others the block symlink and > require_osd_release file remain. I can remove the new <fsid>/osd.0 > and use ceph-volume to activate the OSD again. > > cephadm.log reports that every file has been moved and the ownership > set - even for files that still remain in the legacy directory. It > doesn't report any other errors (apart from the directory not empty > error. when it crashes). > > I really can't find where to go from here. Any pointers appreciated... > > Thanks, Chris > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Huge (and increasing) list of removed_snaps
by Work Ceph 14 Jan '26

14 Jan '26

Hello guys, We are using Ceph Nautilus, and we are facing an unexpected situation. We see a huge list of removed_snaps, and they do not seem to be cleaned. We also see increasing use of CPUs in Ceph monitors (we have thousands of these elements). However, we are not sure if this situation is related. In the logs, we noticed the following message, which seems to be related to this situation: ``` got unmanaged-snap pool op from entity with insufficient privileges. message: pool_op ``` Has anyone faced similar situations with Ceph and RBD snapshots management?

1 0

Re: Downgrading from Tentacle to Squid
by Redouane Kachach 14 Jan '26

14 Jan '26

Hi Michael, Unfortunately downgrade is not supported between two Ceph major releases. You will need to re-install the old release. Best, Redouane. On Tue, Jan 13, 2026 at 3:48 PM Michael Worsham via ceph-users < ceph-users(a)ceph.io> wrote: > Currently in a test-lab / sandbox environment, I am running the latest > version of Ceph Tentacle release (v20.2.0) in a virtualized environment. If > I wanted to downgrade back to Squid (v19.2.3), what steps would I go about > doing that? I am not worried about any of the data in the environment as > its strictly for testing purposes, patch assessment for OS and Ceph, etc. > > ubuntu@ceph-mon1:~$ sudo ceph versions > { > "mon": { > "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) > tentacle (stable - RelWithDebInfo)": 2 > }, > "mgr": { > "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) > tentacle (stable - RelWithDebInfo)": 2 > }, > "osd": { > "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) > tentacle (stable - RelWithDebInfo)": 3 > }, > "mds": { > "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) > tentacle (stable - RelWithDebInfo)": 1 > }, > "rgw": { > "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) > tentacle (stable - RelWithDebInfo)": 2 > }, > "overall": { > "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) > tentacle (stable - RelWithDebInfo)": 10 > } > } > > Environment is running Ubuntu v22.04 and Ceph was deployed via cephadm. > > -- Michael > > This message and its attachments are from Data Dimensions and are intended > only for the use of the individual or entity to which it is addressed, and > may contain information that is privileged, confidential, and exempt from > disclosure under applicable law. If the reader of this message is not the > intended recipient, or the employee or agent responsible for delivering the > message to the intended recipient, you are hereby notified that any > dissemination, distribution, or copying of this communication is strictly > prohibited. If you have received this communication in error, please notify > the sender immediately and permanently delete the original email and > destroy any copies or printouts of this email as well as any attachments. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Ceph Tech Talks 2026
by Anthony Middleton 13 Jan '26

13 Jan '26

Hey Ceph Friends, Exciting news for tomorrow:* Ceph Tech Talks are happening*! Be sure not to miss our very first session of 2026: “*Running Teuthology Outside of the Sepia Lab*” with the wonderful Deepshikha Singh. *Join us on Jan 14, 2026, at 8:30 am PDT* via Jitsi (virtual) at https://meet.jit.si/ceph-tech-talks. And don’t worry if you can’t make it live, the session will be recorded and shared on YouTube! See you tomorrow. -- Anthony Middleton Community Manager, Ceph Foundation community-manager(a)ceph.io LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: Advice on managing adopted OSDs
by Chris Palmer 13 Jan '26

13 Jan '26

Ok, that was the conclusion that I was starting to come to. Thanks so much for the notes - I'll have a careful look through them. I found this note in Service Management\Declarative State that reinforces that... /The “osd” service used to track OSDs that are not tied to any specific service spec is special and will always be marked unmanaged. Attempting to modify it with |ceph orch set-unmanaged| or |ceph orch set-managed| will result in a message |No service of name osd found. Check "ceph orch ls" for all known services|/ I also found a note in OSD Service\Creating New OSDs... /When deploying new OSDs with |cephadm|, ensure that the |ceph-osd| package is not already installed on the target host. If it is installed, conflicts may arise in the management and control of the OSD that may lead to errors or unexpected behavior./ ...so will remove all the packages first (except cephadm). It would be useful to have some definitive statements about managing OSDs in the Converting an existing cluster to cephadm doc, other than just the initial OSD adoption itself. Regards, Chris On 13/01/2026 17:45, Julien Laurenceau via ceph-users wrote: > Hello, > > To my knowledge there is no “adopt” or in-place migration procedure in > Squid. > > You must remove and redeploy OSDs to bring them under management. > > Migration Steps: > a. Identify unmanaged OSDs > ``` > ceph orch ls --service-type osd > ceph orch ps --daemon-type osd > ``` > or > ``` > ceph osd tree > # easy way to retrieve the device used by a specific osd : > ceph device ls > # ceph device ls |grep -v nvme | awk -F '.' '{print $NF}' | sort -n > ceph osd metadata ${osdid} |grep -e '"hostname"' -e '"devices"' > > listosd=$(ceph device ls |grep $h |grep osd | awk -F '.' '{print $NF}' > | awk '{print $1}'| sort -n) > osd_devices=$(ceph device ls |grep $h |grep osd | awk -F "${h}:" > '{print $NF}' | awk '{print $1}'| sort -n) > for osdid in ${listosd}; do ceph osd metadata ${osdid} |grep -e > '"hostname"' -e '"devices"'; done > ``` > b. Mark OSDs out and wait for healthy > ``` > ceph osd out ${osdid} > for osdid in ${listosd}; do ceph osd out ${osdid}; done > # Wait for until no object degraded and no Reduced data availability > ``` > Wait until there are no degraded objects. > c. Mark the OSD as “down” > ``` > ceph osd down ${osdid} > for osdid in ${listosd}; do ceph osd down ${osdid}; done > ``` > d. Stop the OSD daemon on the host > On the host where the OSD runs, stop the OSD service: > ``` > systemctl list-units --no-legend --no-pager "ceph-*(a)osd.*" > systemctl stop ceph-${clusterid}(a)osd.${osdid} > for osdid in ${listosd}; do systemctl stop > ceph-${clusterid}(a)osd.${osdid}; done > ``` > e. Remove the OSD > ``` > ceph osd purge ${osdid} --yes-i-really-mean-it > ceph orch daemon rm osd.$osdid --force > for osdid in ${listosd}; do ceph osd purge ${osdid} > --yes-i-really-mean-it; done > for osdid in ${listosd}; do ceph orch daemon rm osd.$osdid --force; done > ``` > f. Clean the device on the host > ``` > # osd_devices="nvme0n1 nvme1n1 nvme2n1 nvme3n1 nvme4n1 nvme5n1 nvme6n1 > nvme7n1" > # osd_devices="sda sdb sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn > sdo sdp sdq sdr sds sdt sdu sdv sdw sdx" > ceph-volume lvm zap /dev/sdX --destroy > # or > sgdisk --zap-all /dev/sdX > for d in ${osd_devices}; do sgdisk --zap-all /dev/${d}; done > > if zap it is not sufficient NUKE THE DRIVE : > for dm in $(dmsetup ls | grep '^ceph--' | awk '{print $1}'); do > dmsetup remove ${dm}; done > for vg in $(vgdisplay -s | grep '"ceph-' |awk '{print $1}' | tr -d > '"') ; do vgremove --force $vg; done > for d in ${osd_devices}; do echo "nuke $d"; pvremove /dev/${d}; > wipefs -af /dev/${d}; sfdisk --delete /dev/${d}; partprobe /dev/${d}; > done > ``` > g. Create and apply a DriveGroup YAML for the host/device > > (see section above) > ``` > cat << EOF >ceph-osd-$h.yaml > service_type: osd > service_id: osd-${h} > placement: > hosts: > - ${h} > data_devices: > paths: > EOF > for dev in $(echo ${osd_devices} | tr -s ',' ' '); do > echo " - /dev/${dev}" >> ceph-osd-$h.yaml > done > ceph orch apply -i ceph-osd-${h}.yaml > ceph orch ls --service-type osd > # ceph orch rm osd.osd-$h > > # apply device-class if needed : > mydevclass=nvme-all > listosd=$(ceph device ls |grep $h |grep osd | awk -F '.' '{print $NF}' > | awk '{print $1}'| sort -n) > for osdid in ${listosd}; do ceph osd crush set-device-class > ${mydevclass} ${osdid}; done > ``` > > h. Verify > ``` > ceph orch ps --daemon-type osd > ceph osd tree > ceph -s > ``` > > > On 1/13/26 17:00, Kirby Haze via ceph-users wrote: >> There should be a >> `orch set-managed` and `orch set-unmanaged` >> so doing `ceph orch set-managed osd` should work, then it should >> reflect in >> the yaml shortly after. >> >> - Kirby >> >> On Tue, Jan 13, 2026 at 1:24 AM Chris Palmer via ceph-users < >> ceph-users(a)ceph.io> wrote: >> >>> I would appreciate advice on the "best" way to manage adopted OSDs. >>> >>> This is a test squid 19.2.3 cluster, originally installed using >>> centos 9 >>> packages. I am testing the procedure for adopting, then will use >>> cephadm >>> to update to tentacle. Once I am happy with the whole process I will >>> repeat on a small production cluster. >>> >>> I have successfully adopted mon, mgr, osd, rgw & mds as described in >>> the >>> docs. That left rgw & mds managed, but mon, mgr & osds unmanaged. >>> Setting the mon & mgr services managed was easy by tweaking the yml and >>> applying. But I am unclear how to best handle osds. That leaves me in >>> this state: >>> >>> [root@c4 ~]# ceph orch ls >>> NAME PORTS RUNNING REFRESHED AGE PLACEMENT >>> mds.cephfs 2/2 51s ago 21h c1;c2 >>> mgr 3/3 52s ago 21h c1;c2;c4 >>> mon 3/3 51s ago 21h c1;c2;c3 >>> osd 12 51s ago - <unmanaged> >>> rgw.rgw ?:80 2/2 51s ago 21h c1;c2 >>> >>> >>> The exported yml had the following for osds, which had to be commented >>> out as it would not parse (missing placement, missing data_devices >>> etc): >>> >>> service_type: osd >>> service_name: osd >>> unmanaged: true >>> spec: >>> filter_logic: AND >>> objectstore: bluestore >>> >>> >>> My questions are: >>> >>> * With the OSDs unmanaged, what functionality am I missing? In >>> particular, things like using cephadm to replace a failed disk... >>> * These are only small clusters, so I am happy to have the OSDs >>> enumerated in the yml, and not have ceph try to find and >>> automatically use available disks. Is it possible to have the full >>> functionality except from automatically grabbing available disks? >>> * How can I generate yml that reflects the current state of the >>> OSDs? >>> We currently have pools (per application) for NVMe, SSD, and HDD >>> with shared NVMe DB/WAL. >>> * I can't find documentation for transitioning OSDs from >>> unmanaged to >>> managed. Have I just overlooked it? >>> * Is any part of this easier on tentacle than squid? If so I can >>> upgrade the package squid to tentacle before adopting. (But unless >>> there is a good reason, I would rather have familiarised myself >>> with >>> the cephadm upgrade process before starting any of this on the >>> production cluster...) >>> >>> Many thanks for any insights. >>> >>> Chris >>> _______________________________________________ >>> ceph-users mailing list --ceph-users(a)ceph.io >>> To unsubscribe send an email toceph-users-leave(a)ceph.io >>> >> _______________________________________________ >> ceph-users mailing list --ceph-users(a)ceph.io >> To unsubscribe send an email toceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Advice on managing adopted OSDs
by Julien Laurenceau 13 Jan '26

13 Jan '26

Hello, To my knowledge there is no “adopt” or in-place migration procedure in Squid. You must remove and redeploy OSDs to bring them under management. Migration Steps: a. Identify unmanaged OSDs ``` ceph orch ls --service-type osd ceph orch ps --daemon-type osd ``` or ``` ceph osd tree # easy way to retrieve the device used by a specific osd : ceph device ls # ceph device ls |grep -v nvme | awk -F '.' '{print $NF}' | sort -n ceph osd metadata ${osdid} |grep -e '"hostname"' -e '"devices"' listosd=$(ceph device ls |grep $h |grep osd | awk -F '.' '{print $NF}' | awk '{print $1}'| sort -n) osd_devices=$(ceph device ls |grep $h |grep osd | awk -F "${h}:" '{print $NF}' | awk '{print $1}'| sort -n) for osdid in ${listosd}; do ceph osd metadata ${osdid} |grep -e '"hostname"' -e '"devices"'; done ``` b. Mark OSDs out and wait for healthy ``` ceph osd out ${osdid} for osdid in ${listosd}; do ceph osd out ${osdid}; done # Wait for until no object degraded and no Reduced data availability ``` Wait until there are no degraded objects. c. Mark the OSD as “down” ``` ceph osd down ${osdid} for osdid in ${listosd}; do ceph osd down ${osdid}; done ``` d. Stop the OSD daemon on the host On the host where the OSD runs, stop the OSD service: ``` systemctl list-units --no-legend --no-pager "ceph-*(a)osd.*" systemctl stop ceph-${clusterid}(a)osd.${osdid} for osdid in ${listosd}; do systemctl stop ceph-${clusterid}(a)osd.${osdid}; done ``` e. Remove the OSD ``` ceph osd purge ${osdid} --yes-i-really-mean-it ceph orch daemon rm osd.$osdid --force for osdid in ${listosd}; do ceph osd purge ${osdid} --yes-i-really-mean-it; done for osdid in ${listosd}; do ceph orch daemon rm osd.$osdid --force; done ``` f. Clean the device on the host ``` # osd_devices="nvme0n1 nvme1n1 nvme2n1 nvme3n1 nvme4n1 nvme5n1 nvme6n1 nvme7n1" # osd_devices="sda sdb sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn sdo sdp sdq sdr sds sdt sdu sdv sdw sdx" ceph-volume lvm zap /dev/sdX --destroy # or sgdisk --zap-all /dev/sdX for d in ${osd_devices}; do sgdisk --zap-all /dev/${d}; done if zap it is not sufficient NUKE THE DRIVE : for dm in $(dmsetup ls | grep '^ceph--' | awk '{print $1}'); do dmsetup remove ${dm}; done for vg in $(vgdisplay -s | grep '"ceph-' |awk '{print $1}' | tr -d '"') ; do vgremove --force $vg; done for d in ${osd_devices}; do echo "nuke $d"; pvremove /dev/${d}; wipefs -af /dev/${d}; sfdisk --delete /dev/${d}; partprobe /dev/${d}; done ``` g. Create and apply a DriveGroup YAML for the host/device (see section above) ``` cat << EOF >ceph-osd-$h.yaml service_type: osd service_id: osd-${h} placement: hosts: - ${h} data_devices: paths: EOF for dev in $(echo ${osd_devices} | tr -s ',' ' '); do echo " - /dev/${dev}" >> ceph-osd-$h.yaml done ceph orch apply -i ceph-osd-${h}.yaml ceph orch ls --service-type osd # ceph orch rm osd.osd-$h # apply device-class if needed : mydevclass=nvme-all listosd=$(ceph device ls |grep $h |grep osd | awk -F '.' '{print $NF}' | awk '{print $1}'| sort -n) for osdid in ${listosd}; do ceph osd crush set-device-class ${mydevclass} ${osdid}; done ``` h. Verify ``` ceph orch ps --daemon-type osd ceph osd tree ceph -s ``` On 1/13/26 17:00, Kirby Haze via ceph-users wrote: > There should be a > `orch set-managed` and `orch set-unmanaged` > so doing `ceph orch set-managed osd` should work, then it should reflect in > the yaml shortly after. > > - Kirby > > On Tue, Jan 13, 2026 at 1:24 AM Chris Palmer via ceph-users < > ceph-users(a)ceph.io> wrote: > >> I would appreciate advice on the "best" way to manage adopted OSDs. >> >> This is a test squid 19.2.3 cluster, originally installed using centos 9 >> packages. I am testing the procedure for adopting, then will use cephadm >> to update to tentacle. Once I am happy with the whole process I will >> repeat on a small production cluster. >> >> I have successfully adopted mon, mgr, osd, rgw & mds as described in the >> docs. That left rgw & mds managed, but mon, mgr & osds unmanaged. >> Setting the mon & mgr services managed was easy by tweaking the yml and >> applying. But I am unclear how to best handle osds. That leaves me in >> this state: >> >> [root@c4 ~]# ceph orch ls >> NAME PORTS RUNNING REFRESHED AGE PLACEMENT >> mds.cephfs 2/2 51s ago 21h c1;c2 >> mgr 3/3 52s ago 21h c1;c2;c4 >> mon 3/3 51s ago 21h c1;c2;c3 >> osd 12 51s ago - <unmanaged> >> rgw.rgw ?:80 2/2 51s ago 21h c1;c2 >> >> >> The exported yml had the following for osds, which had to be commented >> out as it would not parse (missing placement, missing data_devices etc): >> >> service_type: osd >> service_name: osd >> unmanaged: true >> spec: >> filter_logic: AND >> objectstore: bluestore >> >> >> My questions are: >> >> * With the OSDs unmanaged, what functionality am I missing? In >> particular, things like using cephadm to replace a failed disk... >> * These are only small clusters, so I am happy to have the OSDs >> enumerated in the yml, and not have ceph try to find and >> automatically use available disks. Is it possible to have the full >> functionality except from automatically grabbing available disks? >> * How can I generate yml that reflects the current state of the OSDs? >> We currently have pools (per application) for NVMe, SSD, and HDD >> with shared NVMe DB/WAL. >> * I can't find documentation for transitioning OSDs from unmanaged to >> managed. Have I just overlooked it? >> * Is any part of this easier on tentacle than squid? If so I can >> upgrade the package squid to tentacle before adopting. (But unless >> there is a good reason, I would rather have familiarised myself with >> the cephadm upgrade process before starting any of this on the >> production cluster...) >> >> Many thanks for any insights. >> >> Chris >> _______________________________________________ >> ceph-users mailing list --ceph-users(a)ceph.io >> To unsubscribe send an email toceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list --ceph-users(a)ceph.io > To unsubscribe send an email toceph-users-leave(a)ceph.io

1 0

Re: Advice on managing adopted OSDs
by Kirby Haze 13 Jan '26

13 Jan '26

There should be a `orch set-managed` and `orch set-unmanaged` so doing `ceph orch set-managed osd` should work, then it should reflect in the yaml shortly after. - Kirby On Tue, Jan 13, 2026 at 1:24 AM Chris Palmer via ceph-users < ceph-users(a)ceph.io> wrote: > I would appreciate advice on the "best" way to manage adopted OSDs. > > This is a test squid 19.2.3 cluster, originally installed using centos 9 > packages. I am testing the procedure for adopting, then will use cephadm > to update to tentacle. Once I am happy with the whole process I will > repeat on a small production cluster. > > I have successfully adopted mon, mgr, osd, rgw & mds as described in the > docs. That left rgw & mds managed, but mon, mgr & osds unmanaged. > Setting the mon & mgr services managed was easy by tweaking the yml and > applying. But I am unclear how to best handle osds. That leaves me in > this state: > > [root@c4 ~]# ceph orch ls > NAME PORTS RUNNING REFRESHED AGE PLACEMENT > mds.cephfs 2/2 51s ago 21h c1;c2 > mgr 3/3 52s ago 21h c1;c2;c4 > mon 3/3 51s ago 21h c1;c2;c3 > osd 12 51s ago - <unmanaged> > rgw.rgw ?:80 2/2 51s ago 21h c1;c2 > > > The exported yml had the following for osds, which had to be commented > out as it would not parse (missing placement, missing data_devices etc): > > service_type: osd > service_name: osd > unmanaged: true > spec: > filter_logic: AND > objectstore: bluestore > > > My questions are: > > * With the OSDs unmanaged, what functionality am I missing? In > particular, things like using cephadm to replace a failed disk... > * These are only small clusters, so I am happy to have the OSDs > enumerated in the yml, and not have ceph try to find and > automatically use available disks. Is it possible to have the full > functionality except from automatically grabbing available disks? > * How can I generate yml that reflects the current state of the OSDs? > We currently have pools (per application) for NVMe, SSD, and HDD > with shared NVMe DB/WAL. > * I can't find documentation for transitioning OSDs from unmanaged to > managed. Have I just overlooked it? > * Is any part of this easier on tentacle than squid? If so I can > upgrade the package squid to tentacle before adopting. (But unless > there is a good reason, I would rather have familiarised myself with > the cephadm upgrade process before starting any of this on the > production cluster...) > > Many thanks for any insights. > > Chris > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Downgrading from Tentacle to Squid
by Eugen Block 13 Jan '26

13 Jan '26

There's no downgrade supported in Ceph. Although it might work for some daemons, for example mgr daemons, it's not safe in general. Zitat von Michael Worsham via ceph-users <ceph-users(a)ceph.io>: > Currently in a test-lab / sandbox environment, I am running the > latest version of Ceph Tentacle release (v20.2.0) in a virtualized > environment. If I wanted to downgrade back to Squid (v19.2.3), what > steps would I go about doing that? I am not worried about any of the > data in the environment as its strictly for testing purposes, patch > assessment for OS and Ceph, etc. > > ubuntu@ceph-mon1:~$ sudo ceph versions > { > "mon": { > "ceph version 20.2.0 > (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - > RelWithDebInfo)": 2 > }, > "mgr": { > "ceph version 20.2.0 > (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - > RelWithDebInfo)": 2 > }, > "osd": { > "ceph version 20.2.0 > (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - > RelWithDebInfo)": 3 > }, > "mds": { > "ceph version 20.2.0 > (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - > RelWithDebInfo)": 1 > }, > "rgw": { > "ceph version 20.2.0 > (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - > RelWithDebInfo)": 2 > }, > "overall": { > "ceph version 20.2.0 > (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - > RelWithDebInfo)": 10 > } > } > > Environment is running Ubuntu v22.04 and Ceph was deployed via cephadm. > > -- Michael > > This message and its attachments are from Data Dimensions and are > intended only for the use of the individual or entity to which it is > addressed, and may contain information that is privileged, > confidential, and exempt from disclosure under applicable law. If > the reader of this message is not the intended recipient, or the > employee or agent responsible for delivering the message to the > intended recipient, you are hereby notified that any dissemination, > distribution, or copying of this communication is strictly > prohibited. If you have received this communication in error, please > notify the sender immediately and permanently delete the original > email and destroy any copies or printouts of this email as well as > any attachments. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: NVMe-first crush rule
by Eugen Block 13 Jan '26

13 Jan '26

Hi, first one note: mixing device classes as you plan will only be beneficial for reads, those will be faster. But writes still need to be acked from all replicas before acked to the client. For the rest see responses inline. Zitat von Jan Kasprzak via ceph-users <ceph-users(a)ceph.io>: > Hello, Ceph users, > > how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs, > with all replicas on different hosts? > > One of my clusters has been upgraded to two brand new ~6TB NVMe disks > together with previously used two 12TB HDDs per host. > Right now I have two HDD-based OSDs per host, each with ~200GB partition > on NVMe for metadata. The rest of NVMe is free, so I can now create > two additional ~6TB NVMe OSDs per host. > > My question is how to create a CRUSH rule which would have three replicas > on three different hosts, with one of these replicas on NVMe, and the > remaining two on HDD. > > In the docs here: > https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-ru… > there is an example of a SSD-first rule, which is close to what I would like > to do, but it seems the SSD replica in that rule can end up on the same host > as one of the HDD replicas. How can I avoid that? I assume you're referring to this rule: step take default class ssd step chooseleaf firstn 1 type host step emit step take default class hdd step chooseleaf firstn 0 type host step emit The first 'step chooseleaf firstn 1 type host' advises CRUSH to select an OSD on one (exactly 1) host within the SSD class. Then it has to select the remaining hosts from the HDD class ('firstn 0' means the remaining number of replicas to distribute). Since you specify your failure domain to "host" (by using "type host" within the crush rule), all replicas of the same PG will be distributed across different hosts. > And a noob question -- how can I create a crush rule from the > "rule something { ... }" text in that example? If you need some advanced rule, you're going to have to use crushtool on the extracted crushmap. I usually start by creating a rule via CLI, then extract the crushmap, decompile it, make changes to it, compile and test it before injecting it back into the cluster. For example: # create simple rule ceph osd crush rule create-replicated test-rack-rule default rack hdd # dump crushmap ceph osd getcrushmap -o crushmap.bin # decompile crushtool -d crushmap.bin -o crushmap.txt # edit rule with any text editor # compile crushtool -c crushmap.bin -o crushmap.new.bin # test crushtool -i crushmap.new.bin --test --num-rep 3 --rule <RULE ID> --show-bad-mappings crushtool -i crushmap.new.bin --test --num-rep 3 --rule <RULE ID> --show-mappings | less If --show-bad-mappings does show any output, you made a mistake somewhere. With --show-mappings you can check the distribution of the PGs, the result is an acting set of OSDs, e.g.: CRUSH rule 5 x 0 [6,3,7] Then you can check the OSD's device class: for i in {6,3,7}; do ceph osd metadata $i | jq -r '.bluestore_bdev_type'; done hdd hdd hdd > Thanks! > > -Yenya > > -- > | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | > | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | > I don't like Python; its lack of inline, anonymous, multi-statement > functions makes me sad. --Eric Wastl > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 1

Downgrading from Tentacle to Squid
by Michael Worsham 13 Jan '26

13 Jan '26

Currently in a test-lab / sandbox environment, I am running the latest version of Ceph Tentacle release (v20.2.0) in a virtualized environment. If I wanted to downgrade back to Squid (v19.2.3), what steps would I go about doing that? I am not worried about any of the data in the environment as its strictly for testing purposes, patch assessment for OS and Ceph, etc. ubuntu@ceph-mon1:~$ sudo ceph versions { "mon": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 2 }, "mgr": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 2 }, "osd": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 3 }, "mds": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 1 }, "rgw": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 2 }, "overall": { "ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)": 10 } } Environment is running Ubuntu v22.04 and Ceph was deployed via cephadm. -- Michael This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

1 0

Re: NVMe-first crush rule
by Reed Dier 13 Jan '26

13 Jan '26

So I have a bit of an ugly setup because of history... That said, this is what I/we call a hybrid pool with the first/primary PG on SSD, and secondary/tertiary replicas of the PG on HDD. My SSD and HDD OSD's are in physically separate hosts, no intermingling of media, which may make things more tricky. And because this was all stood up ages ago before device classes were in ceph, the SSD hosts are in a totally separate root in crush. So in my scenario, I have it take from a host in the ssd root. And for the second take, it chooses two hosts from the hdd root. > { > "rule_id": {{ HYBRID RULE ID }}, > "rule_name": "{{ HYBRID RULE NAME }}", > "type": 1, > "steps": [ > { > "op": "take", > "item": {{ SSD ROOT NUM }}, > "item_name": "{{ SSD ROOT }}" > }, > { > "op": "chooseleaf_firstn", > "num": 1, > "type": "{{ SSD FAILURE DOMAIN }}" > }, > { > "op": "emit" > }, > { > "op": "take", > "item": -1, > "item_name": "{{ HDD ROOT }}" > }, > { > "op": "chooseleaf_firstn", > "num": -1, > "type": "{{ HDD FAILURE DOMAIN }}" > }, > { > "op": "emit" > } > ] > }, Since these are intermingled device classes intra-host. It might just be easiest to create a small pool with a few PGs, and then look to see what the up PG set is > ceph pg dump | grep ^${newPoolID}.0 | awk '{print $17}' And compare the array of OSDs to see if the OSDs are actually distributed across different hosts or if it places two on the same host. Reed > On Jan 12, 2026, at 1:53 PM, Jan Kasprzak via ceph-users <ceph-users(a)ceph.io> wrote: > > Hello, Ceph users, > > how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs, > with all replicas on different hosts? > > One of my clusters has been upgraded to two brand new ~6TB NVMe disks > together with previously used two 12TB HDDs per host. > Right now I have two HDD-based OSDs per host, each with ~200GB partition > on NVMe for metadata. The rest of NVMe is free, so I can now create > two additional ~6TB NVMe OSDs per host. > > My question is how to create a CRUSH rule which would have three replicas > on three different hosts, with one of these replicas on NVMe, and the > remaining two on HDD. > > In the docs here: > https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-ru… > there is an example of a SSD-first rule, which is close to what I would like > to do, but it seems the SSD replica in that rule can end up on the same host > as one of the HDD replicas. How can I avoid that? > > And a noob question -- how can I create a crush rule from the > "rule something { ... }" text in that example? > > Thanks! > > -Yenya > > -- > | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | > | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | > I don't like Python; its lack of inline, anonymous, multi-statement > functions makes me sad. --Eric Wastl > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [Nautilus][Upgrade] OS migration and Ceph upgrade best practices?
by Janne Johansson 13 Jan '26

13 Jan '26

> We are planning to migrate and upgrade an existing production Ceph cluster and > would like to confirm the recommended approach. > > Current setup: > - OS: CentOS 7.9 > - Ceph: Nautilus 14.2.22 > - Deployment: bare metal > - Services: OSD, MON and RGW > > Goals: > - Move away from CentOS 7 (EOL) > - Upgrade Ceph to a supported release > - Use Ubuntu LTS as the target OS > > Our current plan is: > 1. Introduce new nodes running Ubuntu 22.04 > 2. Gradually replace CentOS 7 nodes while keeping Ceph at Nautilus > 3. After OS migration is complete, upgrade Ceph step by step > (Nautilus -> Octopus -> Pacific -> Quincy / Reef) > > We would like to confirm: > 1. Is node replacement generally preferred over in-place OS upgrades for > production Ceph clusters? > 2. Is Ubuntu 22.04 currently the recommended OS for production deployments? > 3. Is Ubuntu 24.04 still considered early for Ceph production use? > > Any feedback or experience would be appreciated. We have done the same, but in our case we did: Centos7 -> upgrade ceph to Octopus (latest one can find for C7) Switch OS one host at a time to Ubuntu 20.x (can install octopus) Upgrade ubuntu 20.x hosts to Quincy or Reef Upgrade ubuntu OS to 22.x (or 24.x) Upgrade to Squid,Tentacle or wherever you plan to end up. Takes a long time on the calendar but works. -- May the most significant bit of your life be positive.

1 0

Re: Advice on managing adopted OSDs
by Eugen Block 13 Jan '26

13 Jan '26

Hi, that is expected since the orchestrator didn't create those OSDs, just adopted them. It won't be different in any other release. You can create a new spec file that reflects your current setup, the docs contain a couple of examples, here's a simple one: ---snip--- # cat osd-specs.yaml service_type: osd service_id: osd-hdd-ssd service_name: osd.osd-hdd-ssd placement: hosts: - host1 - host2 - host3 spec: data_devices: rotational: 1 size: 15G db_devices: rotational: 0 size: 13G:16G filter_logic: AND objectstore: bluestore ---snip--- Note that the above spec is a different one than in the following example. If you're sure that your new spec will match, you can apply it, but nothing will happen to the existing OSDs. If there are unused block devices that would apply to the specs, the orchestrator would try to deploy new OSDs. Once you have a different osd spec in place, you can set the appropriate string in the unit.meta file of the OSD: # grep service /var/lib/ceph/FSID/osd.0/unit.meta "service_name": "osd.osd-hdd-ssd", After a couple of minutes, the orchestrator will refresh its view on OSDs (transition from the unmanaged service to the managed one): # cat osd-standalone.yaml service_type: osd service_name: osd.standalone service_id: osd.standalone placement: label: osd spec: data_devices: rotational: 1 filter_logic: AND objectstore: bluestore After chaning the first OSD's unit.meta file NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd 5 2s ago - <unmanaged> osd.osd.standalone 1 2s ago 5m label:osd As soon as all OSDs have been transitioned, the unmanaged service will vanish: NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd.osd.standalone 6 0s ago 8m label:osd I'm not sure if there's a smoother way, but that's how I've done it in the past because it's relatively quick (although it's been a while since I last adopted a prod cluster to cephadm). Regards, Eugen Zitat von Chris Palmer via ceph-users <ceph-users(a)ceph.io>: > I would appreciate advice on the "best" way to manage adopted OSDs. > > This is a test squid 19.2.3 cluster, originally installed using > centos 9 packages. I am testing the procedure for adopting, then > will use cephadm to update to tentacle. Once I am happy with the > whole process I will repeat on a small production cluster. > > I have successfully adopted mon, mgr, osd, rgw & mds as described in > the docs. That left rgw & mds managed, but mon, mgr & osds > unmanaged. Setting the mon & mgr services managed was easy by > tweaking the yml and applying. But I am unclear how to best handle > osds. That leaves me in this state: > > [root@c4 ~]# ceph orch ls > NAME PORTS RUNNING REFRESHED AGE PLACEMENT > mds.cephfs 2/2 51s ago 21h c1;c2 > mgr 3/3 52s ago 21h c1;c2;c4 > mon 3/3 51s ago 21h c1;c2;c3 > osd 12 51s ago - <unmanaged> > rgw.rgw ?:80 2/2 51s ago 21h c1;c2 > > > The exported yml had the following for osds, which had to be > commented out as it would not parse (missing placement, missing > data_devices etc): > > service_type: osd > service_name: osd > unmanaged: true > spec: > filter_logic: AND > objectstore: bluestore > > > My questions are: > > * With the OSDs unmanaged, what functionality am I missing? In > particular, things like using cephadm to replace a failed disk... > * These are only small clusters, so I am happy to have the OSDs > enumerated in the yml, and not have ceph try to find and > automatically use available disks. Is it possible to have the full > functionality except from automatically grabbing available disks? > * How can I generate yml that reflects the current state of the OSDs? > We currently have pools (per application) for NVMe, SSD, and HDD > with shared NVMe DB/WAL. > * I can't find documentation for transitioning OSDs from unmanaged to > managed. Have I just overlooked it? > * Is any part of this easier on tentacle than squid? If so I can > upgrade the package squid to tentacle before adopting. (But unless > there is a good reason, I would rather have familiarised myself with > the cephadm upgrade process before starting any of this on the > production cluster...) > > Many thanks for any insights. > > Chris > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Advice on managing adopted OSDs
by Chris Palmer 13 Jan '26

13 Jan '26

I would appreciate advice on the "best" way to manage adopted OSDs. This is a test squid 19.2.3 cluster, originally installed using centos 9 packages. I am testing the procedure for adopting, then will use cephadm to update to tentacle. Once I am happy with the whole process I will repeat on a small production cluster. I have successfully adopted mon, mgr, osd, rgw & mds as described in the docs. That left rgw & mds managed, but mon, mgr & osds unmanaged. Setting the mon & mgr services managed was easy by tweaking the yml and applying. But I am unclear how to best handle osds. That leaves me in this state: [root@c4 ~]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT mds.cephfs 2/2 51s ago 21h c1;c2 mgr 3/3 52s ago 21h c1;c2;c4 mon 3/3 51s ago 21h c1;c2;c3 osd 12 51s ago - <unmanaged> rgw.rgw ?:80 2/2 51s ago 21h c1;c2 The exported yml had the following for osds, which had to be commented out as it would not parse (missing placement, missing data_devices etc): service_type: osd service_name: osd unmanaged: true spec: filter_logic: AND objectstore: bluestore My questions are: * With the OSDs unmanaged, what functionality am I missing? In particular, things like using cephadm to replace a failed disk... * These are only small clusters, so I am happy to have the OSDs enumerated in the yml, and not have ceph try to find and automatically use available disks. Is it possible to have the full functionality except from automatically grabbing available disks? * How can I generate yml that reflects the current state of the OSDs? We currently have pools (per application) for NVMe, SSD, and HDD with shared NVMe DB/WAL. * I can't find documentation for transitioning OSDs from unmanaged to managed. Have I just overlooked it? * Is any part of this easier on tentacle than squid? If so I can upgrade the package squid to tentacle before adopting. (But unless there is a good reason, I would rather have familiarised myself with the cephadm upgrade process before starting any of this on the production cluster...) Many thanks for any insights. Chris

1 0

Re: Change placement omf mds-daemons
by Nizamudeen A 13 Jan '26

13 Jan '26

Hi Johan, Thanks for raising this issue. We tested it yesterday and figured out that its an issue in 19.2.3 but fixed in later versions and it was a regression in that release alone. We opened a tracker: https://tracker.ceph.com/issues/74376 and a fix as well: https://github.com/ceph/ceph/pull/66884. We will soon get it merged in that branch. Will be fixed in the next release. I saw you also opened a separate tracker later, I'll probably close it as the duplicate of the one we created. Sorry to inform you late on this. but thanks for reporting. Regards, Nizam On Sat, Jan 10, 2026 at 10:57 PM Johan via ceph-users <ceph-users(a)ceph.io> wrote: > Ceph 19.2.3 using cephadm/docker > > I have tried to change placement of a mds to label using the dashboard. > When I do, ceph creates a new mds-service, mds.label, instead of > changing the original one. > > Im certain I have done this, using the dashboard, before on earlier > ceph-versions. > > I can without problem change the placement using the command-line > ceph orch apply mds filesystem --placement="label:mds" > > Anyone else seen this? Is this behaviour by the dashboard intentional or > a bug? > > /Johan > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- Nizamudeen A Sr. Software Engineer - IBM Partner Engineer IBM and Red Hat Ceph Storage Red Hat <https://www.redhat.com/> <https://www.redhat.com/>

1 0

Re: [Nautilus][Upgrade] OS migration and Ceph upgrade best practices?
by Anthony D'Atri 13 Jan '26

13 Jan '26

1 0

[Nautilus][Upgrade] OS migration and Ceph upgrade best practices?
by jie wang 12 Jan '26

12 Jan '26

Hi Ceph users, We are planning to migrate and upgrade an existing production Ceph cluster and would like to confirm the recommended approach. Current setup: - OS: CentOS 7.9 - Ceph: Nautilus 14.2.22 - Deployment: bare metal - Services: OSD, MON and RGW Goals: - Move away from CentOS 7 (EOL) - Upgrade Ceph to a supported release - Use Ubuntu LTS as the target OS Our current plan is: 1. Introduce new nodes running Ubuntu 22.04 2. Gradually replace CentOS 7 nodes while keeping Ceph at Nautilus 3. After OS migration is complete, upgrade Ceph step by step (Nautilus -> Octopus -> Pacific -> Quincy / Reef) We would like to confirm: 1. Is node replacement generally preferred over in-place OS upgrades for production Ceph clusters? 2. Is Ubuntu 22.04 currently the recommended OS for production deployments? 3. Is Ubuntu 24.04 still considered early for Ceph production use? Any feedback or experience would be appreciated. Thanks.

1 0

Re: CSC minutes 2026-01-12
by Gregory Farnum 12 Jan '26

12 Jan '26

On Mon, Jan 12, 2026 at 12:02 PM Patrick Donnelly via Dev <dev(a)ceph.io> wrote: > Hello, > > Today's CSC meeting minutes: > > * Status of Reef? Branch reports being locked, confounding docs > backports. From Ville "I was looking at a problem where docs for all > the EOL releases are missing the supposed warning admonition at the > top "This document is for an unsupported version of Ceph". Would have > wanted to get this fixed before Reef actual EOL." [aad] > + Patrick D. (me) sent an email about this separately to the dev list. > * (v18.2.8 QA): David still working on getting the lab on its feet to > run QA tests > + Postgres issue right now investigating > + Hopefully midweek it'll be operational > * Other Lab updates > + Folio + jenkins builders are being reracked > + Smithis next to be reracked/cabled > + http/https access to lab-extras repo is not accessible from the > lab due to security blocks by IBM This isn’t quite right and it’s important — http*s* acces is working fine; it’s just http port 80 we aren’t going to get. :) > - install-deps and ansible have been updated > * Ceph Community Manager Updates > + Teams webpage updated on Ceph.io - https://ceph.io/en/community/team/ > + CSC Election process document in progress. > - Seeking feedback on document from the Chair of Foundation > * CVE testing and testing releases with each Ceph distribution > + (present practice is to) use security list or maybe ceph-maintainers > to sync > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Red Hat Partner Engineer > IBM, Inc. > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io > >

1 0

CSC minutes 2026-01-12
by Patrick Donnelly 12 Jan '26

12 Jan '26

Hello, Today's CSC meeting minutes: * Status of Reef? Branch reports being locked, confounding docs backports. From Ville "I was looking at a problem where docs for all the EOL releases are missing the supposed warning admonition at the top "This document is for an unsupported version of Ceph". Would have wanted to get this fixed before Reef actual EOL." [aad] + Patrick D. (me) sent an email about this separately to the dev list. * (v18.2.8 QA): David still working on getting the lab on its feet to run QA tests + Postgres issue right now investigating + Hopefully midweek it'll be operational * Other Lab updates + Folio + jenkins builders are being reracked + Smithis next to be reracked/cabled + http/https access to lab-extras repo is not accessible from the lab due to security blocks by IBM - install-deps and ansible have been updated * Ceph Community Manager Updates + Teams webpage updated on Ceph.io - https://ceph.io/en/community/team/ + CSC Election process document in progress. - Seeking feedback on document from the Chair of Foundation * CVE testing and testing releases with each Ceph distribution + (present practice is to) use security list or maybe ceph-maintainers to sync -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

NVMe-first crush rule
by Jan Kasprzak 12 Jan '26

12 Jan '26

Hello, Ceph users, how can I create a CRUSH rule for one replica on NVMe and the rest on HDDs, with all replicas on different hosts? One of my clusters has been upgraded to two brand new ~6TB NVMe disks together with previously used two 12TB HDDs per host. Right now I have two HDD-based OSDs per host, each with ~200GB partition on NVMe for metadata. The rest of NVMe is free, so I can now create two additional ~6TB NVMe OSDs per host. My question is how to create a CRUSH rule which would have three replicas on three different hosts, with one of these replicas on NVMe, and the remaining two on HDD. In the docs here: https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-ru… there is an example of a SSD-first rule, which is close to what I would like to do, but it seems the SSD replica in that rule can end up on the same host as one of the HDD replicas. How can I avoid that? And a noob question -- how can I create a crush rule from the "rule something { ... }" text in that example? Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl

1 0

Re: Change placement omf mds-daemons
by Eugen Block 11 Jan '26

11 Jan '26

Yes, I can reproduce this on 19.2.3 and 19.2.1. On Tentacle it seems to work as expected though, the MDS service is changed to use label. The last Reef version (18.2.7) also just changes the service instead of creating a new one. So that seems to be a regression in Squid. I would recommend to create a tracker issue if there isn't one already. Zitat von Johan via ceph-users <ceph-users(a)ceph.io>: > Ceph 19.2.3 using cephadm/docker > > I have tried to change placement of a mds to label using the > dashboard. When I do, ceph creates a new mds-service, mds.label, > instead of changing the original one. > > Im certain I have done this, using the dashboard, before on earlier > ceph-versions. > > I can without problem change the placement using the command-line > ceph orch apply mds filesystem --placement="label:mds" > > Anyone else seen this? Is this behaviour by the dashboard > intentional or a bug? > > /Johan > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Change placement omf mds-daemons
by Johan 10 Jan '26

10 Jan '26

Ceph 19.2.3 using cephadm/docker I have tried to change placement of a mds to label using the dashboard. When I do, ceph creates a new mds-service, mds.label, instead of changing the original one. Im certain I have done this, using the dashboard, before on earlier ceph-versions. I can without problem change the placement using the command-line ceph orch apply mds filesystem --placement="label:mds" Anyone else seen this? Is this behaviour by the dashboard intentional or a bug? /Johan

1 0

Re: Question concerning selection of secondary OSDs
by Janne Johansson 10 Jan '26

10 Jan '26

> Furthermore, my expectation for a pool that has size=3 and min_size=2 is that for any given write, the primary OSD on nodes 1 or 2 will select a secondary OSD in node 1/2 respectively and one in node 3. Which would then lead me to believe that any client writing into the cluster from node 1 will only ever have the latency between node 1 and node 2 as an actual performance penalty because > > * client selects primary OSD on node 1 or node 2 > * primary OSD selects secondary OSDs and starts transfer in parallel > * Write to OSD with lower latency will finish much sooner than the one to the other OSD, leading to the write acknowledgement being sent to the client, because min_size=2 > > But that appears not to be the case. priority-affinity has a very slight impact, but the overall performance That is indeed not the case, the write will only be accepted as completed when all three replicas have ACKed the write. With your situation, it would be rather quick for a client to fill up any buffers on the fast OSD hosts so that the slow OSD host can't keep up, and you would be in a situation where the cluster "promises" to keep 3 copies but in reality you only have two, and if the client keeps on writing or changing data on the fast OSDs, you would be sure to have a really bad situation in case one of the two fast nodes restarts, since you now have a lot of PGs for which the slow OSD host has bad/old/stale data and the one remaining fast OSD host would give you repl=1 "safety" for those PGs. So ceph will wait until it has 3 ACKs before client gets the ACK. -- May the most significant bit of your life be positive.

1 0