July 2025 - ceph-users - lists.ceph.io

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

CephFS: no MDS does join the filesystem
by Robert Sander 24 Mar '26

24 Mar '26

Hi, we are having an issue at a customer site where a 3PB CephFS is in failed state. The cluster itself is unhealthy and awaits replacements disks: # ceph -s cluster: id: 28ca2bfa-d87e-11ed-83a3-1070fddda30f health: HEALTH_ERR 4 failed cephadm daemon(s) There are daemons running an older version of ceph 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged 8 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 46 pgs backfill_toofull Possible data damage: 4 pgs inconsistent Degraded data redundancy: 6427646/15858772167 objects degraded (0.041%), 8 pgs degraded, 8 pgs undersized 6 pool(s) nearfull (muted: OSDMAP_FLAGS OSD_SCRUB_ERRORS(2d) PG_NOT_DEEP_SCRUBBED PG_NOT_SCRUBBED) services: mon: 3 daemons, quorum sn01,sn03,sn02 (age 3w) mgr: sn03.crlpzh(active, since 33h), standbys: sn01.tegfya, sn02.mzvgcr mds: 18/19 daemons up, 1 standby osd: 181 osds: 174 up (since 4d), 172 in (since 4d); 206 remapped pgs flags nodeep-scrub data: volumes: 2/3 healthy, 1 recovering; 1 damaged pools: 12 pools, 3585 pgs objects: 1.93G objects, 1.3 PiB usage: 2.5 PiB used, 501 TiB / 3.0 PiB avail pgs: 6427646/15858772167 objects degraded (0.041%) 293845758/15858772167 objects misplaced (1.853%) 2844 active+clean 532 active+clean+scrubbing 147 active+remapped+backfill_wait 28 active+remapped+backfill_toofull 11 active+remapped+backfill_wait+backfill_toofull 10 active+remapped+backfilling 6 active+undersized+degraded+remapped+backfill_toofull 2 active+clean+inconsistent 1 active+clean+scrubbing+deep+inconsistent+repair 1 active+undersized+remapped+backfilling 1 active+undersized+degraded+remapped+backfilling 1 active+recovering+degraded+remapped 1 active+remapped+inconsistent+backfill_toofull io: recovery: 183 MiB/s, 312 objects/s The CephFS metadata pool is not affected by the inconsistent PGs. The MDSs have this line in their logfile: "Monitors have assigned me to become a standby." The filesystem is joinable: # ceph fs lsflags storage_cluster joinable allow_snaps allow_multimds_snaps refuse_client_session But no MDS joins: # ceph fs status storage_cluster - 0 clients =============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed POOL TYPE USED AVAIL cephfs_metadata metadata 490G 12.9T cephfs_data data 970T 54.1T shared_data data 1351T 22.5T STANDBY MDS storage_cluster.sn04.cbvzzu MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) Why? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

6 13

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Ceph orchestrator not refreshing device list
by Bob Gibson 14 Sep '25

14 Sep '25

Hi, We recently converted a legacy cluster running Quincy v17.2.7 to cephadm. The conversion went smoothly and left all osds unmanaged by the orchestrator as expected. We’re now in the process of converting the osds to be managed by the orchestrator. We successfully converted a few of them, but then the orchestrator somehow got confused. `ceph health detail` reports a “stray daemon” for the osd we’re trying to convert, and the orchestrator is unable to refresh its device list so it doesn’t see any available devices. From the perspective of the osd node, the osd has been wiped and is ready to be reinstalled. We’ve also rebooted the node for good measure. `ceph osd tree` shows that the osd has been destroyed, but the orchestrator won’t reinstall it because it thinks the device is still active. The orchestrator device information is stale, but we’re unable to refresh it. The usual recommended workaround of failing over the mgr hasn’t helped. We’ve also tried `ceph orch device ls —refresh` to no avail. In fact after running that command subsequent runs of `ceph orch device ls` produce no output until the mgr is failed over again. Is there a way to force the orchestrator to refresh its list of devices when in this state? If not, can anyone offer any suggestions on how to fix this problem? Cheers, /rjg P.S. Some additional information in case it’s helpful... We’re using the following command to replace existing devices so that they’re managed by the orchestrator: ``` ceph orch osd rm <osd> --replace —zap ``` and we’re currently stuck on osd 88. ``` ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.88 on host ceph-osd31 not managed by cephadm ``` `ceph osd tree` shows that the osd has been destroyed and is ready to be replaced: ``` ceph osd tree-from ceph-osd31 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -46 34.93088 host ceph-osd31 84 ssd 3.49309 osd.84 up 1.00000 1.00000 85 ssd 3.49309 osd.85 up 1.00000 1.00000 86 ssd 3.49309 osd.86 up 1.00000 1.00000 87 ssd 3.49309 osd.87 up 1.00000 1.00000 88 ssd 3.49309 osd.88 destroyed 0 1.00000 89 ssd 3.49309 osd.89 up 1.00000 1.00000 90 ssd 3.49309 osd.90 up 1.00000 1.00000 91 ssd 3.49309 osd.91 up 1.00000 1.00000 92 ssd 3.49309 osd.92 up 1.00000 1.00000 93 ssd 3.49309 osd.93 up 1.00000 1.00000 ``` The cephadm log shows a claim on node `ceph-osd31` for that osd: ``` 2024-09-25T14:15:45.699348-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims -> {'ceph-osd31': ['88']} 2024-09-25T14:15:45.699534-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} ``` `ceph orch device ls` shows that the device list isn’t refreshing: ``` ceph orch device ls ceph-osd31 HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS ceph-osd31 /dev/sdc ssd INTEL_SSDSC2KG038T8_PHYG039603PE3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdd ssd INTEL_SSDSC2KG038T8_PHYG039600AY3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sde ssd INTEL_SSDSC2KG038T8_PHYG039600CW3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdf ssd INTEL_SSDSC2KG038T8_PHYG039600CM3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdg ssd INTEL_SSDSC2KG038T8_PHYG039600UB3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdh ssd INTEL_SSDSC2KG038T8_PHYG039603753P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdi ssd INTEL_SSDSC2KG038T8_PHYG039603R63P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdj ssd INTEL_SSDSC2KG038TZ_PHYJ4011032M3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdk ssd INTEL_SSDSC2KG038TZ_PHYJ3234010J3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdl ssd INTEL_SSDSC2KG038T8_PHYG039603NS3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ``` `ceph node ls` thinks the osd still exists ``` ceph node ls osd | jq -r '."ceph-osd31"' [ 84, 85, 86, 87, 88, <— this shouldn’t exist 89, 90, 91, 92, 93 ] ``` Each osd node has 10x 3.8 TB ssd drives for osds. On `ceph-osd31`, cephadm doesn’t see osd.88 as expected: ``` cephadm ls --no-detail [ { "style": "cephadm:v1", "name": "osd.93", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.93" }, { "style": "cephadm:v1", "name": "osd.85", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.85" }, { "style": "cephadm:v1", "name": "osd.90", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.90" }, { "style": "cephadm:v1", "name": "osd.92", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.92" }, { "style": "cephadm:v1", "name": "osd.89", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.89" }, { "style": "cephadm:v1", "name": "osd.87", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.87" }, { "style": "cephadm:v1", "name": "osd.86", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.86" }, { "style": "cephadm:v1", "name": "osd.84", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.84" }, { "style": "cephadm:v1", "name": "osd.91", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.91" } ] ``` `lsblk` shows that `/dev/sdg` has been wiped. ``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 223.6G 0 disk |-sda1 8:1 0 94M 0 part `-sda2 8:2 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdb 8:16 0 223.6G 0 disk |-sdb1 8:17 0 94M 0 part `-sdb2 8:18 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdc 8:32 1 3.5T 0 disk `-ceph--03782b4c--9faa--49f5--b554--98e7b8515834-osd--block--ba272724--daa6--45f5--9f69--789cc0bda077 253:3 0 3.5T 0 lvm `-keCkP2-o6h8-jKkw-RKiD-UBFf-A8EL-JDJGPR 253:9 0 3.5T 0 crypt sdd 8:48 1 3.5T 0 disk `-ceph--c07907d8--4a75--4ba3--b5e1--2ebf49ecbdf6-osd--block--58d1d50d--6228--4e6f--9a52--2a305ba00700 253:7 0 3.5T 0 lvm `-WB8Mxn-qCHI-4T01-imiG-hNBR-by60-YuxgfD 253:11 0 3.5T 0 crypt sde 8:64 1 3.5T 0 disk `-ceph--6f9d4df4--7ce6--44a4--a7b1--62c85af8cfe0-osd--block--aabcb30d--0084--490a--969b--78f7af6e94da 253:8 0 3.5T 0 lvm `-g9qErH-vTXY-JQbs-eh61-W0Mn-TAV8-gof4zy 253:12 0 3.5T 0 crypt sdf 8:80 1 3.5T 0 disk `-ceph--d6b728f8--e365--46db--b30f--6c00805c752b-osd--block--88426db7--2322--4807--ac2e--b49929e170d6 253:6 0 3.5T 0 lvm `-LNG2gB-pa0w-gl2v-hVQ3-6qTd-aXsR-Lenri3 253:10 0 3.5T 0 crypt sdg 8:96 1 3.5T 0 disk sdh 8:112 1 3.5T 0 disk `-ceph--de2cfee6--8e0a--4aa0--9e6b--90c09025768c-osd--block--a3b86251--2799--4243--a857--f218fa90f29a 253:2 0 3.5T 0 lvm sdi 8:128 1 3.5T 0 disk `-ceph--30dee450--0fdd--46ea--9eec--6a4c7706df9c-osd--block--bfc090db--dde4--47dd--a1c9--1cd838ea43b3 253:4 0 3.5T 0 lvm sdj 8:144 1 3.5T 0 disk `-ceph--78febcf5--43f4--4820--8dc7--0f6c22816c9f-osd--block--da1e69c7--6427--4562--8290--90bcb9526747 253:0 0 3.5T 0 lvm sdk 8:160 1 3.5T 0 disk `-ceph--fe210281--b1f5--4d5e--9ab0--2f226612af00-osd--block--6bb9f308--e853--4303--83ea--553c3a3513e1 253:1 0 3.5T 0 lvm sdl 8:176 1 3.5T 0 disk `-ceph--9f21c916--f211--4d1b--8214--6ad1cecac810-osd--block--572d850c--c201--4af4--ac42--0ed2a6ed73ed 253:5 0 3.5T 0 lvm ```

4 22

SuSE stops building Ceph packages for its distributions
by Robert Sander 15 Aug '25

15 Aug '25

Hi, SuSE will not build even the Ceph client packages for its distributions any more: https://lists.opensuse.org/archives/list/packaging@lists.opensuse.org/threa… https://www.suse.com/releasenotes/x86_64/SUSE-SLES/15-SP6/index.html#jsc-PE… This means that this distribution cannot consume Ceph storage any more, if no other party will build these packages. If you are using SuSE Linux Enterprise Server or OpenSuSE as a client for Ceph storage you need to plan for this. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

4 4

tentacle 20.1.0 RC QE validation status
by Yuri Weinstein 14 Aug '25

14 Aug '25

Details of this release are summarized here: https://tracker.ceph.com/issues/72316#note-1 Release Notes - TBD LRC upgrade - TBD Dev Leads, please review the list of suites for completeness, as this is a new release. Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson fs: Venly orch: Adam King rbd: Ilya krbd: Ilya upgrade/squid-x: Laura upgrade/reef-x: Laura upgrade/telemetry-upgrade: PASSED upgrade/tentacle-p2p: N/A upgrade-clients:client-upgrade-reef-tentacle-tentacle: N/A upgrade-clients:client-upgrade-squid-tentacle-tentacle: N/A perf-basic: PASSED crimson-rados: PASSED powercycle: PASSED ceph-volume - Guillaume TIA

8 11

Announcing go-ceph v0.32.0
by Sven Anderson 12 Aug '25

12 Aug '25

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.32.0 The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Sven

1 1

Debugging OSD cache thrashing
by Hector Martin 11 Aug '25

11 Aug '25

Hi all, I have a small 3-node cluster (4 HDD + 1 SSD OSD per node, ceph version 19.2.2) and noticed that during snaptrim ops (?), the OSDs max out CPU usage and cluster performance plummets. Most of the CPU usage was accounted as "system", while actual disk I/O usage was low, so that didn't sound right. I perf traced the system, and found that most of the usage is in __arch_copy_to_user in the kernel, in read() syscalls. That sounds like the OSD is thrashing buffered block device reads which the kernel satisfies from the page cache (hence no real I/O load, most of the CPU usage is from data copies), so it's repeatedly reading the same disk blocks, which doesn't sound right. I increased osd_memory_target to 2.4G (from 1.2G) live with `ceph config`, and CPU usage immediately dropped to near zero. However, waiting a bit eventually the CPU thrashing returns after memory usage increases. Restarting an OSD has a similar effect. I believe that something is wrong with the OSD bluestore cache allocation/flush policy, and when the cache becomes full it starts thrashing reads instead of evicting colder cached data (or perhaps some cache bucket is starving another cache bucket of space). I would appreciate some hints on how to debug this. Are there any cache stats I should be looking at, or info on how the cache is partitioned? Here is a perf call trace of the thrashing: > - OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*) > - 95.04% ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&) > - OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&) > - 95.03% PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&) > - PGBackend::handle_message(boost::intrusive_ptr<OpRequest>) > - ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>) > - 88.86% ECBackend::handle_sub_write(pg_shard_t, boost::intrusive_ptr<OpRequest>, ECSubWrite&, ZTracer::Trace const&, ECListener&) > - 88.78% non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>) > - BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusiv > - 88.75% BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*) > - 88.71% BlueStore::_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&) > - BlueStore::_do_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&) > - 88.69% BlueStore::_do_truncate(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&, unsigned long, std > - 88.68% BlueStore::_wctx_finish(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&, BlueStore::Writ > - 88.65% BlueStore::Collection::load_shared_blob(boost::intrusive_ptr<BlueStore::SharedBlob>) > - 88.64% RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::ch > - 88.61% rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*) > - rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::DBImpl::GetImplOptions&) > - 88.43% rocksdb::Version::Get(rocksdb::ReadOptions const&, rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::PinnableWideColumns*, std > - 88.41% rocksdb::TableCache::Get(rocksdb::ReadOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, rocksdb::Sl > - rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::SliceTransform const*, bo > - 88.32% rocksdb::BlockBasedTable::FullFilterKeyMayMatch(rocksdb::FilterBlockReader*, rocksdb::Slice const&, bool, rocksdb::SliceTransfor > - rocksdb::FullFilterBlockReader::MayMatch(rocksdb::Slice const&, bool, rocksdb::GetContext*, rocksdb::BlockCacheLookupContext*, rocks > - 88.29% rocksdb::FilterBlockReaderCommon<rocksdb::ParsedFullFilterBlock>::GetOrReadFilterBlock(bool, rocksdb::GetContext*, rocksdb > - rocksdb::FilterBlockReaderCommon<rocksdb::ParsedFullFilterBlock>::ReadFilterBlock(rocksdb::BlockBasedTable const*, rocksdb::Fi > - rocksdb::Status rocksdb::BlockBasedTable::RetrieveBlock<rocksdb::ParsedFullFilterBlock>(rocksdb::FilePrefetchBuffer*, rocks > - rocksdb::Status rocksdb::BlockBasedTable::MaybeReadBlockAndLoadToCache<rocksdb::ParsedFullFilterBlock>(rocksdb::FilePref > - 88.25% rocksdb::BlockFetcher::ReadBlockContents() [ note: split, the rest is + 19.42% rocksdb::VerifyBlockChecksum] > - 68.80% rocksdb::RandomAccessFileReader::Read(rocksdb::IOOptions const&, unsigned long, unsigned long, rocksdb::Sli [ note: at this level the call trace splits into 4, but it all leads to the same place ] > - 35.99% 0xaaaac0992e40 > BlueRocksRandomAccessFile::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const > BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long, char*) > - KernelDevice::read_random(unsigned long, unsigned long, char*, bool) > - 35.98% __libc_pread > - el0_svc > - invoke_syscall > - 35.97% __arm64_sys_pread64 > - 35.96% vfs_read > - blkdev_read_iter > - 35.93% filemap_read > - 35.28% copy_page_to_iter > 35.01% __arch_copy_to_user - Hector

7 17

osd latencies and grafana dashboards, squid 19.2.2
by Christopher Durham 11 Aug '25

11 Aug '25

In my 19.2.2/squid cluster, (Rocky 9 Linux) I am trying to determine if I am having issues with OSD latency. The following URL: https://sysdig.com/blog/monitoring-ceph-prometheus/ states the following about prometheus metrics: * ceph_osd_op_r_latency_count: Returns the number of reading operations running. * ceph_osd_op_r_latency_sum: Returns the time, in milliseconds, taken by the reading operations. This metric includes the queue time. * ceph_osd_op_w_latency_count: Returns the number of writing operations running. * ceph_osd_op_w_latency_sum: Returns the time, in milliseconds, taken by the writing operations. This metric includes the queue time. and * ceph_osd_commit_latency_ms: Returns the time it takes OSD to read or write to the journal. * ceph_osd_apply_latency_ms: Returns the time it takes to write the journal to the physical disk. The first set states 'includes the queue time'. What exactly does this mean? Does this mean that this is the time waiting before writing to the journal while in the memory of the ceph-osd daemon? If so, does the latter two metrics mean that once the writes start, this is the time it takes to write to the journal or the disk? Does the first set of metrics *include* the latter? In other words, are the apply/commit latencies included in the *[r,w]_latency_sum? The URL above suggests that to calculate the write latency for a given OSD, you do the following: (rate(ceph_osd_op_w_latency_sum[5m]) / rate(ceph_osd_op_w_latency_count[5m]) >= 0) However, the grafana dashboard 'OSD Overview' (in ceph-dashboards-19.2.2 rpm), does something very similar for max,avg,quantile: max(rate(ceph_osd_op_w_max_latency_sum(cluster=\$cluster\,}[$__rate_interval])/ on (ceph_daemon) rate(ceph_osd_op_w_max_latency_count(cluster=\$cluster\,}[$__rate_interval]) * 1000)) The extra multiple of 1000 seems extraneous based on the fact that the *latency_count is already in milliseconds, and the graph itself shows 'ms'. and led me to think I have disk latency issues, as these numbers are high. (maybe I do, and maybe I misunderstand something) Other dashboards such as 'Ceph Cluster - Advanced' which, in the panel 'OSD Commit Latency Distribution' use similar promQL expressions but without the extra multiple of 1000, which looks alot better for evaluation of my latencies. So the question is, is the extra multiple of 1000 incorrect in the 'OSD Overview' dashboard? Or am I not understanding things correctly? Also, does 'ceph osd perf' just show the apply/commit sum/count metrics from above? Thanks for any assistance. -Chris

2 2

Squid 19.2.1 dashboard javascript error
by Chris Palmer 09 Aug '25

09 Aug '25

Firstly thank you so much for the 19.2.1 release. Initial testing suggests that the blockers that we had in 19.2.0 have all been resolved, so we are proceeding with further testing. We have noticed one small problem in 19.2.1 that was not present in 19.2.0 though. We use the older-style dashboard (mgr/dashboard/FEATURE_TOGGLE_DASHBOARD false). The problem happens on the Dashboard screen when health changes to WARN. If you click on WARN you get a small empty dropdown instead of the list of warnings. A javascript error is logged, and using browser inspection there is the additional bit of info that it happens in polyfill: 2025-02-07T15:59:00.970+0000 7f1d63877640 0 [dashboard ERROR frontend.error] (https://<redacted>:8443/#/dashboard): NG0901 Error: NG0901 at d.find (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3342365) at le.ngDoCheck (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3173112) at Qe (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225586) at bt (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225341) at cs (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225051) at $m (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259043) at jf (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3266563) at S1 (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259790) at $m (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259801) at fg (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3267248) Also, after this happens, no dropdowns work again until the page is forcibly refreshed. Environment is RPM install on Centos 9 Stream. I've created issue [0]. Thanks, Chris [0] https://tracker.ceph.com/issues/69867 <https://tracker.ceph.com/issues/69867?next_issue_id=69865&prev_issue_id=90>

4 9

Pgs troubleshooting
by GLE, Vivien 05 Aug '25

05 Aug '25

Hi, After replacing 2 OSD (data corruption), this is the stats of my testing ceph cluster ceph pg stat 498 pgs: 37 peering, 1 active+remapped+backfilling, 1 active+clean+remapped, 1 active+recovery_wait+undersized+remapped, 1 backfill_unfound+undersized+degraded+remapped+peered, 1 remapped+peering, 12 active+clean+scrubbing+deep, 1 active+undersized, 442 active+clean, 1 active+recovering+undersized+remapped 34 GiB data, 175 GiB used, 6.2 TiB / 6.4 TiB avail; 1.7 KiB/s rd, 1 op/s; 31/39768 objects degraded (0.078%); 6/39768 objects misplaced (0.015%); 1/13256 objects unfound (0.008%) ceph osd stat 7 osds: 7 up (since 20h), 7 in (since 20h); epoch: e427538; 4 remapped pgs Anyone had an idea of where to start to get a healthy cluster ? Thanks ! Vivien

4 22

v19.2.3 Squid released
by Yuri Weinstein 04 Aug '25

04 Aug '25

We're happy to announce the third backport release in the Squid series. https://ceph.io/en/news/blog/2025/v19-2-3-squid-released/ Notable Changes --------------- * RGW: PutObjectLockConfiguration can now be used to enable S3 Object Lock on an existing versioning-enabled bucket that was not created with Object Lock enabled. * RADOS: A new command, `ceph osd rm-pg-upmap-primary-all`, has been added that allows users to clear all pg-upmap-primary mappings in the osdmap when desired. Related trackers: - https://tracker.ceph.com/issues/67179 - https://tracker.ceph.com/issues/66867 * RBD: Moving an image that is a member of a group to trash is no longer allowed. `rbd trash mv` command now behaves the same way as `rbd rm` in this scenario. * MGR: MGR's always-on modules/plugins can now be force-disabled. This can be necessary in cases where MGR(s) needs to be prevented from being flooded by the module commands when corresponding Ceph service is down/degraded. * RGW: An authentication bypass vulnerability in STS [CVE-2023-43040] has been fixed. * RGW: S3 policy now enforces ARN-based conditionals. * RGW: Copying an object to itself no longer causes data loss. Potential corruption on ETIMEDOUT (not enabled by default), was also fixed. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-19.2.3.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: c92aebb279828e9c3c1f5d24613efca272649e62

11 22

squid 19.2.2 - osd_memory_target_autotune - best practices when host has lots of RAM
by Steven Vacaroaia 01 Aug '25

01 Aug '25

Hi What is the best practice / your expert advice about using osd_memory_target_autotune on hosts with lots of RAM ? My hosts have 1 TB RAM , only 3 NVMEs , 12 HDD and 12 SSD Should I disable autotune and allocate more RAM? I saw some suggestion for 16GB to NVME , 8GB to SSD and 6 to HDD Many thanks Steven

3 5

Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected on all newly added osd disks
by gagan tiwari 01 Aug '25

01 Aug '25

Hi Guys, I am setting up a new ceph cluster. After adding osd devices when I did run *ceph orch device ls* It shows * Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected for all osd devices.* All the OSD devices are new nvme disks. I tried to rm , zap , destroy disks and re-added them but again same messages for all disks. Please let me know how to fix this. Thanks, Gagan

3 5

RBD Mirror - Failed to unlink peer
by Scott Cairns 01 Aug '25

01 Aug '25

Hi, Following the introduction of an additional node to our Ceph cluster, we've started to see unlink errors when taking a rbd mirror snapshot. We've had RBD mirroring configured for over a year now and it's been working flawlessly, however after we created OSD's on a new node we've receiving the following error: librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f60c80056f0 handle_unlink_peer: failed to unlink peer: (2) No such file or directory This seemed to appear on around 3 of 150 snapshots on the first night and over the weeks has progressed to almost every snapshot. What's odd, is that the snapshot appears to be taken without any issues and does mirror to the DR site - we can see the snapshot ID taken on the source side is mirrored to the destination side when checking the rbd snap ls, and we've tested promoting an image on the DR site to ensure the snapshot does include up to date data, which it does. I can't see any other errors generated when the snapshot is taken to identify what file/directory isn't found - everything appears to be working okay it's just generating an error during the snapshot. I've also tried disabling mirroring on the disk and re-enabling however it doesn't appear to make any difference - there's no error on the initial mirror image, or the first snapshot taken after that, but every subsequent snapshot shows the error again. Any ideas? Thanks, Scott ________________________________ The content of this e-mail and any attachment is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of Tecnica Limited. If you have received this e-mail in error please notify the sender. Any use, dissemination, forwarding, printing, or copying of this e-mail or any attachments thereto, in whole or part, without permission is strictly prohibited. Tecnica Limited Registered office: 5 Castle Court, Carnegie Campus, Dunfermline, Fife, KY11 8PB. Registered in Scotland No. SC250307. VAT No. 827 5110 42. This footnote also confirms that this email message has been swept for the presence of computer viruses.

4 8

Two Quick Reminders
by Anthony Middleton 31 Jul '25

31 Jul '25

The deadline for the Cephalocon 2025 T-shirt design contest <https://ceph.io/en/news/blog/2025/cephalocon25-tshirt-contest/> is tomorrow at 11:59 pm PDT. Please send your designs to me at community-manager(a)ceph.io. Feel free to reach out to me with any questions you might have. The Ceph Community Survey is ongoing. I'm reviewing the data, but there's still time to participate. Please spend 2-3 minutes to share your feedback at https://forms.gle/uv57ahnQkjbu6VSJ8. Best, Anthony Middleton Ceph Community Manager

1 0

squid 19.2.2 deployed with cephadmin - no grafana data on some dashboards ( RGW, MDS)
by Steven Vacaroaia 31 Jul '25

31 Jul '25

Hi, Any of you encounter this issue ? Most of the dashboards works but MDS and RGW specific ones have no data Prometheus ( http:server:9283) does not show anything with RGW or MDS in it so, I am guessing, grafana cannot show what it is not collected :-) ceph-exporter is running Any suggestions/ ideas for how to troubleshoot /fix this ? Many thanks Steven

3 3

bdev_ioring -- true - Drives failing
by Devender Singh 31 Jul '25

31 Jul '25

Hello all Using 19.2.2, when enabling bdev_ioring and rebooting host then osd never comes online as below…. Why? Also Dasboard not matching with cli? Host output in dashboard… showing running. Tried failing manager too but it same. Dashboard showing down # ceph health detail HEALTH_WARN 4 osds down; 1 host (4 osds) down; Degraded data redundancy: 12998/38994 objects degraded (33.333%), 287 pgs degraded, 801 pgs undersized [WRN] OSD_DOWN: 4 osds down osd.4 (root=default,host= host07n) is down osd.5 (root=default,host= host07n) is down osd.6 (root=default,host= host07n) is down osd.7 (root=default,host= host07n) is down [WRN] OSD_HOST_DOWN: 1 host (4 osds) down host host07n (root=default) (4 osds) is down # ceph orch ps |grep -v running. ————> not showing anything down…. NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID # systemctl list-units |grep -i osd var-lib-ceph-osd-ceph\x2d4.mount loaded active mounted /var/lib/ceph/osd/ceph-4 var-lib-ceph-osd-ceph\x2d5.mount loaded active mounted /var/lib/ceph/osd/ceph-5 var-lib-ceph-osd-ceph\x2d6.mount loaded active mounted /var/lib/ceph/osd/ceph-6 var-lib-ceph-osd-ceph\x2d7.mount loaded active mounted /var/lib/ceph/osd/ceph-7 ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.4.service loaded activating auto-restart Ceph osd.4 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.5.service loaded activating auto-restart Ceph osd.5 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.6.service loaded activating auto-restart Ceph osd.6 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.7.service loaded activating auto-restart Ceph osd.7 for 3b850efe-5dec-11f0-af3c-c1a764f7824e system-ceph\x2dosd.slice loaded active active Slice /system/ceph-osd ceph-osd.target loaded active active ceph target allowing to start/stop all ceph-osd@.service instances at once Came to running ….. # systemctl list-units |grep -i osd var-lib-ceph-osd-ceph\x2d4.mount loaded active mounted /var/lib/ceph/osd/ceph-4 var-lib-ceph-osd-ceph\x2d5.mount loaded active mounted /var/lib/ceph/osd/ceph-5 var-lib-ceph-osd-ceph\x2d6.mount loaded active mounted /var/lib/ceph/osd/ceph-6 var-lib-ceph-osd-ceph\x2d7.mount loaded active mounted /var/lib/ceph/osd/ceph-7 ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.4.service loaded active running Ceph osd.4 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.5.service loaded active running Ceph osd.5 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.6.service loaded active running Ceph osd.6 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.7.service loaded active running Ceph osd.7 for 3b850efe-5dec-11f0-af3c-c1a764f7824e system-ceph\x2dosd.slice loaded active active Slice /system/ceph-osd ceph-osd.target loaded active active ceph target allowing to start/stop all ceph-osd@.service instances at once Failed ….. # systemctl list-units |grep -i osd var-lib-ceph-osd-ceph\x2d4.mount loaded active mounted /var/lib/ceph/osd/ceph-4 var-lib-ceph-osd-ceph\x2d5.mount loaded active mounted /var/lib/ceph/osd/ceph-5 var-lib-ceph-osd-ceph\x2d6.mount loaded active mounted /var/lib/ceph/osd/ceph-6 var-lib-ceph-osd-ceph\x2d7.mount loaded active mounted /var/lib/ceph/osd/ceph-7 ● ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.4.service loaded failed failed Ceph osd.4 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ● ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.5.service loaded failed failed Ceph osd.5 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ● ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.6.service loaded failed failed Ceph osd.6 for 3b850efe-5dec-11f0-af3c-c1a764f7824e ● ceph-3b850efe-5dec-11f0-af3c-c1a764f7824e(a)osd.7.service loaded failed failed Ceph osd.7 for 3b850efe-5dec-11f0-af3c-c1a764f7824e system-ceph\x2dosd.slice loaded active active Slice /system/ceph-osd ceph-osd.target loaded active active ceph target allowing to start/stop all ceph-osd@.service instances at once Regards Dev

2 4

Kafka bucket notifications with OAUTHBEARER on Ceph Reef/Squid
by Rao, Shreesha 31 Jul '25

31 Jul '25

Hi Ceph community, We are exploring setting up bucket notifications using Kafka on our Ceph cluster running Reef/Squid (P.S. versions output at the end of this email), and have verified that the Kafka library is integrated in the `radosgw-admin` binary. We attempted to use mTLS authentication by providing certificates and keys to `radosgw-admin`, but it fails to connect to the Kafka broker. From various threads and the open PR here (http://github.com/ceph/ceph/pull/61572) it seems mTLS authentication is not yet supported on the Ceph version we are using. Using username/password is not an option with our Kafka system. The documentation mentions support for the OAUTHBEARER mechanism; however, the payload example at https://docs.ceph.com/en/squid/radosgw/notifications/#create-a-topic does not include a token field. It also does not clarify how token refresh is handled when the token expires. Specifically: • Where can we configure the URL or method for RGW to refresh the auth token? • Are there any example configurations available demonstrating the correct setup and token refresh with OAUTHBEARER for Kafka notifications? Additionally, could you please clarify if these configurations must be done via `radosgw-admin` commands or through an S3 client? Examples of commands for either approach would be very helpful. If OAUTHBEARER and user-password is not an option, is there any other way to configure bucket notifications using Kafka? Please advise. Thank you very much in advance for your guidance. Best regards, Shreesha P.S. ubuntu@fr1obj001:~$ sudo cephadm shell Inferring fsid 877b08fe-f88a-4468-a224-7cf7980c0f47 Inferring config /var/lib/ceph/877b08fe-f88a-4468-a224-7cf7980c0f47/mon.fr1obj001/config root@fr1obj001:/# ceph -v ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) root@fr1obj001:/# ceph versions { "mon": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "mgr": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "osd": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 106 }, "rgw": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "overall": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 115 } } Sensitivity: Company-Internal ==================== This email/fax message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution of this email/fax is prohibited. If you are not the intended recipient, please destroy all paper and electronic copies of the original message.

1 0

Is "ceph -v" command deprecated?
by Rao, Shreesha 31 Jul '25

31 Jul '25

Hi, We were running ceph cluster with reef version (18.2.4) and recently upgraded to the new squid version. However, “ceph -v” still shows the version as reef, even though the upgrade successfully went through without any errors. Are we missing something here? root@fr1obj001:/# ceph -v ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) root@fr1obj001:/# ceph versions { "mon": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "mgr": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "osd": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 106 }, "rgw": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "overall": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 115 } } Best, Shreesha Sensitivity: Company-Internal ==================== This email/fax message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution of this email/fax is prohibited. If you are not the intended recipient, please destroy all paper and electronic copies of the original message.

2 2

Re: Getting SSL Certificate Verify failed Error while installation
by John Mulligan 30 Jul '25

30 Jul '25

On Tuesday, July 29, 2025 10:50:06 AM Eastern Daylight Time gagan tiwari wrote: > Hi John, > Thanks for reply! > > curl seems to be working fine. Here is output of the command :- > > curl https://download.ceph.com > <html> > <head><title>Index of /</title></head> > <body> > <h1>Index of /</h1><hr><pre><a href="../">../</a> > <a href="archive/">archive/</a> > 13-Nov-2024 03:25 - > <a href="calamari/">calamari/</a> > 29-Oct-2015 18:40 - > <a href="ceph-iscsi/">ceph-iscsi/</a> > 22-May-2024 03:18 - > <a href="cephmetrics/">cephmetrics/</a> > 29-Jun-2017 22:07 - > <a href="cloudinit/">cloudinit/</a> > 11-Aug-2020 14:47 - > <a href="debian-15.1.0/">debian-15.1.0/</a> > 13-Mar-2020 22:19 - > <a href="debian-15.1.1/">debian-15.1.1/</a> > 13-Mar-2020 22:21 - > <a href="debian-15.2.0/">debian-15.2.0/</a> > 23-Mar-2020 22:43 - > <a href="debian-15.2.1/">debian-15.2.1/</a> > 09-Apr-2020 16:11 - > <a href="debian-15.2.10/">debian-15.2.10/</a> > 18-Mar-2021 13:41 - > <a href="debian-15.2.11/">debian-15.2.11/</a> > 19-Apr-2021 22:17 - > <a href="debian-15.2.12/">debian-15.2.12/</a> > 13-May-2021 23:06 - > <a href="debian-15.2.13/">debian-15.2.13/</a> > 26-May-2021 22:26 - > <a href="debian-15.2.14/">debian-15.2.14/</a> > 05-Aug-2021 21:32 - > <a href="debian-15.2.15/">debian-15.2.15/</a> > 20-Oct-2021 18:33 - > <a href="debian-15.2.16/">debian-15.2.16/</a> > 01-Mar-2022 12:18 - > <a href="debian-15.2.17/">debian-15.2.17/</a> > 09-Aug-2022 20:59 > > > I have tried to install the latest version of squid 19.2.3 and 19.2.2. > Facing the same SSL issue with both versions. > > Thanks, > Gagan > It dawns on me that curl might not be the best test as I think it can use a different cert bundle by default. Does this command fail the same way as cephadm? ``` python3 -c 'import urllib.request; print(len(urllib.request.urlopen("https://download.ceph.com").read()))' ``` PS. Please keep the list CC'ed. We want others to be able to help as well and keep a record of what was discussed.

4 4

MDS Client Request Load
by Devender Singh 30 Jul '25

30 Jul '25

Hello How to lower this load? Cephfs is on EC pool sitting over replicated data and meta.. What settings can improve it? Regards Dev

2 1

Upgrade from 19.2.2 to .3 pauses on 'phantom' duplicate osd?
by Harry G Coin 30 Jul '25

30 Jul '25

Need a clue about what appears to be a phantom duplicate osd automagically created/discovered via the upgrade process -- which blocks the upgrade. The upgrade process on a known-good 19.2.2 to 19.2.3 proceeded normally through the mgrs and mons. It upgraded most of the osds, then stopped with the complaint "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon osd.1 on host noc3 failed." The roster in the "Daemon Versions" table on the dashboard looks normal except: There are two entries for 'osd.1' One of them has the correct version number, 19.2.2, the other is blank. The upgrade appears 'stuck'. An attempt to 'resume' resulted in the same error. The cluster operations are normal with all osds up and in. The cluster is ipv6. Oddly ceph -s reports: root@noc1:~# ceph -s cluster: id: 406xxxxxxx0f8 health: HEALTH_WARN Public/cluster network defined, but can not be found on any host Upgrading daemon osd.1 on host noc3 failed. services: mon: 5 daemons, quorum noc4,noc2,noc1,noc3,sysmon1 (age 39m) mgr: noc2.yhyuxd(active, since 4h), standbys: noc3.sybsfb, noc4.tvhgac, noc1.jtteqg mds: 1/1 daemons up, 3 standby osd: 27 osds: 27 up (since 3h), 27 in (since 10d) data: volumes: 1/1 healthy pools: 16 pools, 1809 pgs objects: 14.77M objects, 20 TiB usage: 52 TiB used, 58 TiB / 111 TiB avail pgs: 1808 active+clean 1 active+clean+scrubbing io: client: 835 KiB/s rd, 1.0 MiB/s wr, 24 op/s rd, 105 op/s wr progress: Upgrade to 19.2.3 (4h) [============................] (remaining: 4h) Related log entry: 29/7/25 02:40 PM[ERR]cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-4067126d-01cb-40af-824a-881c130140f8-osd-1 /usr/bin/docker: stdout /usr/bin/docker: stderr Error response from daemon: No such container: ceph-4067126dXXXXXXXXXXXXXXXXXXX40f8-osd-1 Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-4067126dXXXXXXXXXXXXXXXXXXX40f8-osd.1 /usr/bin/docker: stdout /usr/bin/docker: stderr Error response from daemon: No such container: ceph-4067126dXXXXXXXXXXXXXXXXXXX40f8-osd.1 Reconfig daemon osd.1 ... Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 5581, in File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 5569, in main File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 3051, in command_deploy_from File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 3086, in _common_deploy File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 3106, in _deploy_daemon_container File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 1077, in deploy_daemon File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 765, in create_daemon_dirs File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/cephadmlib/file_utils.py", line 52, in write_new IsADirectoryError: [Errno 21] Is a directory: '/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/osd.1/config.new' -> '/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/osd.1/config' Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1145, in _check_daemons self.mgr._daemon_action(daemon_spec, action=action) File "/usr/share/ceph/mgr/cephadm/module.py", line 2545, in _daemon_action return self.wait_async( File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async return self.event_loop.get_result(coro, timeout) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result return future.result(timeout) File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result return self.__get_result() File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/serve.py", line 1381, in _create_daemon out, err, code = await self._run_cephadm( File "/usr/share/ceph/mgr/cephadm/serve.py", line 1724, in _run_cephadm raise OrchestratorError( orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-4067126dXXXXXXXXXXXXXXXXXXX40f8-osd-1 /usr/bin/docker: stdout /usr/bin/docker: stderr Error response from daemon: No such container: ceph-4067126dXXXXXXXXXXXXXXXXXXX40f8-osd-1 Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-4067126dXXXXXXXXXXXXXXXXXXX40f8-osd.1 /usr/bin/docker: stdout /usr/bin/docker: stderr Error response from daemon: No such container: ceph-4067126dXXXXXXXXXXXXXXXXXXX40f8-osd.1 Reconfig daemon osd.1 ... Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 5581, in File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 5569, in main File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 3051, in command_deploy_from File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 3086, in _common_deploy File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 3106, in _deploy_daemon_container File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 1077, in deploy_daemon File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/__main__.py", line 765, in create_daemon_dirs File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36/cephadmlib/file_utils.py", line 52, in write_new IsADirectoryError: [Errno 21] Is a directory: '/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/osd.1/config.new' -> '/var/lib/ceph/4067126dXXXXXXXXXXXXXXXXXXX40f8/osd.1/config'

2 1

Still Time to Take the Ceph Community Survey
by Anthony Middleton 29 Jul '25

29 Jul '25

Hey Ceph Users, I'm still collecting responses for the *Ceph Community Survey*, and your input would be greatly appreciated! This brief survey helps us understand: - How Ceph is used in production today - Why some organizations have moved away - - The value Ceph provides to your teams - Who might be interested in sharing their story at *Cephalocon 2025* in Vancouver I will begin reviewing submissions later this week, so please submit your feedback as soon as possible! Survey link: https://forms.gle/uQPiKkEbQAvjPfp46. Thanks to everyone who has contributed so far! Your insights are shaping the future of Ceph. Best, Anthony Middleton Ceph Community Manager

1 0

How to create buckets in secondary zonegroup?
by Adam Prycki 29 Jul '25

29 Jul '25

Hello, I have a question about ceph multisite theory. How to create buckets in non-master zonegroup? I'm trying to create crate a test enviroment with 2 regions. I've crated deployment which looks like this REALM region-1 (master zonegroup) region-1-zone-a (master zone) region-1-zone-b region-2 region-2-zone-a (master zone) The goal would be to have 2 regions, each with separate data and 2 active-active zones in each. Just after creating region-2-zone-a I've decided to create a bucket there in region-2 and upload something. But I cannot create a bucket in new region. Rgw allows you to create buckets only on master zone. If I try to use s3cmd to create bucket on region-1-zone-b endpoint I get error 503. (as expected) Trying to list region-1 bucket on region-2 zone gives me 301 (moved permanently) so region-2 has metadata and working authentication. But I cannot figure out how to make a bucket in region-2. s3cmd .s3cfg-region-2 mb s3://bucket gives me error 503 like if I were trying to create a bucket on non-master zone. Creating bucket on region-1 endpoint with s3cmd flags specifying LocationConstraint region-2 creates bucket in region-1 I see in region-1-zone-a logs that rgw sees region-2 in LocationConstraint argument but creates bucket in region-1 I'm working on ceph 19.2.2 Best regards Adam Prycki

2 5

Warning about ceph discard on kioxia CD6 KCD61LUL7T68 NVMes
by Adam Prycki 29 Jul '25

29 Jul '25

Hi, I'm just throwing it here as a warning to other users. Kioxia KCD61LUL7T68 NVMe device have terrible discard performance. (All devices report firmware 8001) I have multiple of these kioxia SSDs in a ceph cluster. I normally configured bdev_async_discard_threads and bdev_enable_discard on all SSD. I have a script which makes new rbd snapshots and clears old ones. It has plenty of sleep commands to make whole process easier on the cluster, but these SSDs just cannot handle it. Example iostat: avg-cpu: %user %nice %system %iowait %steal %idle 45.6% 0.0% 27.0% 10.4% 0.0% 17.0% r/s rkB/s rrqm/s %rrqm r_await rareq-sz Device 1482.00 55.9M 208.00 12.3% 0.22 38.6k nvme1n1 13.00 220.0k 0.00 0.0% 0.15 16.9k nvme2n1 w/s wkB/s wrqm/s %wrqm w_await wareq-sz Device 7333.00 92.6M 15303.00 67.6% 0.15 12.9k nvme1n1 1470.00 13.2M 294.00 16.7% 0.10 9.2k nvme2n1 d/s dkB/s drqm/s %drqm d_await dareq-sz Device 807.00 3.8M 0.00 0.0% 2.27 4.8k nvme1n1 924.00 7.5M 0.00 0.0% 2.08 8.4k nvme2n1 NVMes start to choke around 700-800 discard IOPS. These discard ops sizes are probably terrible for the SSDs but afaik there is no way to fix this without just disabling discard on bluestore. In this case SSDs were about half full. I've used multiple brands of SSDs on ceph with discard enabled and I've never seen results this bad. Best regards Adam Prycki

1 0

squid 19.2.2 - RGW performance tuning
by Steven Vacaroaia 29 Jul '25

29 Jul '25

Hi, Are there any performance tuning settings for RGW that you can share please ? With the hardware I have, I believe I should be able to "squeeze" more 7 Supermicro dual CPU, 128 cores ( Intel Gold 6530), 1 TB RAM, 2 x 100GB bonded cluster network, 2 x 25 GB bonded private network cephadm, 5 RGW daemons (one per server) with haproxy in front of them Uploading a 5.5 GB file using s3cmd takes 2m21 ( the client has a 10GB connection) hardware/other tweaks HDD +DB/WAL on NVME pool EC 4+2 with meta on NVME vm.swapineess = 0 CPU intel_idle.max_cstate=0 processor.max_cstate=0 The servers have a less than 2 % CPU utilization and 30% memory usage Cluster is not under heavy usage - I only have one client using the bucket which has 7001 shards I have been trying these ones with very little success (taken from here https://links.imagerelay.com/cdn/3404/ql/3cd809aeba7c42f395ca8a7256ba488a/B… ) rgw thread pool size = 512 #default 100, increase for heavier load rgw max chunk size = 4194304# default is 128k, increase to 4MB helps with larger objects rgw_obj_stripe_size = 4194304 # (default 4M for luminous) rgw_list_bucket_min_readahead = 4000 #(default 1000) rgw_max_listing_results = 4000 rgw_cache_expiry_interval = 1800 #(default 900s) rgw_enable_usage_log = false rgw_enable_ops_log = false rgw dynamic resharding = false rgw override bucket index max shards = 50 # rgw bucket index max aio = 16 # default 8 rgw cache lru size = 50000 #number of entries in the cache rgw_gc_obj_min_wait = 21600 #in seconds(default 2_hr) rgw gc processor period = 7200 #(default 1hr) rgw objexp gc interval = 3600 # default 10_min, objecter inflight op bytes = 1073741824 # default 100_M, this is 1GB objecter inflight ops = 24576 Many thanks Steven

1 0

Re: SMB service configuration
by Robert Sander 29 Jul '25

29 Jul '25

Hi, Am 7/29/25 um 09:18 schrieb Ryan Sleeth: > Have you seen this? https://ceph.io/en/news/blog/2025/smb-manager- > module/ <https://ceph.io/en/news/blog/2025/smb-manager-module/> When I try to enable the smb mgr module on a 19.2.3 cluster, it tells me that it is not available: # ceph mgr module enable smb Error ENOENT: all mgr daemons do not support module 'smb', pass --force to force enablement Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

3 4

Getting SSL Certificate Verify failed Error while installation
by gagan tiwari 29 Jul '25

29 Jul '25

Hi Guys, I am trying to install via cephadm but getting the following error. There is no firewall running on the server. Please advise [root@ceph-mon1 ~]# curl --silent --remote-name --location https://download.ceph.com/rpm-${CEPH_RELEASE}/el9/noarch/cephadm [root@ceph-mon1 ~]# [root@ceph-mon1 ~]# chmod +x cephadm [root@ceph-mon1 ~]# [root@ceph-mon1 ~]# getenforce ^C [root@ceph-mon1 ~]# ./cephadm install Installing packages ['cephadm']... [root@ceph-mon1 ~]# [root@ceph-mon1 ~]# ./cephadm add-repo --release squid Traceback (most recent call last): File "/usr/lib64/python3.9/urllib/request.py", line 1346, in do_open h.request(req.get_method(), req.selector, req.data, headers, File "/usr/lib64/python3.9/http/client.py", line 1285, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib64/python3.9/http/client.py", line 1331, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output self.send(msg) File "/usr/lib64/python3.9/http/client.py", line 980, in send self.connect() File "/usr/lib64/python3.9/http/client.py", line 1454, in connect self.sock = self._context.wrap_socket(self.sock, File "/usr/lib64/python3.9/ssl.py", line 501, in wrap_socket return self.sslsocket_class._create( File "/usr/lib64/python3.9/ssl.py", line 1074, in _create self.do_handshake() File "/usr/lib64/python3.9/ssl.py", line 1343, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1147) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/root/./cephadm/__main__.py", line 5581, in <module> File "/root/./cephadm/__main__.py", line 5569, in main File "/root/./cephadm/__main__.py", line 4576, in command_add_repo File "/root/./cephadm/cephadmlib/packagers.py", line 412, in validate File "/usr/lib64/python3.9/urllib/request.py", line 214, in urlopen return opener.open(url, data, timeout) File "/usr/lib64/python3.9/urllib/request.py", line 517, in open response = self._open(req, data) File "/usr/lib64/python3.9/urllib/request.py", line 534, in _open result = self._call_chain(self.handle_open, protocol, protocol + File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain result = func(*args) File "/usr/lib64/python3.9/urllib/request.py", line 1389, in https_open return self.do_open(http.client.HTTPSConnection, req, File "/usr/lib64/python3.9/urllib/request.py", line 1349, in do_open raise URLError(err) urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1147)>

2 2

Odd RBD stats metrics
by Christopher James 29 Jul '25

29 Jul '25

Hi, We have encountered some issues regarding the RBD stats metrics. We have some Grafana panels that show the per-pool rate of changes in RBD IOPS and Throughput. We use these queries for IOPS round(sum(irate(ceph_rbd_write_ops[1m])) by (pool)) round(sum(irate(ceph_rbd_read_ops[1m])) by (pool)) and these queries for throughput round(sum(irate(ceph_rbd_write_bytes[1m])) by (pool)) round(sum(irate(ceph_rbd_read_bytes[1m])) by (pool)) Now, the problem we encounter here is that at some points in time, there are some odd outputs. In a way that the output has a lot of spikes, and at many points in time, there seems to be no change in the mentioned values, hence the irate()'s output would become zero, and the next data point would show that the data has somehow doubled from the last time. I investigated the mgr/prometheus module to see if it is related to the cache mechanism, and it doesn't seem to be about that because the metrics collection method is done in less than 10 seconds most of the time and there is no "collecting data took more than ..." log. I also investigated the raw time series data and saw that the two datapoints are exactly the same at the times that the irate() returns zero. I have put some screenshots of the panels in this read-only Google Doc: https://docs.google.com/document/d/1Rf0dl4qAWnOtG80BsxVY9cjljQqliG2JHxgzVxD… It is somewhat odd, to be honest, that every image in every pool does not change in IOPS or throughput. Is this the natural way the metrics are exposed? Should I not use irate()? Has anybody else encountered this issue as well? Cheers, Chris

2 2

Re: Odd RBD stats metrics
by Christopher James 29 Jul '25

29 Jul '25

The scrape_interval is set to 15 seconds on prom. The mgr configuration is also set to 15 seconds. I'm using irate() to try to observe sudden increases in the mentioned metrics. Most RBD pools are on SSD OSDs, and the HDD OSDs use NVMe drives for DB. On Mon, Jul 28, 2025 at 4:51 PM Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > What is your scrape period? The templated default 10s? > > My thoughts: > > * Why are you using irate? Try rate() instead, and try a longer period > than one minute, especially if that isn’t more than a few scrape intervals > * Is your DB on SSDs? > > > On Jul 28, 2025, at 7:39 AM, Christopher James < > christopher.jamesjr2000(a)gmail.com> wrote: > > Is this the natural way the metrics are exposed? Should I not use irate()? > > >

1 0

SMB service configuration
by Robert Sander 29 Jul '25

29 Jul '25

Hi, where do I find more information about the config objects needed to deploy an smb service? https://docs.ceph.com/en/latest/cephadm/services/smb/ mentions "config_uri" but does not tell me what to put there. It also mentions "cluster_meta_uri" for a clustered setup but does not tell me what to put there. I am familiar with setting up a Samba CTDB cluster manually but wanted to try the new smb service of the orchestrator. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

CephFS limp mode when fullest OSD is between nearfull & backfillfull value
by Jeremi-Ernst Avenant 28 Jul '25

28 Jul '25

Good day We've been struggling with this issue since we've upgraded post 16.2.11 to 16.2.15 and now up to Reef 18.2.7. We didn't have this issue prior to upgrading from 16.2.11 to 16.2.15. The moment the fullest OSD % is between the nearful and backfillful value, CephFS goes into a limp mode. Meaning our Client IO goes from ~30 *GiB*/s to 100 *MiB*/s. Doesn't matter what the threshold values are set. If my nearful% is 75% and backfillfull 80%, with the highest OSD 75.01% it will limp. If my nearful% is 85% and backfillfull 90%, with the highest OSD 85.01% it will go into limp mode. The only way to get CephFS operational again, is to set both ratios to either far below or above the highest OSD value. e.g. if the fullest is 72% , I need to set it to ceph osd set-nearfull-ratio 0.76 & ceph osd set-backfillfull-ratio 0.81 When I deploy a brand new cluster from scratch on our testbed I get the same issue, Cephadm / Reef 18.2.4. Back in the days I was able to manipulate these settings with the following as well, but I think since Pacific they are no longer tuneable: ceph tell osd.1231 config show | grep -E 'osd_nearfull_ratio|osd_backfillfull_ratio|osd_full_ratio' "mon_osd_backfillfull_ratio": "0.900000", "mon_osd_full_ratio": "0.950000", "mon_osd_nearfull_ratio": "0.850000", ceph tell osd.$osd injectargs '--osd_nearfull_ratio=0.85' ceph tell osd.$osd injectargs '--osd_backfillfull_ratio=0.90' ceph tell osd.$osd injectargs '--osd_full_ratio=0.95' URL to the issue: https://tracker.ceph.com/issues/70129 Any ideas would be greatly appreciated. -- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi(a)idia.ac.za <mfundo(a)idia.ac.za> Rondebosch, Cape Town, 7600, South Africa

2 2

Squid: successfully drained host can't be removed
by Eugen Block 26 Jul '25

26 Jul '25

Hi *, an unexpected issue occurred today, at least twice, so it seems kind of reproducable. I've been preparing a demo in a (virtual) lab cluster (19.2.2) and wanted to drain multiple hosts. The first time I didn't pay much attention, but the draining seemed stuck (kind of a common issue these days), so I intervened and cleaned up until I got into a healthy state, all good. Then I did my thing, changed the crush tree, added the removed hosts again, cephadm created the OSDs, backfill finished successfully. Now I wanted to reset the cluster again to my starting point, so I issued the drain command again for multiple hosts (each host has 2 OSDs): # for i in {5..8}; do ceph orch host drain host$i; done This time all OSDs were drained successfully (I watched 'ceph orch osd rm status'), so I wanted to remove the hosts, but it failed: # for i in {5..8}; do ceph orch host rm host$i --rm-crush-entry; done Removed host 'host5' Removed host 'host6' Removed host 'host7' Error EINVAL: Not allowed to remove host8 from cluster. The following daemons are running in the host: type id -------------------- --------------- osd 6 Please run 'ceph orch host drain host8' to remove daemons from host But there was nothing to drain anymore, osd.6 was already successfully removed from the crush tree. But on host8 there was still a daemon I had to clean up manually: host8:~ # cephadm rm-daemon --name osd.6 --fsid 543967bc-e586-32b8-bd2c-2d8b8b168f02 --force I compared the cephadm.log files (3 out of 4 to-be-drained hosts were successfully drained) and on host8 the command rm-daemon was never executed (until I did manually). Is this a known issue? It doesn't seem to happen with only on host, at least I didn't notice in the past. Should I create a tracker for this? Thanks, Eugen

2 3

MDS Mount Issue - 19.2.2
by Devender Singh 26 Jul '25

26 Jul '25

Hello all I have cephfs working fine with running ads but still not able to mount, getting error mount error: no mds (Metadata Server) is up Below are details # ceph osd dump | grep min_compat_client require_min_compat_client reef min_compat_client reef # ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.002882", "last_optimize_started": "Tue Jul 15 21:16:41 2025", "mode": "upmap-read", "no_optimization_needed": true, "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", "plans": [] } root@van2-host06n:~# dpkg -l ceph-common Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==============-=============-============-================================================================== ii ceph-common 19.2.2-1jammy amd64 common utilities to mount and interact with a ceph storage cluster root@van2-host06n:~# ceph fs status van2.dev - 0 clients ======== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active van2.dev.van2-host06n.wqttsi Reqs: 0 /s 10 13 12 0 1 active van2.dev.van2-host07n.fngvlt Reqs: 0 /s 10 13 11 0 POOL TYPE USED AVAIL cephfs.van2.dev.meta metadata 230k 53.0T cephfs.van2.dev.data data 0 53.0T STANDBY MDS van2-dev.van2-host07n.jnacmh van2-dev.van2-host06n.wpzshe van2-dev.van2-host08n.slrfej MDS version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable) SYstemd file. =========== # cat cephfs.mount # # this mounts on /cephfs [Unit] Description=mount /cephfs After=network-online.target Requires=network-online.target [Mount] Where=/cephfs What=10.101.1.246:6789,10.101.1.247:6789,10.101.1.248:6789:/ Type=ceph Options=fs=van2.dev,name=admin,noatime,_netdev,x-systemd.automount [Install] WantedBy=multi-user.target root@van2-host06n:~# journalctl -u cephfs.mount Jul 15 19:31:10 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:32:12 van2-host06n.van2.example.com mount[1809096]: mount error: no mds (Metadata Server) is up. The clu> Jul 15 19:32:12 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=> Jul 15 19:32:12 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:32:12 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:34:59 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:35:59 van2-host06n.van2.example.com mount[1810584]: mount error: no mds (Metadata Server) is up. The clu> Jul 15 19:35:59 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=> Jul 15 19:35:59 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:35:59 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:36:16 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:37:17 van2-host06n.van2.example.com mount[1811498]: mount error: no mds (Metadata Server) is up. The clu> Jul 15 19:37:17 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=> Jul 15 19:37:17 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:37:17 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:49:38 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:50:40 van2-host06n.van2.example.com mount[1817486]: mount error: no mds (Metadata Server) is up. The clu> Jul 15 19:50:40 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=> Jul 15 19:50:40 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:50:40 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:51:58 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=killed, status=> Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'signal'. Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Unit process 1817555 (mount.ceph) remains > Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: Unmounted mount /cephfs. Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Found left-over process 1817555 (mount.cep> Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: This usually indicates unclean termination of a previous> Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:52:59 van2-host06n.van2.example.com mount[1817555]: mount error: no mds (Metadata Server) is up. The clu> lines 1-29...skipping... Jul 15 19:31:10 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:32:12 van2-host06n.van2.example.com mount[1809096]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be> Jul 15 19:32:12 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:32:12 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:32:12 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:34:59 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:35:59 van2-host06n.van2.example.com mount[1810584]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be> Jul 15 19:35:59 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:35:59 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:35:59 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:36:16 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:37:17 van2-host06n.van2.example.com mount[1811498]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be> Jul 15 19:37:17 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:37:17 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:37:17 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:49:38 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:50:40 van2-host06n.van2.example.com mount[1817486]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be> Jul 15 19:50:40 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:50:40 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:50:40 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:51:58 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=killed, status=15/TERM Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'signal'. Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Unit process 1817555 (mount.ceph) remains running after unit stopped. Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: Unmounted mount /cephfs. Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Found left-over process 1817555 (mount.ceph) in control group while starting uni> Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: This usually indicates unclean termination of a previous run, or service implementation defici> Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:52:59 van2-host06n.van2.example.com mount[1817555]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be> Jul 15 19:53:26 van2-host06n.van2.example.com mount[1817572]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be> Jul 15 19:53:26 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:53:26 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:53:26 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:56:29 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:57:29 van2-host06n.van2.example.com mount[1817655]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be> Jul 15 19:57:29 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:57:29 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:57:29 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. lines 1-38...skipping... Jul 15 19:31:10 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:32:12 van2-host06n.van2.example.com mount[1809096]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized Jul 15 19:32:12 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:32:12 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:32:12 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:34:59 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:35:59 van2-host06n.van2.example.com mount[1810584]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized Jul 15 19:35:59 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:35:59 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:35:59 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:36:16 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:37:17 van2-host06n.van2.example.com mount[1811498]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized Jul 15 19:37:17 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:37:17 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:37:17 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:49:38 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:50:40 van2-host06n.van2.example.com mount[1817486]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized Jul 15 19:50:40 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:50:40 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:50:40 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:51:58 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=killed, status=15/TERM Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'signal'. Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Unit process 1817555 (mount.ceph) remains running after unit stopped. Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: Unmounted mount /cephfs. Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Found left-over process 1817555 (mount.ceph) in control group while starting unit. Ignoring. Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:52:59 van2-host06n.van2.example.com mount[1817555]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized Jul 15 19:53:26 van2-host06n.van2.example.com mount[1817572]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized Jul 15 19:53:26 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:53:26 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:53:26 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:56:29 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:57:29 van2-host06n.van2.example.com mount[1817655]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized Jul 15 19:57:29 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:57:29 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:57:29 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:58:50 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:59:51 van2-host06n.van2.example.com mount[1818287]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized Jul 15 19:59:51 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:59:51 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:59:51 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 20:05:43 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 20:06:44 van2-host06n.van2.example.com mount[1818912]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized lines 1-45

3 11

2 MDSs behind on trimming on my Ceph Cluster since the upgrade from 18.2.6 (reef) to 19.2.2 (squid)
by Edouard FAZENDA 25 Jul '25

25 Jul '25

Dear all, I have the following issue on my Ceph cluster MDSs behind on trimming on my Ceph Cluster since the upgrade using cephadm from 18.2.6 to 19.2.2. Here some cluster logs : 8/5/25 09:00 AM[WRN]overall HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:50 AM[WRN]overall HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:40 AM[WRN]mds.cephfs.node2.isqjza(mds.0): Behind on trimming (326/128) max_segments: 128, num_segments: 326 8/5/25 08:40 AM[WRN]mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming (326/128) max_segments: 128, num_segments: 326 8/5/25 08:40 AM[WRN][WRN] MDS_TRIM: 2 MDSs behind on trimming 8/5/25 08:40 AM[WRN]Health detail: HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:33 AM[WRN]Health check update: 2 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:33 AM[WRN]Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:30 AM[INF]overall HEALTH_OK 8/5/25 08:22 AM[INF]Cluster is now healthy 8/5/25 08:22 AM[INF]Health check cleared: MDS_TRIM (was: 1 MDSs behind on trimming) 8/5/25 08:22 AM[INF]MDS health message cleared (mds.?): Behind on trimming (525/128) 8/5/25 08:22 AM[WRN]Health check update: 1 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:22 AM[INF]MDS health message cleared (mds.?): Behind on trimming (525/128) 8/5/25 08:20 AM[WRN]overall HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:10 AM[WRN]mds.cephfs.node2.isqjza(mds.0): Behind on trimming (332/128) max_segments: 128, num_segments: 332 8/5/25 08:10 AM[WRN]mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming (332/128) max_segments: 128, num_segments: 332 8/5/25 08:10 AM[WRN][WRN] MDS_TRIM: 2 MDSs behind on trimming 8/5/25 08:10 AM[WRN]Health detail: HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:03 AM[WRN]Health check update: 2 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:03 AM[WRN]Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:00 AM[INF]overall HEALTH_OK #ceph fs status cephfs - 50 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephfs.node1.ojmpnk Reqs: 10 /s 305k 294k 91.8k 6818 0-s standby-replay cephfs.node2.isqjza Evts: 0 /s 551k 243k 90.6k 0 POOL TYPE USED AVAIL cephfs_metadata metadata 2630M 2413G cephfs_data data 12.7T 3620G STANDBY MDS cephfs.node3.vdicdn MDS version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable) # ceph versions { "mon": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "mgr": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 2 }, "osd": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 18 }, "mds": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "rgw": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 6 }, "overall": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 32 } } #ceph orch ps --daemon-type mds NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mds.cephfs.node1.ojmpnk rke-sh1-1 running (18h) 4m ago 19M 1709M - 19.2.2 4892a7ef541b 8dd8db30a1de mds.cephfs.node2.isqjza rke-sh1-2 running (18h) 2m ago 3y 1720M - 19.2.2 4892a7ef541b 7b9d5b692764 mds.cephfs.node3.vdicdn rke-sh1-3 running (18h) 108s ago 18M 27.9M - 19.2.2 4892a7ef541b d2de22a15e18 root@node1:~# ceph config show-with-defaults mds.cephfs.rke-sh1-3.vdicdn | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate" mds_cache_memory_limit 4294967296 default mds_cache_trim_decay_rate 1.000000 default mds_cache_trim_threshold 262144 default mds_recall_max_caps 30000 default mds_recall_max_decay_rate 1.500000 default root@node2:~# ceph config show-with-defaults mds.cephfs.rke-sh1-2.isqjza | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate" mds_cache_memory_limit 4294967296 default mds_cache_trim_decay_rate 1.000000 default mds_cache_trim_threshold 262144 default mds_recall_max_caps 30000 default mds_recall_max_decay_rate 1.500000 default root@node3:~# ceph config show-with-defaults mds.cephfs.rke-sh1-1.ojmpnk | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate" mds_cache_memory_limit 4294967296 default mds_cache_trim_decay_rate 1.000000 default mds_cache_trim_threshold 262144 default mds_recall_max_caps 30000 default mds_recall_max_decay_rate 1.500000 default # ceph mds stat cephfs:1 {0=cephfs.node1.ojmpnk=up:active} 1 up:standby-replay 1 up:standby Do you have an idea on what could happen ? Should I increate mds_cache_trim_decay_rate ? I saw the folloing issue : <https://tracker.ceph.com/issues/66948> Bug #66948: mon.a (mon.0) 326 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log - CephFS - Ceph ( <https://github.com/ceph/ceph/pull/60838> squid: mds: trim mdlog when segments exceed threshold and trim was idle by vshankar · Pull Request #60838 · ceph/ceph · GitHub ) maybe related ? Thanks for the help 😊 Best Regards, Edouard Fazenda. <https://www.csti.ch/> Swiss Cloud Provider Edouard Fazenda Technical Support <https://www.csti.ch/> Chemin du Curé-Desclouds, 2 CH-1226 Thonex +41 22 869 04 40 www.csti.ch

4 6

Squid 19.2.2 - mon_target_pg_per_osd change not applied
by Steven Vacaroaia 25 Jul '25

25 Jul '25

Hi, I wanted to increase the number of PG per OSD ....and did so by using ceph config set global mon_target_pg_per_osd 800 Although the OSD config has the new value , I am still unable to create pools that end up creating more than 250PG per OSD I have restarted the OSDs ...and the monitors ... Any ideas or suggestions for properly applying the change would be appreciated Steven [image: image.png]

4 3

Question about Dashboard and SSO (version 19.2.2)
by Taylor, Kevin P. (AOS) 25 Jul '25

25 Jul '25

Hi. I've spent quite a bit working through how to get SSO working with Keycloak using SAML, and I succeeded finally in getting that to go. We have 3 nodes in our test cluster and a round-robin DNS entry pointing to all 3 of them (just calling them node-1, node-2, and node-3 here. The configuration we did for this to work involved creating an SP cert and key, which I had to copy into the mgr containers in order to work: ceph dashboard sso setup saml2 https://node-1:8443 https://keycloak/realms/ceph/protocol/saml/descriptor username https://keycloak/realms/ceph /tmp/sp-cert.txt /tmp/sp-key.txt I'm able to log in fine (get sent to keycloak and directed back) as long as the management server is running on node-1, if I fail it over to node-2, I get prompted to go to keycloak and log in, but then receive this error in the browser: {"status": "415 Unsupported Media Type", "detail": "Expected an entity of content type application/json, text/javascript", "request_id": "dd5a9f68-a86c-462f-b25d-3d881e0fc350"} I know I didn't post all details of everything in this setup, but maybe my description of what's going on is useful. Does anyone have any ideas about this, or have suggestions as to getting proper failover + SSO for the dashboard? Thanks for any suggestions.

1 0

[ceph-user]ceph-ansible pacific || RGW integration with ceph dashboard
by Danish Khan 25 Jul '25

25 Jul '25

Dear Team, I am trying to integrate ceph dashboard with RGW so that I can create bucket from ceph dashboard. I am following below document for it: https://docs.ceph.com/en/pacific/mgr/dashboard/ when I run below command then it says "RGW credentials configured" #ceph dashboard set-rgw-credentials But my ceph dashboard is still not integrated with RGW and ceph dashboard still says that "The Object Gateway Service is not configured". In octopus release I used to run below commands which still works fine: $ ceph dashboard set-rgw-api-host <host> $ ceph dashboard set-rgw-api-port <port> but now these commands don't work in pacific release. Do we have any command in pacific release to specify rgw endpoint to integrate it with ceph dashboards? Regards, Danish Khan

2 1

CEPH performance all Flash lower than local
by Devender Singh 24 Jul '25

24 Jul '25

Hello all I tried doing some fio test on local disk(NVME) and ceph rbd. Why ceph is having low IO whereas it’s also on all NVME. What to tune to reach equal amount of IO? root@node01:~/fio-cdm# python3 fio-cdm ./ tests: 5, size: 1.0GiB, target: /root/fio-cdm 6.3GiB/64.4GiB |Name | Read(MB/s)| Write(MB/s)| |------------|------------|------------| |SEQ1M Q8 T1 | 8441.37| 3588.71| |SEQ1M Q1 T1 | 3074.86| 1172.46| |RND4K Q32T16| 723.65| 733.76| |. IOPS | 176671.80| 179141.74| |. latency us| 2892.49| 2839.37| |RND4K Q1 T1 | 71.05| 57.88| |. IOPS | 17347.13| 14131.57| |. latency us| 56.13| 66.40| ### When vm moved to ceph storage... root@node01:~/fio-cdm# python3 fio-cdm ./ tests: 5, size: 1.0GiB, target: /root/fio-cdm 9.3GiB/64.4GiB |Name | Read(MB/s)| Write(MB/s)| |------------|------------|------------| |SEQ1M Q8 T1 | 1681.40| 889.89| |SEQ1M Q1 T1 | 310.74| 852.11| |RND4K Q32T16| 403.04| 274.23| |. IOPS | 98397.32| 66951.49| |. latency us| 5196.98| 7637.10| |RND4K Q1 T1 | 4.69| 45.18| |. IOPS | 1146.17| 11029.39| |. latency us| 869.47| 87.50| Regards Dev

5 5

Re: squid 19.2.2 - cannot bootstrap - error writing to /tmp/monmap (21) Is a directory
by Steven Vacaroaia 24 Jul '25

24 Jul '25

Thanks Anthony The volume bind command looks like this -v /tmp/ceph-tmp-wvh4u18:/tmp/monmap:z so i guess the /tmp/ceph-tmp-xxxx never gets created so there is nothing to map What would be the solution though ...I tried removing and reinstalling docker Update and upgrade ( Ubuntu 24.04.2 LTS) On Thu, 24 Jul 2025 at 11:09, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > > How did you get that attachment through the list? Curious. > > I think /tmp/monmap is relative to the directory tree that the container > sees: > > -v /tmp/ceph-tmp-wvh4u18:/tmp/monmapz > > Because of the wrapping I’m not sure about the z on the next line, but -v > can be considered like a bind mount of the first path so that container > sees it on the second path. > > Now, as to what happened to /tmp/ceph-tmpxxxx, I can’t say. > > > > On Jul 24, 2025, at 10:53 AM, Steven Vacaroaia <stef97(a)gmail.com> wrote: > > > > Hi, > > > > I had to zap and start from scratch my cluster and now I cannot > bootstrap using cephadm > > > > The error is " monmaptool: error writing to /tmp/monmap: (21) Is a > directory " > > > > There is no /tmp/monmap directory though > > > > I also made sure there is nothing in /var/lib/ceph, no "ceph" process , > nothing in /etc/systemd/system/ceph* or /etc/ceph AND rebooted the server > > > > Any help will be appreciated > > > > Many thanks > > Steven > > > > <image.png> > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 3

squid 19.2.2 - cannot bootstrap - error writing to /tmp/monmap (21) Is a directory
by Steven Vacaroaia 24 Jul '25

24 Jul '25

Hi, I had to zap and start from scratch my cluster and now I cannot bootstrap using cephadm The error is " monmaptool: error writing to /tmp/monmap: (21) Is a directory " There is no /tmp/monmap directory though I also made sure there is nothing in /var/lib/ceph, no "ceph" process , nothing in /etc/systemd/system/ceph* or /etc/ceph AND rebooted the server Any help will be appreciated Many thanks Steven [image: image.png]

1 0

Ceph sf
by Szabo, Istvan (Agoda) 24 Jul '25

24 Jul '25

Hi, Somehow this output always made me confused. Can you confirm my calculation is good here? I have this output: --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED nvme 89 TiB 82 TiB 6.6 TiB 6.6 TiB 7.43 rbd 31 TiB 16 TiB 16 TiB 16 TiB 49.52 ssd 2.3 PiB 1.0 PiB 1.2 PiB 1.2 PiB 54.14 TOTAL 2.4 PiB 1.1 PiB 1.2 PiB 1.2 PiB 52.36 SSD: ec 4:2 rbd: repl 3 I want to know the stored data, which means the user side data. In ceph in EC 4:2 a 100MB object is stored on 150MB. In case of ssd, I want to know the used 1.2PiB means how much real user data? For me 1.2PiB used SSD means ~903TB user data on ec 4:2. In case of rbd, ~6TB. Am I correct? Ty ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Newby woes with ceph
by Stéphane Barthes 24 Jul '25

24 Jul '25

6 22

July Ceph Science Virtual User Group, new date.
by Mattia Belluco 24 Jul '25

24 Jul '25

Dear All, This is a personal reminder for the Ceph Science meeting. This month we will be having a Ceph science/research/big cluster call on Tuesday July 29th instead of Tuesday 22nd as per ceph.io calendar. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments or simply don't want to receive further announcements, you are most welcome to contact me. This round Tom Byrne from STFC will be presenting a short investigation about power and carbon cost of Ceph storage. Pad URL: https://pad.ceph.com/p/Ceph_Science_User_Group_20250729 Virtual event details: Jul 29th, 2025 9am Central US (CDT) 14:00 UTC 16:00 Central European (Summer Time) See the Ceph Community Calendar<https://ceph.io/en/community/meetups/> for the series information. (please double check the meeting time for your timezone!) Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph YouTube channel. To join the meeting on a computer or mobile phone: https://meet.jit.si/ceph-science-wg Slides from previous presentations are available here: https://github.com/cephscience/cephscience-slides About the user group: This is an informal open call of community members mostly from hpc/htc/research/big cluster environments, though anyone is welcome, where we discuss whatever is on our minds regarding Ceph. Updates, outages, features, maintenance, etc. There is no single set presenter for the length of the meeting, but it is somewhat moderated to keep the conversation flowing. See you next Tuesday, Mattia Belluco

1 0

July Ceph Science Virtual User Group, new date.
by Belluco Mattia (ID) 24 Jul '25

24 Jul '25

Dear All, This is a personal reminder for the Ceph Science meeting jointly organized by the person in cc on this message. This month we will be having a Ceph science/research/big cluster call on Tuesday July 29th instead of Tuesday 22nd as per ceph.io calendar. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments or simply don't want to receive further announcements, you are most welcome to contact us. This round Tom Byrne from STFC will be presenting a short investigation about power and carbon cost of Ceph storage. Pad URL: https://pad.ceph.com/p/Ceph_Science_User_Group_20250729 Virtual event details: Jul 29th, 2025 9am Central US (CDT) 14:00 UTC 16:00 Central European (Summer Time) See the Ceph Community Calendar<https://ceph.io/en/community/meetups/> for the series information. (please double check the meeting time for your timezone!) Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph YouTube channel. To join the meeting on a computer or mobile phone: https://meet.jit.si/ceph-science-wg Slides from previous presentations are available here: https://github.com/cephscience/cephscience-slides About the user group: This is an informal open call of community members mostly from hpc/htc/research/big cluster environments, though anyone is welcome, where we discuss whatever is on our minds regarding Ceph. Updates, outages, features, maintenance, etc. There is no single set presenter for the length of the meeting, but it is somewhat moderated to keep the conversation flowing. See you next Tuesday, Mattia Belluco -- ETH Zurich Scientific IT Services Binzmühlestrasse 130 (OCT G 35) 8092 Zurich, Switzerland

1 0

Re: Upgrading cephadm cluster
by Alan Murrell 23 Jul '25

23 Jul '25

Ok, thanks for the confirmation. Just as a followup, our Ceph nodes are running on Debian + Docker -- currently Debian 12.4. I assume that installing any OS-level updates should not affect anything Ceph-wise, as long as there are no issues with Docker? In other words, upgrading the underlying OS and Ceph are basically two very separate(d) things? Regards, Alan On Tue, 2025-07-22 at 18:34 -0400, Anthony D'Atri wrote: > Yes. > > > On Jul 22, 2025, at 6:33 PM, Alan Murrell <Alan(a)t-net.ca> wrote: > > > > Hello, > > > > I just wanted to check quickly, but feel I am being a little *too* > > cautious... > > > > We have a Ceph cluster (installed via cephadm) that is currently > > version 18.2.2. The "Upgrade" tab gives me an option to upgrade to > > 18.2.7 (its default selection), or I can upgrade to another version > > where I can choose "earlier" upgrades (from 18.2.3 to the offered > > 18.2.7) > > > > Is it generally OK and safe to "jump" upgrades like this (i.e. go > > straight from 18.2.2 straight to 18.2.7) or would it be recommended > > to > > upgrade in order (18.2.2 -> 18.2.3 -> 18.2.4 -> etc.)? > > > > I assume once I am on latest 18.2.7, there might be an option to > > upgrade to 19.2.x (not that I would necessarily want to do that > > straight away)? > > > > Thanks! :-) > > > > Regards, > > > > Alan > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io

5 5

Ceph OSD down (unable to mount object store)
by GLE, Vivien 23 Jul '25

23 Jul '25

Hi, I did a fio bench and believe that it destroy one of my OSD, this is the command used : fio --ioengine=libaio --filename=/dev/sda --direct=1 --sync=1 --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtime=60 --time_based --name=fio fio --ioengine=libaio --filename=/dev/sda --direct=1 --sync=1 --rw=write --bs=4M --numjobs=1 --iodepth=1 --runtime=60 --time_based --name=fio Log file after the command : 2025-07-23T08:41:44.448+0000 734bf7f6e680 1 bdev(0x59ba86d4ee00 /var/lib/ceph/osd/ceph-2/block) close 2025-07-23T08:41:44.719+0000 734bf7f6e680 1 bdev(0x59ba86d4ee00 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block 2025-07-23T08:41:44.719+0000 734bf7f6e680 0 bdev(0x59ba86d4ee00 /var/lib/ceph/osd/ceph-2/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-2/block failed: (22) Invalid argument 2025-07-23T08:41:44.720+0000 734bf7f6e680 1 bdev(0x59ba86d4ee00 /var/lib/ceph/osd/ceph-2/block) open size 1000203091968 (0xe8e0c00000, 932 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2025-07-23T08:41:44.723+0000 734bf7f6e680 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-2/block at offset 66: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] 2025-07-23T08:41:44.724+0000 734bf7f6e680 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-2/block at offset 4096: End of buffer [buffer:2] 2025-07-23T08:41:44.724+0000 734bf7f6e680 -1 bluestore(/var/lib/ceph/osd/ceph-2) _check_main_bdev_label not all labels read properly 2025-07-23T08:41:44.724+0000 734bf7f6e680 1 bdev(0x59ba86d4ee00 /var/lib/ceph/osd/ceph-2/block) close 2025-07-23T08:41:44.983+0000 734bf7f6e680 -1 osd.2 0 OSD:init: unable to mount object store 2025-07-23T08:41:44.983+0000 734bf7f6e680 -1 ** ERROR: osd init failed: (5) Input/output error After checking I saw that the block soft link might be wrong On a healthy osd : # ll /var/lib/ceph/cluster-id/osd.5/ total 72 drwx------ 2 167 167 4096 Jul 22 10:36 ./ drwx------ 12 167 167 4096 Jul 22 10:30 ../ lrwxrwxrwx 1 167 167 93 Jul 15 14:39 block -> /dev/ceph-c31f0e16-0460-4bc5-9470-468270b4c68a/osd-block-72aa3074-e2f9-45f8-a468-03c02d36f1de On my breaking osd : /var/lib/ceph/cluster-id/osd.2# ll total 72 drwx------ 2 167 167 4096 Jul 23 10:41 ./ drwx------ 11 167 167 4096 Jul 22 10:35 ../ lrwxrwxrwx 1 167 167 111 Jul 23 10:41 block -> /dev/mapper/ceph--bd0ca671--2b89--4530--be2b--f41443822a91-osd--block--342578c3--4603--4023--a564--8fca6dcc1040 Is there a correct way of changing it or I'm going in the wrong direction ? Thanks Vivien

3 5

Upgrading cephadm cluster
by Alan Murrell 22 Jul '25

22 Jul '25

Hello, I just wanted to check quickly, but feel I am being a little *too* cautious... We have a Ceph cluster (installed via cephadm) that is currently version 18.2.2. The "Upgrade" tab gives me an option to upgrade to 18.2.7 (its default selection), or I can upgrade to another version where I can choose "earlier" upgrades (from 18.2.3 to the offered 18.2.7) Is it generally OK and safe to "jump" upgrades like this (i.e. go straight from 18.2.2 straight to 18.2.7) or would it be recommended to upgrade in order (18.2.2 -> 18.2.3 -> 18.2.4 -> etc.)? I assume once I am on latest 18.2.7, there might be an option to upgrade to 19.2.x (not that I would necessarily want to do that straight away)? Thanks! :-) Regards, Alan

1 0

Rocky8 (el8) client for squid 19.2.2
by Steven Vacaroaia 22 Jul '25

22 Jul '25

Hi, I noticed there is no client /rpms for Rocky8 (el8) on 19.2.2 repository Would ceph-common for reef allow me to mount cephfs file systems without issues ? Many thanks Steven

5 7

19.2.2. OSDs are up but still showing error on daemon
by Devender Singh 22 Jul '25

22 Jul '25

Hello all Need some help to clean up below… # ceph orch ps |grep -v running NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID osd.0 host06n.van2.netskrt.org error 6m ago 11d - 58.5G <unknown> <unknown> <unknown> osd.1 host06n.van2.netskrt.org error 6m ago 11d - 58.5G <unknown> <unknown> <unknown> osd.2 host06n.van2.netskrt.org error 6m ago 11d - 58.5G <unknown> <unknown> <unknown> osd.3 host06n.van2.netskrt.org error 6m ago 11d - 58.5G <unknown> <unknown> <unknown> osd.4 host07n.van2.netskrt.org unknown 52s ago 11d - 59.5G <unknown> <unknown> <unknown> osd.5 host07n.van2.netskrt.org unknown 52s ago 11d - 59.5G <unknown> <unknown> <unknown> osd.6 host07n.van2.netskrt.org unknown 52s ago 11d - 59.5G <unknown> <unknown> <unknown> osd.7 host07n.van2.netskrt.org unknown 52s ago 11d - 59.5G <unknown> <unknown> <unknown> osd.8 host08n.van2.netskrt.org error 6m ago 11d - 60.2G <unknown> <unknown> <unknown> osd.9 host08n.van2.netskrt.org error 6m ago 11d - 60.2G <unknown> <unknown> <unknown> osd.10 host08n.van2.netskrt.org error 6m ago 11d - 60.2G <unknown> <unknown> <unknown> osd.11 host08n.van2.netskrt.org error 6m ago 11d - 60.2G <unknown> <unknown> <unknown> # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 167.66626 root default -3 55.88815 host host06n 0 ssd 13.97249 osd.0 up 1.00000 1.00000 1 ssd 13.97249 osd.1 up 1.00000 1.00000 2 ssd 13.97069 osd.2 up 1.00000 1.00000 3 ssd 13.97249 osd.3 up 1.00000 1.00000 -5 55.88995 host host07n 4 ssd 13.97249 osd.4 up 1.00000 1.00000 5 ssd 13.97249 osd.5 up 1.00000 1.00000 6 ssd 13.97249 osd.6 up 1.00000 1.00000 7 ssd 13.97249 osd.7 up 1.00000 1.00000 -7 55.88815 host host08n 8 ssd 13.97249 osd.8 up 1.00000 1.00000 9 ssd 13.97249 osd.9 up 1.00000 1.00000 10 ssd 13.97069 osd.10 up 1.00000 1.00000 11 ssd 13.97249 osd.11 up 1.00000 1.00000 Regards Dev

1 0

Re: RGW: is it possible to restrict a user access to a realm?
by Wissem MIMOUNA - Ceph Users 22 Jul '25

22 Jul '25

Hi , On a Zone there are a list of configured pools the contains users information ( users_keys_pool , user_uid_pool , ...) so the user information is stored on pools attached to a zone . Regards On 7/7/25 18:54, Michel Jouvin wrote: > Hi Wissem, > > Your first answer is a good approach too. It's true that I was looking > at a way to bind users to a realm, zonegroup or zone but I don't see > one. I don't think users are bound to zone in fact as there is no > related attribute I'm the user info if I'm right. > > Michel > Sent from my mobile > > Le 7 juillet 2025 18:25:31 Wissem MIMOUNA - Ceph Users > <ceph-users(a)ik.me> a écrit : > >> I miss understood your question (first): >> >> As I understand a user belong to a zone ( and zone belong to zone group >> and zone group to realm ) , so it's not possible to restrict users to >> realm ( because each user belong to the realm where it was created in ) >> , unless you try to migrate all pools ( and user metadata and data ) to >> one realm ( I don't know if it's possible !? >> >> ceph doc say this >> >> "A realm is a globally unique namespace that consists of one or more >> zonegroups. Zonegroups contain one or more zones. Zones contain buckets. >> Buckets contain objects." >> >> >> Regards >> >> >> >> On 7/7/25 17:35, Wissem MIMOUNA - Ceph Users wrote: >>> Hi Michel, >>> >>> >>> By default each user is isolated on its namespace ( buckets of one >>> user cannot be accessed by other users , unless you allow that ) , for >>> the accounts it's different as an account it's an isolated namespaces >>> that can have multiple users ( inside the same account - see aws iam ) . >>> >>> Each RGW zone has a 'realm_id' attached to it , so if you want that >>> some users use a specific realm for storage ( creating buckets ... ) , >>> then modifiy users property 'default_placement' to use the one from >>> the zone ( the zone that has the relam_id you want ) . >>> >>> >>> Regards >>> >>> On 7/7/25 17:19, Michel Jouvin wrote: >>>> Hi, >>>> >>>> We have several RGW realms hosted in the same Ceph cluster. Looking >>>> at how to restrict access to one realm to some users (among all >>>> existing ones), I don't find the information. Looking at >>>> user/realm/zonegroup/zone parameters, I don't see anything that would >>>> allow this. I saw in >>>> https://docs.ceph.com/en/latest/radosgw/account/#radosgw-account a >>>> few words about tenant isolation but it is not clear for me if it is >>>> the same thing and how you achieve it. >>>> >>>> Thanks in advance for any hint! >>>> >>>> Best regards, >>>> >>>> Michel >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 3

Ceph Foundation 2025: Community, Collaboration, and What’s Next
by Anthony Middleton 22 Jul '25

22 Jul '25

Hey everyone! Our latest blog shares the milestones we've achieved, the collaborations that've shaped us, and the campaigns driving us forward in 2025, including upcoming initiatives such as Cephalocon 2025 and the Ceph User Story projects. Read the full blog post here: https://ceph.io/en/news/blog/2025/Ceph-Foundation-2025/ Best, Anthony Middleton Ceph Community Manager community-manager(a)ceph.io

1 0

Cache removal cause vms to crash
by Vishnu Bhaskar 22 Jul '25

22 Jul '25

Hi, I was attempting to increase the cache pool's PG value. When I used the command `ceph osd pool set volumes_cache 64`, I encountered the error "Error EPERM: splits in cache pools must be followed by scrubs and leave sufficient free space to avoid overfilling. use --yes-i-really-mean-it to force." To address this, I decided to remove the cache from the volumes pool and followed these steps: 1. Set the cache mode to `readproxy`: `ceph osd tier cache-mode volumes_cache readproxy` 2. Started flushing dirty objects: `rados -p volumes_cache cache-flush-evict-all` I stopped this flush prematurely due to the large number of dirty objects. 3. Removed the overlay from the base pool: `ceph osd tier remove-overlay volumes` 4. Resumed flushing the remaining objects in the cache pool: `rados -p volumes_cache cache-flush-evict-all` Error occurred while flushing: rbd_data.f2804088f4e54a.000000000001cb71 failed to flush /rbd_data.f2804088f4e54a.000000000001cb71: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000008b4 failed to flush /rbd_data.f2804088f4e54a.00000000000008b4: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000027 failed to flush /rbd_data.f2804088f4e54a.0000000000000027: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000061 failed to flush /rbd_data.f2804088f4e54a.0000000000000061: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ede6 failed to flush /rbd_data.f2804088f4e54a.000000000001ede6: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000004dd failed to flush /rbd_data.f2804088f4e54a.00000000000004dd: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ebd6 failed to flush /rbd_data.f2804088f4e54a.000000000001ebd6: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000000e7 failed to flush /rbd_data.f2804088f4e54a.00000000000000e7: (2) No such file or directory rbd_data.5f35353775dd58.000000000000436c failed to flush /rbd_data.5f35353775dd58.000000000000436c: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001eace failed to flush /rbd_data.f2804088f4e54a.000000000001eace: (2) No such file or directory rbd_data.5f35353775dd58.0000000000003e7d failed to flush /rbd_data.5f35353775dd58.0000000000003e7d: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001eb6a failed to flush /rbd_data.f2804088f4e54a.000000000001eb6a: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000663 failed to flush /rbd_data.f2804088f4e54a.0000000000000663: (2) No such file or directory rbd_data.5f35353775dd58.0000000000000326 failed to flush /rbd_data.5f35353775dd58.0000000000000326: (2) No such file or directory rbd_data.5f35353775dd58.000000000000031d failed to flush /rbd_data.5f35353775dd58.000000000000031d: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000edd failed to flush /rbd_data.f2804088f4e54a.0000000000000edd: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ef09 failed to flush /rbd_data.f2804088f4e54a.000000000001ef09: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001eef1 failed to flush /rbd_data.f2804088f4e54a.000000000001eef1: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000983 failed to flush /rbd_data.f2804088f4e54a.0000000000000983: (2) No such file or directory rbd_data.5f35353775dd58.000000000001c05c failed to flush /rbd_data.5f35353775dd58.000000000001c05c: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ec9f failed to flush /rbd_data.f2804088f4e54a.000000000001ec9f: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000002 failed to flush /rbd_data.f2804088f4e54a.0000000000000002: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000014726 failed to flush /rbd_data.f2804088f4e54a.0000000000014726: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000031e failed to flush /rbd_data.f2804088f4e54a.000000000000031e: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000007f4 failed to flush /rbd_data.f2804088f4e54a.00000000000007f4: (2) No such file or directory rbd_data.5f35353775dd58.000000000001a7c5 failed to flush /rbd_data.5f35353775dd58.000000000001a7c5: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000658 failed to flush /rbd_data.f2804088f4e54a.0000000000000658: (2) No such file or directory rbd_data.5f35353775dd58.0000000000007f38 failed to flush /rbd_data.5f35353775dd58.0000000000007f38: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000081e failed to flush /rbd_data.f2804088f4e54a.000000000000081e: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000495 failed to flush /rbd_data.f2804088f4e54a.0000000000000495: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000000 failed to flush /rbd_data.f2804088f4e54a.0000000000000000: (2) No such file or directory rbd_data.5f35353775dd58.0000000000002159 failed to flush /rbd_data.5f35353775dd58.0000000000002159: (2) No such file or directory rbd_data.5f35353775dd58.000000000000fdb1 failed to flush /rbd_data.5f35353775dd58.000000000000fdb1: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000101 failed to flush /rbd_data.f2804088f4e54a.0000000000000101: (2) No such file or directory rbd_data.5f35353775dd58.000000000000e6f4 failed to flush /rbd_data.5f35353775dd58.000000000000e6f4: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000fa6 failed to flush /rbd_data.f2804088f4e54a.0000000000000fa6: (2) No such file or directory rbd_data.5f35353775dd58.000000000000bf6c failed to flush /rbd_data.5f35353775dd58.000000000000bf6c: (2) No such file or directory rbd_data.5f35353775dd58.00000000000093cc failed to flush /rbd_data.5f35353775dd58.00000000000093cc: (2) No such file or directory rbd_data.5f35353775dd58.00000000000021a0 failed to flush /rbd_data.5f35353775dd58.00000000000021a0: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001eb6c failed to flush /rbd_data.f2804088f4e54a.000000000001eb6c: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000125d6 failed to flush /rbd_data.f2804088f4e54a.00000000000125d6: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000153 failed to flush /rbd_data.f2804088f4e54a.0000000000000153: (2) No such file or directory rbd_data.5f35353775dd58.000000000001627a failed to flush /rbd_data.5f35353775dd58.000000000001627a: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000024 failed to flush /rbd_data.f2804088f4e54a.0000000000000024: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000825 failed to flush /rbd_data.f2804088f4e54a.0000000000000825: (2) No such file or directory rbd_data.5f35353775dd58.00000000000021dc failed to flush /rbd_data.5f35353775dd58.00000000000021dc: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000000c failed to flush /rbd_data.f2804088f4e54a.000000000000000c: (2) No such file or directory rbd_data.5f35353775dd58.0000000000009454 failed to flush /rbd_data.5f35353775dd58.0000000000009454: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000314 failed to flush /rbd_data.f2804088f4e54a.0000000000000314: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000f98 failed to flush /rbd_data.f2804088f4e54a.0000000000000f98: (2) No such file or directory rbd_data.5f35353775dd58.00000000000002fe failed to flush /rbd_data.5f35353775dd58.00000000000002fe: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000002f failed to flush /rbd_data.f2804088f4e54a.000000000000002f: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000030 failed to flush /rbd_data.f2804088f4e54a.0000000000000030: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000c3d failed to flush /rbd_data.f2804088f4e54a.0000000000000c3d: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000049 failed to flush /rbd_data.f2804088f4e54a.0000000000000049: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ecdc failed to flush /rbd_data.f2804088f4e54a.000000000001ecdc: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000032d failed to flush /rbd_data.f2804088f4e54a.000000000000032d: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000666 failed to flush /rbd_data.f2804088f4e54a.0000000000000666: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000031 failed to flush /rbd_data.f2804088f4e54a.0000000000000031: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000053 failed to flush /rbd_data.f2804088f4e54a.0000000000000053: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000326 failed to flush /rbd_data.f2804088f4e54a.0000000000000326: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000100 failed to flush /rbd_data.f2804088f4e54a.0000000000000100: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001b3dd failed to flush /rbd_data.f2804088f4e54a.000000000001b3dd: (2) No such file or directory rbd_data.5f35353775dd58.0000000000011395 failed to flush /rbd_data.5f35353775dd58.0000000000011395: (2) No such file or directory rbd_data.5f35353775dd58.00000000000002fd failed to flush /rbd_data.5f35353775dd58.00000000000002fd: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000008b2 failed to flush /rbd_data.f2804088f4e54a.00000000000008b2: (2) No such file or directory rbd_data.5f35353775dd58.0000000000000300 failed to flush /rbd_data.5f35353775dd58.0000000000000300: (2) No such file or directory rbd_data.5f35353775dd58.00000000000002ff failed to flush /rbd_data.5f35353775dd58.00000000000002ff: (2) No such file or directory rbd_data.5f35353775dd58.000000000001ed9e failed to flush /rbd_data.5f35353775dd58.000000000001ed9e: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000878 failed to flush /rbd_data.f2804088f4e54a.0000000000000878: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000316 failed to flush /rbd_data.f2804088f4e54a.0000000000000316: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000008b3 failed to flush /rbd_data.f2804088f4e54a.00000000000008b3: (2) No such file or directory rbd_data.5f35353775dd58.0000000000000308 failed to flush /rbd_data.5f35353775dd58.0000000000000308: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001eb70 failed to flush /rbd_data.f2804088f4e54a.000000000001eb70: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000086d failed to flush /rbd_data.f2804088f4e54a.000000000000086d: (2) No such file or directory rbd_data.5f35353775dd58.0000000000013b1f failed to flush /rbd_data.5f35353775dd58.0000000000013b1f: (2) No such file or directory rbd_data.5f35353775dd58.0000000000000325 failed to flush /rbd_data.5f35353775dd58.0000000000000325: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001eb2f failed to flush /rbd_data.f2804088f4e54a.000000000001eb2f: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001b413 failed to flush /rbd_data.f2804088f4e54a.000000000001b413: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000079b failed to flush /rbd_data.f2804088f4e54a.000000000000079b: (2) No such file or directory rbd_data.5f35353775dd58.0000000000000309 failed to flush /rbd_data.5f35353775dd58.0000000000000309: (2) No such file or directory rbd_data.5f35353775dd58.000000000001edc6 failed to flush /rbd_data.5f35353775dd58.000000000001edc6: (2) No such file or directory rbd_data.5f35353775dd58.00000000000104bd failed to flush /rbd_data.5f35353775dd58.00000000000104bd: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001eb2a failed to flush /rbd_data.f2804088f4e54a.000000000001eb2a: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ec77 failed to flush /rbd_data.f2804088f4e54a.000000000001ec77: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000001a failed to flush /rbd_data.f2804088f4e54a.000000000000001a: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000e5d failed to flush /rbd_data.f2804088f4e54a.0000000000000e5d: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000e56 failed to flush /rbd_data.f2804088f4e54a.0000000000000e56: (2) No such file or directory rbd_data.5f35353775dd58.00000000000000a7 failed to flush /rbd_data.5f35353775dd58.00000000000000a7: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000d3e failed to flush /rbd_data.f2804088f4e54a.0000000000000d3e: (2) No such file or directory rbd_data.5f35353775dd58.000000000000f5cb failed to flush /rbd_data.5f35353775dd58.000000000000f5cb: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001eda7 failed to flush /rbd_data.f2804088f4e54a.000000000001eda7: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000317 failed to flush /rbd_data.f2804088f4e54a.0000000000000317: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001eb14 failed to flush /rbd_data.f2804088f4e54a.000000000001eb14: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000010c failed to flush /rbd_data.f2804088f4e54a.000000000000010c: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000331 failed to flush /rbd_data.f2804088f4e54a.0000000000000331: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000659 failed to flush /rbd_data.f2804088f4e54a.0000000000000659: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000070 failed to flush /rbd_data.f2804088f4e54a.0000000000000070: (2) No such file or directory rbd_data.5f35353775dd58.0000000000000303 failed to flush /rbd_data.5f35353775dd58.0000000000000303: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000063 failed to flush /rbd_data.f2804088f4e54a.0000000000000063: (2) No such file or directory rbd_data.5f35353775dd58.0000000000000306 failed to flush /rbd_data.5f35353775dd58.0000000000000306: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ecdb failed to flush /rbd_data.f2804088f4e54a.000000000001ecdb: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001eb34 failed to flush /rbd_data.f2804088f4e54a.000000000001eb34: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000063c failed to flush /rbd_data.f2804088f4e54a.000000000000063c: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000876 failed to flush /rbd_data.f2804088f4e54a.0000000000000876: (2) No such file or directory rbd_data.5f35353775dd58.0000000000009453 failed to flush /rbd_data.5f35353775dd58.0000000000009453: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000008ad failed to flush /rbd_data.f2804088f4e54a.00000000000008ad: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000672 failed to flush /rbd_data.f2804088f4e54a.0000000000000672: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ede8 failed to flush /rbd_data.f2804088f4e54a.000000000001ede8: (2) No such file or directory rbd_data.5f35353775dd58.000000000000f5f8 failed to flush /rbd_data.5f35353775dd58.000000000000f5f8: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001efde failed to flush /rbd_data.f2804088f4e54a.000000000001efde: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000edc failed to flush /rbd_data.f2804088f4e54a.0000000000000edc: (2) No such file or directory rbd_data.5f35353775dd58.000000000000bf45 failed to flush /rbd_data.5f35353775dd58.000000000000bf45: (2) No such file or directory rbd_data.5f35353775dd58.00000000000002fc failed to flush /rbd_data.5f35353775dd58.00000000000002fc: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ec9d failed to flush /rbd_data.f2804088f4e54a.000000000001ec9d: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ec17 failed to flush /rbd_data.f2804088f4e54a.000000000001ec17: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000000b1 failed to flush /rbd_data.f2804088f4e54a.00000000000000b1: (2) No such file or directory rbd_data.5f35353775dd58.000000000000f21d failed to flush /rbd_data.5f35353775dd58.000000000000f21d: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000004bc failed to flush /rbd_data.f2804088f4e54a.00000000000004bc: (2) No such file or directory rbd_data.5f35353775dd58.000000000000030a failed to flush /rbd_data.5f35353775dd58.000000000000030a: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ee03 failed to flush /rbd_data.f2804088f4e54a.000000000001ee03: (2) No such file or directory rbd_data.5f35353775dd58.0000000000006110 failed to flush /rbd_data.5f35353775dd58.0000000000006110: (2) No such file or directory rbd_data.5f35353775dd58.00000000000074d3 failed to flush /rbd_data.5f35353775dd58.00000000000074d3: (2) No such file or directory rbd_data.5f35353775dd58.000000000000618f failed to flush /rbd_data.5f35353775dd58.000000000000618f: (2) No such file or directory rbd_data.5f35353775dd58.0000000000013aef failed to flush /rbd_data.5f35353775dd58.0000000000013aef: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000318 failed to flush /rbd_data.f2804088f4e54a.0000000000000318: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000009db failed to flush /rbd_data.f2804088f4e54a.00000000000009db: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000ede failed to flush /rbd_data.f2804088f4e54a.0000000000000ede: (2) No such file or directory rbd_data.5f35353775dd58.000000000000030b failed to flush /rbd_data.5f35353775dd58.000000000000030b: (2) No such file or directory rbd_data.5f35353775dd58.0000000000011369 failed to flush /rbd_data.5f35353775dd58.0000000000011369: (2) No such file or directory rbd_data.5f35353775dd58.0000000000018044 failed to flush /rbd_data.5f35353775dd58.0000000000018044: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000330 failed to flush /rbd_data.f2804088f4e54a.0000000000000330: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000110 failed to flush /rbd_data.f2804088f4e54a.0000000000000110: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000492 failed to flush /rbd_data.f2804088f4e54a.0000000000000492: (2) No such file or directory rbd_data.5f35353775dd58.0000000000000307 failed to flush /rbd_data.5f35353775dd58.0000000000000307: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001eb6b failed to flush /rbd_data.f2804088f4e54a.000000000001eb6b: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000018c failed to flush /rbd_data.f2804088f4e54a.000000000000018c: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001eb30 failed to flush /rbd_data.f2804088f4e54a.000000000001eb30: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ebd7 failed to flush /rbd_data.f2804088f4e54a.000000000001ebd7: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001eca3 failed to flush /rbd_data.f2804088f4e54a.000000000001eca3: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000004e8 failed to flush /rbd_data.f2804088f4e54a.00000000000004e8: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000004c5 failed to flush /rbd_data.f2804088f4e54a.00000000000004c5: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000e5a failed to flush /rbd_data.f2804088f4e54a.0000000000000e5a: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000315 failed to flush /rbd_data.f2804088f4e54a.0000000000000315: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000004d9 failed to flush /rbd_data.f2804088f4e54a.00000000000004d9: (2) No such file or directory rbd_data.5f35353775dd58.00000000000052c0 failed to flush /rbd_data.5f35353775dd58.00000000000052c0: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ede7 failed to flush /rbd_data.f2804088f4e54a.000000000001ede7: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000004c2 failed to flush /rbd_data.f2804088f4e54a.00000000000004c2: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000a29 failed to flush /rbd_data.f2804088f4e54a.0000000000000a29: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000494 failed to flush /rbd_data.f2804088f4e54a.0000000000000494: (2) No such file or directory rbd_data.5f35353775dd58.0000000000013afe failed to flush /rbd_data.5f35353775dd58.0000000000013afe: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001a50e failed to flush /rbd_data.f2804088f4e54a.000000000001a50e: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001eb31 failed to flush /rbd_data.f2804088f4e54a.000000000001eb31: (2) No such file or directory rbd_data.5f35353775dd58.0000000000009799 failed to flush /rbd_data.5f35353775dd58.0000000000009799: (2) No such file or directory rbd_data.5f35353775dd58.0000000000003043 failed to flush /rbd_data.5f35353775dd58.0000000000003043: (2) No such file or directory rbd_data.5f35353775dd58.000000000001eea2 failed to flush /rbd_data.5f35353775dd58.000000000001eea2: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000008b7 failed to flush /rbd_data.f2804088f4e54a.00000000000008b7: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ec35 failed to flush /rbd_data.f2804088f4e54a.000000000001ec35: (2) No such file or directory rbd_data.5f35353775dd58.000000000000031e failed to flush /rbd_data.5f35353775dd58.000000000000031e: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000d24 failed to flush /rbd_data.f2804088f4e54a.0000000000000d24: (2) No such file or directory rbd_data.5f35353775dd58.000000000001bb7c failed to flush /rbd_data.5f35353775dd58.000000000001bb7c: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000006c failed to flush /rbd_data.f2804088f4e54a.000000000000006c: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ede0 failed to flush /rbd_data.f2804088f4e54a.000000000001ede0: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ede5 failed to flush /rbd_data.f2804088f4e54a.000000000001ede5: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ede3 failed to flush /rbd_data.f2804088f4e54a.000000000001ede3: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000002e failed to flush /rbd_data.f2804088f4e54a.000000000000002e: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000e3c failed to flush /rbd_data.f2804088f4e54a.0000000000000e3c: (2) No such file or directory rbd_data.5f35353775dd58.000000000000f108 failed to flush /rbd_data.5f35353775dd58.000000000000f108: (2) No such file or directory rbd_data.5f35353775dd58.000000000001a7a1 failed to flush /rbd_data.5f35353775dd58.000000000001a7a1: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000008b0 failed to flush /rbd_data.f2804088f4e54a.00000000000008b0: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000004b6 failed to flush /rbd_data.f2804088f4e54a.00000000000004b6: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000005a9 failed to flush /rbd_data.f2804088f4e54a.00000000000005a9: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000032 failed to flush /rbd_data.f2804088f4e54a.0000000000000032: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000008b6 failed to flush /rbd_data.f2804088f4e54a.00000000000008b6: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000013b failed to flush /rbd_data.f2804088f4e54a.000000000000013b: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000f90 failed to flush /rbd_data.f2804088f4e54a.0000000000000f90: (2) No such file or directory rbd_data.5f35353775dd58.0000000000000320 failed to flush /rbd_data.5f35353775dd58.0000000000000320: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000004bb failed to flush /rbd_data.f2804088f4e54a.00000000000004bb: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000813 failed to flush /rbd_data.f2804088f4e54a.0000000000000813: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000004b0 failed to flush /rbd_data.f2804088f4e54a.00000000000004b0: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000783 failed to flush /rbd_data.f2804088f4e54a.0000000000000783: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001eb33 failed to flush /rbd_data.f2804088f4e54a.000000000001eb33: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000009d failed to flush /rbd_data.f2804088f4e54a.000000000000009d: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000154 failed to flush /rbd_data.f2804088f4e54a.0000000000000154: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000001e failed to flush /rbd_data.f2804088f4e54a.000000000000001e: (2) No such file or directory rbd_data.5f35353775dd58.00000000000002f6 failed to flush /rbd_data.5f35353775dd58.00000000000002f6: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000fa1 failed to flush /rbd_data.f2804088f4e54a.0000000000000fa1: (2) No such file or directory rbd_data.5f35353775dd58.000000000001e7df failed to flush /rbd_data.5f35353775dd58.000000000001e7df: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000006 failed to flush /rbd_data.f2804088f4e54a.0000000000000006: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001eb29 failed to flush /rbd_data.f2804088f4e54a.000000000001eb29: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000006b failed to flush /rbd_data.f2804088f4e54a.000000000000006b: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000005cd failed to flush /rbd_data.f2804088f4e54a.00000000000005cd: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000319 failed to flush /rbd_data.f2804088f4e54a.0000000000000319: (2) No such file or directory rbd_data.5f35353775dd58.000000000000302b failed to flush /rbd_data.5f35353775dd58.000000000000302b: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ede2 failed to flush /rbd_data.f2804088f4e54a.000000000001ede2: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000e0a failed to flush /rbd_data.f2804088f4e54a.0000000000000e0a: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000005cf failed to flush /rbd_data.f2804088f4e54a.00000000000005cf: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000320 failed to flush /rbd_data.f2804088f4e54a.0000000000000320: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000e58 failed to flush /rbd_data.f2804088f4e54a.0000000000000e58: (2) No such file or directory rbd_data.5f35353775dd58.000000000001049a failed to flush /rbd_data.5f35353775dd58.000000000001049a: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001eca7 failed to flush /rbd_data.f2804088f4e54a.000000000001eca7: (2) No such file or directory rbd_data.5f35353775dd58.000000000000e22f failed to flush /rbd_data.5f35353775dd58.000000000000e22f: (2) No such file or directory rbd_data.5f35353775dd58.000000000001801b failed to flush /rbd_data.5f35353775dd58.000000000001801b: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000812 failed to flush /rbd_data.f2804088f4e54a.0000000000000812: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000031a failed to flush /rbd_data.f2804088f4e54a.000000000000031a: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000072 failed to flush /rbd_data.f2804088f4e54a.0000000000000072: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000008 failed to flush /rbd_data.f2804088f4e54a.0000000000000008: (2) No such file or directory rbd_data.5f35353775dd58.000000000001e80b failed to flush /rbd_data.5f35353775dd58.000000000001e80b: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000055 failed to flush /rbd_data.f2804088f4e54a.0000000000000055: (2) No such file or directory rbd_data.5f35353775dd58.00000000000001aa failed to flush /rbd_data.5f35353775dd58.00000000000001aa: (2) No such file or directory rbd_data.5f35353775dd58.000000000000e202 failed to flush /rbd_data.5f35353775dd58.000000000000e202: (2) No such file or directory rbd_data.5f35353775dd58.0000000000000302 failed to flush /rbd_data.5f35353775dd58.0000000000000302: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ede4 failed to flush /rbd_data.f2804088f4e54a.000000000001ede4: (2) No such file or directory rbd_data.5f35353775dd58.0000000000007f74 failed to flush /rbd_data.5f35353775dd58.0000000000007f74: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000152 failed to flush /rbd_data.f2804088f4e54a.0000000000000152: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000019 failed to flush /rbd_data.f2804088f4e54a.0000000000000019: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000008af failed to flush /rbd_data.f2804088f4e54a.00000000000008af: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000004dc failed to flush /rbd_data.f2804088f4e54a.00000000000004dc: (2) No such file or directory rbd_data.5f35353775dd58.000000000000030c failed to flush /rbd_data.5f35353775dd58.000000000000030c: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001a4f2 failed to flush /rbd_data.f2804088f4e54a.000000000001a4f2: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000086f failed to flush /rbd_data.f2804088f4e54a.000000000000086f: (2) No such file or directory rbd_data.5f35353775dd58.0000000000000305 failed to flush /rbd_data.5f35353775dd58.0000000000000305: (2) No such file or directory rbd_data.5f35353775dd58.0000000000000301 failed to flush /rbd_data.5f35353775dd58.0000000000000301: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000028 failed to flush /rbd_data.f2804088f4e54a.0000000000000028: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ede9 failed to flush /rbd_data.f2804088f4e54a.000000000001ede9: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000032c failed to flush /rbd_data.f2804088f4e54a.000000000000032c: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ec4b failed to flush /rbd_data.f2804088f4e54a.000000000001ec4b: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000006f failed to flush /rbd_data.f2804088f4e54a.000000000000006f: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000033 failed to flush /rbd_data.f2804088f4e54a.0000000000000033: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000e59 failed to flush /rbd_data.f2804088f4e54a.0000000000000e59: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000001228 failed to flush /rbd_data.f2804088f4e54a.0000000000001228: (2) No such file or directory rbd_data.5f35353775dd58.0000000000000352 failed to flush /rbd_data.5f35353775dd58.0000000000000352: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000008bd failed to flush /rbd_data.f2804088f4e54a.00000000000008bd: (2) No such file or directory rbd_data.5f35353775dd58.0000000000009433 failed to flush /rbd_data.5f35353775dd58.0000000000009433: (2) No such file or directory rbd_data.f2804088f4e54a.00000000000008b5 failed to flush /rbd_data.f2804088f4e54a.00000000000008b5: (2) No such file or directory rbd_data.5f35353775dd58.000000000000e6f1 failed to flush /rbd_data.5f35353775dd58.000000000000e6f1: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000782 failed to flush /rbd_data.f2804088f4e54a.0000000000000782: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ed02 failed to flush /rbd_data.f2804088f4e54a.000000000001ed02: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001eb6e failed to flush /rbd_data.f2804088f4e54a.000000000001eb6e: (2) No such file or directory rbd_data.5f35353775dd58.0000000000003efa failed to flush /rbd_data.5f35353775dd58.0000000000003efa: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000001315 failed to flush /rbd_data.f2804088f4e54a.0000000000001315: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ed8a failed to flush /rbd_data.f2804088f4e54a.000000000001ed8a: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000022 failed to flush /rbd_data.f2804088f4e54a.0000000000000022: (2) No such file or directory rbd_data.5f35353775dd58.0000000000009308 failed to flush /rbd_data.5f35353775dd58.0000000000009308: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000fa2 failed to flush /rbd_data.f2804088f4e54a.0000000000000fa2: (2) No such file or directory rbd_data.5f35353775dd58.000000000000031c failed to flush /rbd_data.5f35353775dd58.000000000000031c: (2) No such file or directory rbd_data.5f35353775dd58.000000000000f238 failed to flush /rbd_data.5f35353775dd58.000000000000f238: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000313 failed to flush /rbd_data.f2804088f4e54a.0000000000000313: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000d4f failed to flush /rbd_data.f2804088f4e54a.0000000000000d4f: (2) No such file or directory rbd_data.5f35353775dd58.0000000000002abc failed to flush /rbd_data.5f35353775dd58.0000000000002abc: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000f75 failed to flush /rbd_data.f2804088f4e54a.0000000000000f75: (2) No such file or directory rbd_data.5f35353775dd58.00000000000000a8 failed to flush /rbd_data.5f35353775dd58.00000000000000a8: (2) No such file or directory rbd_data.5f35353775dd58.000000000000e721 failed to flush /rbd_data.5f35353775dd58.000000000000e721: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000085e failed to flush /rbd_data.f2804088f4e54a.000000000000085e: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001e10d failed to flush /rbd_data.f2804088f4e54a.000000000001e10d: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000103 failed to flush /rbd_data.f2804088f4e54a.0000000000000103: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ecbb failed to flush /rbd_data.f2804088f4e54a.000000000001ecbb: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000073 failed to flush /rbd_data.f2804088f4e54a.0000000000000073: (2) No such file or directory rbd_data.5f35353775dd58.000000000001ef05 failed to flush /rbd_data.5f35353775dd58.000000000001ef05: (2) No such file or directory rbd_data.5f35353775dd58.0000000000007552 failed to flush /rbd_data.5f35353775dd58.0000000000007552: (2) No such file or directory rbd_data.5f35353775dd58.0000000000000304 failed to flush /rbd_data.5f35353775dd58.0000000000000304: (2) No such file or directory rbd_data.f2804088f4e54a.000000000000031f failed to flush /rbd_data.f2804088f4e54a.000000000000031f: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000e0b failed to flush /rbd_data.f2804088f4e54a.0000000000000e0b: (2) No such file or directory rbd_data.f2804088f4e54a.0000000000000fa7 failed to flush /rbd_data.f2804088f4e54a.0000000000000fa7: (2) No such file or directory rbd_data.f2804088f4e54a.000000000001ef32 failed to flush /rbd_data.f2804088f4e54a.000000000001ef32: (2) No such file or directory cache-flush-evict-all finished with errors Is there any way we can resolve this issue? Thanks and Regards Vishnu Bhaskar Acceleron Labs Pvt Ltd Bangalore, India

2 10

ceph-volume partial failure with multiple OSDs per device
by Elias Carter 22 Jul '25

22 Jul '25

I provision OSDs with the following command: ``` ceph-volume --log-path /tmp/ceph-log lvm batch --prepare --bluestore --yes --dmcrypt --osds-per-device 2 /dev/nvme0n1 --crush-device-class nvme ``` What I have observed is that sometimes ceph-volume will provision the first OSD successfully, but then it will fail to provision the second OSD. When this happens ceph-volume does not attempt to clean up the first OSD which was created. My question is what should be done in this scenario? Can I re-run the ceph-volume command and it is smart enough to see the first OSD is already provisioned and then create the missing second OSD? Or should I zap and purge the OSD which was created and then re-run the ceph-volume command? Elias

2 1

HELP! Cluster usage increased after adding new nodes/osd's
by mhnx 22 Jul '25

22 Jul '25

Hello! Few years ago I build a "dc-a:12 + dc-b:12 = 24" node ceph cluster with Nautilus v14.2.16 A year ago the cluster upgraded to Octopus and it was running fine. Recently I added 4+4=8 new nodes with identical hardware and SSD drives. When I created OSD's with Octopus, The cluster usage increased from %50 to %78!! The weird problem is, the new OSD's become nearfull and hold more size even if they have the same or less amount of PG's. I had to reweight new OSD's to 0.9 to make them equal size usage.. I increased the PG count 8192 to 16384 and ran balancer, it became worse and I have %84 usage now! I guess OSD or PG code changed between nautilus <-> octopus and it generates this problem. Can anyone help me with experience or knowledge about this? What should I do? My solution idea: I'm thinking of destroy and re-create old OSD's as a solution but I need to re-create 144x3.8TB Sas SSD OSD's and it means 4-5 days of maintenance. Also I have 2 osd per drive because it was recommended at Nautilus times. How about this? Should I keep the config or should I use 1 osd per 3.8TB SAS SSD ? What is the recommendation for Octopus and Quincy? - Best regards.

4 4

Re: squid 19.2.2 - cannot remove 'unknown" OSD
by Steven Vacaroaia 21 Jul '25

21 Jul '25

Hi Anthony, There was a leftover folder which I renamed I have also disablled the service and been able to remove it from crush and delete auth but still is showing when using ceph orch ps and still unable to "ceph osd rm 8" because "unable to find OSDs ['8']" Anything else that I can try (other than a reboot) ? Many thanks Steven On Mon, 21 Jul 2025 at 17:25, Anthony D'Atri <aad(a)dreamsnake.net> wrote: > Look at /var/lib/ceph on ceph-host-7 for a leftover directory for osd.8 > > Also try > ceph osd crush remove osd.8 > ceph auth del osd.8 > ceph osd rm 8 > > > On Jul 21, 2025, at 5:12 PM, Steven Vacaroaia <stef97(a)gmail.com> wrote: > > > > Hi, > > > > I meant to replace a defective NVME used by osd.8 > > but I might have used the wrong approach as I ended up with a new OSD > while osd.8 show as "unknown" > > > > I tried "ceph orch osd rm " but it say "not found" > > > > I would appreciate some guidance > > > > I can remove the new one as the server is still "noout" and recreate > properly > > or > > remove osd.8 ( but how ?) > > > > Please let me know what would be the best action > > > > <image.png> > > Steven > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

squid 19.2.2 - cannot remove 'unknown" OSD
by Steven Vacaroaia 21 Jul '25

21 Jul '25

Hi, I meant to replace a defective NVME used by osd.8 but I might have used the wrong approach as I ended up with a new OSD while osd.8 show as "unknown" I tried "ceph orch osd rm " but it say "not found" I would appreciate some guidance I can remove the new one as the server is still "noout" and recreate properly or remove osd.8 ( but how ?) Please let me know what would be the best action [image: image.png] Steven

1 0

Cephfs client deadlock (OSD op state mismatch after stuck ops?)
by Hector Martin 21 Jul '25

21 Jul '25

Hi all, I'm running a small converged Ceph cluster (mons, OSDs, mds on the same 3 hosts, one of which is also a CephFS client). I noticed that sometimes the CephFS client seems to deadlock. There are other CephFS clients on the network, but they generally don't collide in file/dir accesses much with the local client. I found this old thread that describes a similar situation: https://www.mail-archive.com/search?l=ceph-users@lists.ceph.com&q=subject:%… The kernel version is 6.14.6 for the main hosts, 6.14.8 for a few other unrelated clients, and 6.3.2 for one older client. Ceph version is 19.2.2. Ceph status was clean and dmesg didn't show any warnings around the time of the hang, other than a few of these (which I always get and seem to be related to SELinux attributes, but appear to be harmless): [3447915.841214] ceph: [667e28f4-8964-454b-b777-03ce12b67742 6505232]: sync 00000000f6e49a9a 100033726fb.152 during filling trace [3447915.851171] SELinux: inode_doinit_use_xattr: getxattr returned 16 for dev=ceph ino=1099565573883 Restarting the mds (letting it failover to the standby-replay mds) did not help the situation, but after finding the above thread I noticed that all the requests were stuck on the same OSD: > root@flamingo:~ -# cat /sys/kernel/debug/ceph/*/osdc > REQUESTS 17 homeless 0 > 46841260 osd17 14.2ab058d9 14.19s0 [17,18,15,6,7,4,3,0,1]/17 [17,18,15,6,7,4,3,0,1]/17 e25593 10002e55314.0000003c 0x400014 1 read > 46841268 osd17 14.571aa108 14.8s0 [17,16,15,0,1,2,4,5,7]/17 [17,16,15,0,1,2,4,5,7]/17 e25593 100030edef7.00000039 0x400014 1 read > 46841311 osd17 14.7b4e2d92 14.12s0 [17,16,15,6,7,4,2,1,3]/17 [17,16,15,6,7,4,2,1,3]/17 e25593 1000318f566.00000179 0x400014 1 read > 46841323 osd17 14.deec222d 14.2ds0 [17,16,18,4,7,5,2,1,0]/17 [17,16,18,4,7,5,2,1,0]/17 e25593 100030edf00.00000028 0x400014 1 read > 46841334 osd17 14.eb940708 14.8s0 [17,16,15,0,1,2,4,5,7]/17 [17,16,15,0,1,2,4,5,7]/17 e25593 10003042460.000001c8 0x400014 1 read > 46841336 osd17 14.93065908 14.8s0 [17,16,15,0,1,2,4,5,7]/17 [17,16,15,0,1,2,4,5,7]/17 e25593 1000318f566.00000181 0x400014 1 read > 46841350 osd17 14.3e9eeaa5 14.25s0 [17,18,15,2,0,1,7,6,4]/17 [17,18,15,2,0,1,7,6,4]/17 e25593 10003498f94.00000000 0x400024 1 write > 46841359 osd17 14.5120499 14.19s0 [17,18,15,6,7,4,3,0,1]/17 [17,18,15,6,7,4,3,0,1]/17 e25593 10002e53590.00000000 0x400014 1 read > 46841367 osd17 14.25e874a5 14.25s0 [17,18,15,2,0,1,7,6,4]/17 [17,18,15,2,0,1,7,6,4]/17 e25593 1000318f566.00000196 0x400014 1 read > 46841376 osd17 14.cd26e465 14.25s0 [17,18,15,2,0,1,7,6,4]/17 [17,18,15,2,0,1,7,6,4]/17 e25593 1000318f566.00000199 0x400014 1 read > 46841405 osd17 14.855295f5 14.35s0 [17,18,15,3,0,2,4,7,5]/17 [17,18,15,3,0,2,4,7,5]/17 e25593 1000318f566.000001a7 0x400014 1 read > 46841421 osd17 14.57d5ef2c 14.2cs0 [17,18,16,1,3,2,4,6,5]/17 [17,18,16,1,3,2,4,6,5]/17 e25593 10002fe07b1.00000008 0x400014 1 read > 46841424 osd17 14.ea554059 14.19s0 [17,18,15,6,7,4,3,0,1]/17 [17,18,15,6,7,4,3,0,1]/17 e25593 1000318f566.000001ac 0x400014 1 read > 46841429 osd17 14.fa91b525 14.25s0 [17,18,15,2,0,1,7,6,4]/17 [17,18,15,2,0,1,7,6,4]/17 e25593 100030916b5.0000004f 0x400014 1 read > 46841447 osd17 14.88f24788 14.8s0 [17,16,15,0,1,2,4,5,7]/17 [17,16,15,0,1,2,4,5,7]/17 e25593 100032d95c7.00000011 0x400014 1 read > 46841468 osd17 12.159b3dd2 12.12 [17,5,2]/17 [17,5,2]/17 e25593 10003307266.00000013 0x400014 1 read > 46841496 osd17 12.a08092d2 12.12 [17,5,2]/17 [17,5,2]/17 e25593 10003493c3f.00000050 0x400014 1 read > LINGER REQUESTS > BACKOFFS Restarting that OSD fixed it. Looking at the OSD log at the exact time of the hang (determined by other logs), I see this: > 2025-07-18T06:39:06.992+0900 fffeed21bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _remove, latency = 5.162732124s, lat = 5s cid =12.3_head oid =#12:c0ccf68d:::1000336b41c.0000009d:13d# > 2025-07-18T06:39:06.992+0900 fffeed21bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency slow operation observed for submit_transact, latency = 5.167396545s > 2025-07-18T06:39:06.993+0900 fffef837bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.168415070s, txc = 0xaaabe3f57000 > 2025-07-18T06:39:14.834+0900 fffeeca0bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _remove, latency = 5.340788364s, lat = 5s cid =12.14_head oid =#12:28e0c837:::10003307273.00000024:13d# > 2025-07-18T06:39:14.835+0900 fffeeca0bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency slow operation observed for submit_transact, latency = 5.342024803s > 2025-07-18T06:39:14.843+0900 fffef837bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.349779129s, txc = 0xaaac19c75b00 > 2025-07-18T06:40:37.552+0900 fffeeb9ebf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _remove, latency = 5.124079704s, lat = 5s cid =12.c_head oid =#12:340862a9:::100033073f1.0000002c:13d# > 2025-07-18T06:40:37.560+0900 fffeeb9ebf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency slow operation observed for submit_transact, latency = 5.132128239s > 2025-07-18T06:40:37.568+0900 fffef837bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.140548706s, txc = 0xaaabf25b8000 > 2025-07-18T06:40:59.228+0900 fffeec1fbf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _remove, latency = 5.677710056s, lat = 5s cid =12.2_head oid =#12:40b82ac1:::1000348d274.00000008:13d# > 2025-07-18T06:40:59.229+0900 fffeec1fbf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency slow operation observed for submit_transact, latency = 5.679266453s > 2025-07-18T06:40:59.232+0900 fffef837bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.682381630s, txc = 0xaaac34479500 > 2025-07-18T06:41:01.799+0900 fffeeb9ebf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _remove, latency = 5.117489815s, lat = 5s cid =12.c_head oid =#12:340e7251:::10003307273.00000027:13d# > 2025-07-18T06:41:01.799+0900 fffeeb9ebf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency slow operation observed for submit_transact, latency = 5.118617058s > 2025-07-18T06:41:01.804+0900 fffef837bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.123104095s, txc = 0xaaabbf99de00 > 2025-07-18T06:42:28.863+0900 fffeec1fbf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _remove, latency = 5.191506386s, lat = 5s cid =12.3_head oid =#12:c2349a5a:::1000336b418.000000f0:13d# > 2025-07-18T06:42:28.863+0900 fffeec1fbf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency slow operation observed for submit_transact, latency = 5.192711353s > 2025-07-18T06:42:28.864+0900 fffef837bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.193384171s, txc = 0xaaabd1e51e00 > 2025-07-18T06:42:34.152+0900 fffeeca0bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _remove, latency = 5.287574291s, lat = 5s cid =12.3_head oid =#12:c2bf668a:::10003307273.0000007b:13d# > 2025-07-18T06:42:34.154+0900 fffeeca0bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency slow operation observed for submit_transact, latency = 5.289901257s > 2025-07-18T06:42:34.156+0900 fffef837bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.291684628s, txc = 0xaaabf9f90c00 > 2025-07-18T06:42:40.801+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:42:40.801+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) [ repeats ... ] > 2025-07-18T06:44:15.954+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:44:15.954+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) > 2025-07-18T06:44:27.591+0900 fffeeca0bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _remove, latency = 5.561402321s, lat = 5s cid =12.0_head oid =#12:064b3406:::1000348bd03.0000004a:13d# > 2025-07-18T06:44:27.594+0900 fffeeca0bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency slow operation observed for submit_transact, latency = 5.565083027s > 2025-07-18T06:44:27.595+0900 fffef837bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.565894604s, txc = 0xaaabe0e98c00 > 2025-07-18T06:44:29.442+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:44:29.442+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) [ repeats ... ] > 2025-07-18T06:44:32.362+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:44:32.362+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 2 slow requests (by type [ 'queued for pg' : 1 'started' : 1 ] most affected pool [ 'cephfs2_data_hr3' : 1 ]) > 2025-07-18T06:44:33.314+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:44:33.314+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 2 slow requests (by type [ 'queued for pg' : 1 'started' : 1 ] most affected pool [ 'cephfs2_data_hr3' : 1 ]) > 2025-07-18T06:44:34.328+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:44:34.328+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 2 slow requests (by type [ 'queued for pg' : 1 'started' : 1 ] most affected pool [ 'cephfs2_data_hr3' : 1 ]) > 2025-07-18T06:44:38.361+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.6505232.0:46841376 14.25s0 14.cd26e465 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:44:38.361+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 2 slow requests (by type [ 'queued for pg' : 2 ] most affected pool [ 'cephfs2_data_hec5.4' : 2 ]) > 2025-07-18T06:44:39.369+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.6505232.0:46841376 14.25s0 14.cd26e465 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:44:39.369+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 2 slow requests (by type [ 'queued for pg' : 2 ] most affected pool [ 'cephfs2_data_hec5.4' : 2 ]) > 2025-07-18T06:44:40.356+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.6505232.0:46841260 14.19s0 14.2ab058d9 (undecoded) ondisk+read+known_if_redirected e25583) > 2025-07-18T06:44:40.356+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 5 slow requests (by type [ 'queued for pg' : 5 ] most affected pool [ 'cephfs2_data_hec5.4' : 5 ]) > 2025-07-18T06:44:41.324+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:44:41.324+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) [ repeats ... ] > 2025-07-18T06:45:14.197+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:45:14.197+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) > 2025-07-18T06:45:15.825+0900 fffeec1fbf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _remove, latency = 5.245476246s, lat = 5s cid =12.b_head oid =#12:d12c2386:::1000336b41a.00000095:13d# > 2025-07-18T06:45:15.826+0900 fffeec1fbf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency slow operation observed for submit_transact, latency = 5.247073650s > 2025-07-18T06:45:15.831+0900 fffef837bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.252201080s, txc = 0xaaabd1e50c00 > 2025-07-18T06:45:19.037+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:45:19.037+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) > 2025-07-18T06:45:20.056+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:45:20.056+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) > 2025-07-18T06:45:21.010+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:45:21.010+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) > 2025-07-18T06:45:21.963+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:45:21.963+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) > 2025-07-18T06:45:22.973+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.6505232.0:46841376 14.25s0 14.cd26e465 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:45:22.973+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 2 slow requests (by type [ 'queued for pg' : 2 ] most affected pool [ 'cephfs2_data_hec5.4' : 2 ]) > 2025-07-18T06:45:23.958+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.6505232.0:46841376 14.25s0 14.cd26e465 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:45:23.958+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 2 slow requests (by type [ 'queued for pg' : 2 ] most affected pool [ 'cephfs2_data_hec5.4' : 2 ]) > 2025-07-18T06:45:32.861+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) [ repeats ... ] > 2025-07-18T06:49:20.553+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) > 2025-07-18T06:49:21.547+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:49:21.547+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) > 2025-07-18T06:49:22.596+0900 ffff014cbf00 -1 osd.17 25584 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.6505232.0:46841429 14.25s0 14.fa91b525 (undecoded) ondisk+read+known_if_redirected e25584) > 2025-07-18T06:49:22.596+0900 ffff014cbf00 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs2_data_hec5.4' : 1 ]) > 2025-07-18T06:49:48.204+0900 fffeec1fbf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _remove, latency = 5.027801514s, lat = 5s cid =12.1a_head oid =#12:5b980d7b:::100033073d8.0000002a:13d# > 2025-07-18T06:49:48.205+0900 fffeec1fbf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency slow operation observed for submit_transact, latency = 5.031510353s > 2025-07-18T06:49:48.214+0900 fffef837bf00 0 bluestore(/var/lib/ceph/osd/ceph-17) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.040008068s, txc = 0xaaabe5c0cf00 Note that osd17 is on a different host (but some of the shards of the affected PGs are on the same host). After this 10-minute period, the logs are clean. So it kind of looks like one op (46841429) above was a slow op for about 10 minutes, but then it stopped being considered slow by the OSD, yet the kernel client never considered the op completed and it got stuck there? Other ops from that client are also mentioned, and this doesn't seem right to me. Op 46841429 is the first one to become "slow", but then other ops like 46841376 are reported as the "oldest" op, and then it goes back to 46841429. Unless this is about different phases for the same op, it doesn't make sense to me that a younger op could be reported as slow, then an older op, then back to a younger op. If the older op is older, it should always be reported as slow first, right? I checked all the other OSD logs and there's nothing interesting other than a couple other OSDs that also got slow ops at around the same time (but much shorter duration). Grafana stats show a snaptrim operation started at exactly that time and lasted 10 minutes. The CPU logs show that this did repro the CPU usage/page cache thrashing anomaly I previously mentioned on the list (high sys >> user CPU usage), but this time the thrashing only lasted 10 minutes, so I would say this is unlikely to be directly caused by that bug, though it could be that the high load triggering slow ops in general then tickled another bug causing the ops to get stuck. Happy to help debug this if it happens again if anyone has pointers (I think this might not be the first time, but it's the first time I found that thread and learned about this issue specifically, and the debug file to check the culprit OSD). - Hector

1 0

Test Cluster / Performance Degradation After Adding Private Network
by Anthony Fecarotta 19 Jul '25

19 Jul '25

Hello, I have a test cluster of some mini-PCs. This one, in particular, runs Proxmox, and has two Ceph RBDs (LXCs & VMs). The purpose of this test cluster was to test Docker Swarm. I wanted to get the feel for orchestration - our five-node production cluster is very simple and Kubernetes would be overkill. Each node boots off of NVMe, and each node has one OSD, PCIe Gen 4 M.2 NVMe. I understand this equipment is not optimal, but please keep in mind this is a test cluster. All things considered, it was running fine for two months, I even made some of our non-critical BETA programs available for internal use within our organization. Yesterday, I connected a 2.5GBe unmanaged switch, to the second 2.5GBe NIC of each node cluster, creating a Private/Cluster Network for Ceph. Since then, each node, VM, LXC, etc. are moving at a glacial pace. Just to give an example, a sudo apt update or just logging in via SSH can take sixty seconds. [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = 172.16.1.0/24 fsid = 3c395d5c-7d46-4dc7-ad4b-8a6761f167b0 mon_allow_pool_delete = true mon_host = 192.168.128.156 192.168.128.150 192.168.128.158 ms_bind_ipv4 = true ms_bind_ipv6 = false osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = 192.168.128.156/24 [client] keyring = /etc/pve/priv/$cluster.$name.keyring [client.crash] keyring = /etc/pve/ceph/$cluster.$name.keyring [mds] keyring = /var/lib/ceph/mds/ceph-$id/keyring [mon.asusNuc1] public_addr = 192.168.128.150 [mon.chyna2gb] public_addr = 192.168.128.156 [mon.chyna4tb] public_addr = 192.168.128.158 # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54 # devices device 0 osd.0 class nvme device 1 osd.1 class nvme device 2 osd.2 class nvme device 3 osd.3 class nvme # types type 0 osd type 1 host type 2 chassis type 3 rack type 4 row type 5 pdu type 6 pod type 7 room type 8 datacenter type 9 zone type 10 region type 11 root # buckets host chyna2gb { id -3 # do not change unnecessarily id -4 class nvme # do not change unnecessarily # weight 1.86299 alg straw2 hash 0 # rjenkins1 item osd.0 weight 1.86299 } host chyna4tb { id -5 # do not change unnecessarily id -6 class nvme # do not change unnecessarily # weight 3.63869 alg straw2 hash 0 # rjenkins1 item osd.1 weight 3.63869 } host nuc { id -7 # do not change unnecessarily id -8 class nvme # do not change unnecessarily # weight 0.90970 alg straw2 hash 0 # rjenkins1 item osd.2 weight 0.90970 } host asusNuc1 { id -9 # do not change unnecessarily id -10 class nvme # do not change unnecessarily # weight 3.63869 alg straw2 hash 0 # rjenkins1 item osd.3 weight 3.63869 } root default { id -1 # do not change unnecessarily id -2 class nvme # do not change unnecessarily # weight 10.05006 alg straw2 hash 0 # rjenkins1 item chyna2gb weight 1.86299 item chyna4tb weight 3.63869 item nuc weight 0.90970 item asusNuc1 weight 3.63869 } # rules rule replicated_rule { id 0 type replicated step take default step chooseleaf firstn 0 type host step emit } # end crush map root@asusNuc1:~# ceph osd perf osd commit_latency(ms) apply_latency(ms) 0 6 6 3 15 15 2 23 23 1 4 4 To be clear, the "dumb switch" is isolated and not connected to the rest of the network. Regards, [image] Anthony Fecarotta Founder & President [image] anthony(a)linehaul.ai <mailto:anthony@linehaul.ai> [image] 224-339-1182 [image] (855) 625-0300 [image] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image] www.linehaul.ai <http://www.linehaul.ai/> [image] <http://www.linehaul.ai/> [image] <https://www.linkedin.com/in/anthony-fec/>

2 1

Reef: cephadm tries to apply specs frequently
by Eugen Block 18 Jul '25

18 Jul '25

Good morning, I noticed something strange on a 18.2.7 cluster, running on Ubuntu 22.04, deployed by cephadm. There are 10 hosts in total, 5 of them are all-flash and those aren't affected. The other 5 hosts are hdd-only, and only 4 of those are affected: The /var/log/ceph/{FSID}/ceph-volume.log is flooded with attempts to apply the osd spec: [2025-07-17 05:40:01,994][ceph_volume.main][INFO ] Running command: ceph-volume lvm batch --no-auto /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw --yes --no-systemd [2025-07-17 05:42:00,216][ceph_volume.main][INFO ] Running command: ceph-volume lvm batch --no-auto /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw --yes --no-systemd [2025-07-17 05:43:50,521][ceph_volume.main][INFO ] Running command: ceph-volume lvm batch --no-auto /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw --yes --no-systemd So that log file alone grows to more than 1 GB per day, as a consequence other logs like syslog grow as well. But for some reason, storage08 is skipped, the mgr reports: [cephadm DEBUG root] skipping apply of storage08 on DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd This is the current hdd-only spec: # ceph orch ls osd --export service_type: osd service_id: hdd-only service_name: osd.hdd-only placement: hosts: - storage06 - storage07 - storage08 - storage09 - storage10 spec: data_devices: rotational: 1 size: '1T:' filter_logic: AND objectstore: bluestore I verified that all disks are deployed as OSDs, so there's no orphan lying around or anything. I failed the mgr (of course), rebooted one host because storage08 was recently rebooted as well. I don't know for how long this has been going on, unfortunately. So I started to look at the code [0], [1], it seems like the mgr cache is not properly updated: if not self.mgr.cache.osdspec_needs_apply(host, drive_group): self.mgr.log.debug("skipping apply of %s on %s (no change)" % ( host, drive_group)) So I looked at all these values: def osdspec_needs_apply(self, host: str, spec: ServiceSpec) -> bool: if ( host not in self.devices or host not in self.last_device_change or host not in self.last_device_update or host not in self.osdspec_last_applied or spec.service_name() not in self.osdspec_last_applied[host] ): but all keys are populated with similar values as on storage08, for example: root@storage01:~# ceph config-key get mgr/cephadm/host.storage10 | jq -r '.last_device_change,.last_device_update,.osdspec_last_applied' 2025-02-12T16:27:21.979015Z 2025-07-17T05:35:18.852618Z { "osd.hdd-only": "2025-07-17T06:03:38.860971Z" } root@storage01:~# ceph config-key get mgr/cephadm/host.storage08 | jq -r '.last_device_change,.last_device_update,.osdspec_last_applied' 2025-03-11T08:23:02.851969Z 2025-07-17T05:43:47.521682Z { "osd.hdd-only": "2025-03-11T08:23:21.494004Z" } Can anyone make sense of it? I'd appreciate any pointers! Thanks! Eugen [0] https://github.com/ceph/ceph/blob/v18.2.7/src/pybind/mgr/cephadm/services/o… [1] https://github.com/ceph/ceph/blob/v18.2.7/src/pybind/mgr/cephadm/inventory.…

1 1

Hardware recommendation
by GLE, Vivien 18 Jul '25

18 Jul '25

Hi, My mission is to make a ceph cluster production ready. We already have a mock-up of 6 nodes deployed via cephadm to test ceph's features We want to install a ceph cluster to get off our dependencies to SAN and to store around 400 TB of data. We'll use mostly block storage for VMs/k8 and a bit of CephFS and Object storage Are those specs good for our purpose ? - 3 MGR/MON/RGW 128 GB RAM 32 cores (is it recommended to put RGW daemon in the same node as MGR/MON ?) - 2 MDS 16 GB RAM 8 cores - * OSD 8 GB RAM 8 cores (with high IOPS per core) We are pretty new to ceph so any advice would be appreciate ! Thanks ! Vivien

2 2

Re: squid 19.2.2 - discrepancies between GUI and CLI
by Steven Vacaroaia 17 Jul '25

17 Jul '25

Thanks for the suggestion Unfortunately "ceph mgr fail" did not solve the issue Is there a better way to "fail" ? Steven On Thu, 17 Jul 2025 at 14:04, Anthony D'Atri <aad(a)dreamsnake.net> wrote: > Try failing the mgr > > > On Jul 17, 2025, at 1:48 PM, Steven Vacaroaia <stef97(a)gmail.com> wrote: > > > > Hi, > > > > The GUI is reporting PGs still being backfilled > > but CLI is not > > > > Who is right ;-) and how can I get them to "agree" ? > > > > <image.png> > > > > <image.png> > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

squid 19.2.2 - discrepancies between GUI and CLI
by Steven Vacaroaia 17 Jul '25

17 Jul '25

Hi, The GUI is reporting PGs still being backfilled but CLI is not Who is right ;-) and how can I get them to "agree" ? [image: image.png] [image: image.png]

1 0

[radosgw-reef] How to move customer data to another ec-pool
by Boris 16 Jul '25

16 Jul '25

Hi, we have an object storage cluster with a lot of backup data. The data is currently in an extra EC pool (jerasure, k=5,m=3), with a custom s3 storage_class called `backup-center-ec`. Those are dedicated buckets for backup jobs and make around half the data of a 2PB cluster. We want to create a new EC pool with the ISA plugin and maybe even adjust the k/m values. We hope to get a little bit more performance out of it. Because we have this EC pool only in one zonegroup, the backup script checks if the `backup-center-ec` class is available (or we even hardcoded it), so we came up with the following idea: 1. create a new pool with correct EC profile 2. rename `backup-center-ec` to `backup-center-ec-legacy-01` 3. recreate the `backup-center-ec` storage class with the new pool This should make it possible that new backups will go into the new pool, and the old backups will slowly rotate out of the old pool. IIRC reading an object does not requrie to know in which storage_class the object is stored. If this would work, I am already very happy. But maybe the migration could be spead up or forceful move data to the new pool? I thought I can maybe set the explicit placement for a bucket to the new pool and do a radosgw-admin bucket rewrite. Or are there other ways? Would that work? Does someone have experience with it? Cheers Boris

1 0

squid 19.1.3 QE validation status
by Yuri Weinstein 16 Jul '25

16 Jul '25

Details of this release are summarized here: https://tracker.ceph.com/issues/71912#note-1 Release Notes - TBD LRC upgrade - TBD Seeking approvals/reviews for: rados - Radek, Laura rgw- Adam Emerson fs - Venky orch - Adam King rbd, krbd - Ilya quincy-x, reef-x - Laura, Neha (can we make it less noisy?) crimson-rados - N/A ceph-volume - Guillaume Pls let me know if any tests were missed from this list.

8 18

ceph fs authorize rely on an old setup
by Patrick Begou 15 Jul '25

15 Jul '25

Hi all, I'm deploying Squid. I configure a first cephfs setup using k=6 and m=8 but this was a bad idea for my 5 nodes cluster. So I try to roll back and build a k=5 and m=3. But it remains some occurrences of my old setup when allowing cephfs clients and I do not understand how to remove them. My previous status was: # ceph osd pool ls .mgr ec-62-pool cephfs.data-ec62-vol.meta cephfs.data-ec62-vol.data to remove it I ran: # ceph fs fail data-ec62-vol # ceph fs rm data-ec62-vol --yes-i-really-mean-it # ceph config set mon mon_allow_pool_delete true # ceph osd pool delete ec-62-pool ec-62-pool --yes-i-really-really-mean-it # ceph osd pool delete cephfs.data-ec62-vol.data cephfs.data-ec62-vol.data --yes-i-really-really-mean-it # ceph osd pool delete cephfs.data-ec62-vol.meta cephfs.data-ec62-vol.meta --yes-i-really-really-mean-it # ceph config set mon mon_allow_pool_delete false # ceph osd pool ls .mgr I then remove my erasure code profile: # ceph osd erasure-code-profile ls default ec-62-profile-isa # ceph osd erasure-code-profile rm ec-62-profile-isa # ceph osd erasure-code-profile ls default And the corresponding crush rule: # ceph osd crush rule ls replicated_rule ec-62-pool # ceph osd crush rule rm ec-62-pool Then I create my new config: # ceph osd erasure-code-profile set ec-53-profile-isa k=5 m=3 crush-failure-domain=host plugin=isa # ceph osd erasure-code-profile ls default ec-53-profile-isa # ceph osd pool create ec-53-pool erasure ec-53-profile-isa pool 'ec-53-pool' created # ceph osd pool set ec-53-pool allow_ec_overwrites true set pool 5 allow_ec_overwrites to true # ceph fs volume create data # ceph fs add_data_pool data ec-53-pool Last I create a folder in this fs and set it to the erasure coding pool: # ceph-fuse /mnt # cd /mnt # mkdir data # setfattr -n ceph.dir.layout.pool -v ec-53-pool /mnt/data But when I want to create the new Key for the clients with: # ceph fs authorize data client.whitaker / r root_squash /data rw This still rely on the old data-ec62-vol setup: [client.whitaker] key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXX== caps mds = "allow r fsname=data-ec62-vol, allow rw fsname=data-ec62-vol path=data, allow r fsname=data root_squash, allow rw fsname=data path=data" caps mon = "allow r fsname=data-ec62-vol, allow r fsname=data" caps osd = "allow rw tag cephfs data=data-ec62-vol, allow rw tag cephfs data=data" And "ceph auth ls" show several remaining occurrences for the old data-ec62-vol setup: mds.data-ec62-vol.whitaker02-ceph.jbqyuh key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX== caps: [mds] allow caps: [mon] profile mds caps: [osd] allow rw tag cephfs *=* mds.data-ec62-vol.whitaker03-ceph.qtaqtn key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX== caps: [mds] allow caps: [mon] profile mds caps: [osd] allow rw tag cephfs *=* client.whitaker key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX== caps: [mds] allow r fsname=data-ec62-vol, allow rw fsname=data-ec62-vol path=data, allow r fsname=data root_squash, allow rw fsname=data path=data caps: [mon] allow r fsname=data-ec62-vol, allow r fsname=data caps: [osd] allow rw tag cephfs data=data-ec62-vol, allow rw tag cephfs data=data How can I build a correct setup for these keys ? Thank Patrick

1 1

Possible bug: Web UI for Ceph
by Ron Gage 13 Jul '25

13 Jul '25

Greetings from Detroit Michigan! Is there a preferred way to report bugs in the WebUI? The Bug: pre-conditions: a single CephFS and MDS created using an EC pool for data and a repl pool for meta. Create a subVolume from the FS. Create an NFS server attached to the created subvol. Triggering the bug: Edit the NFS entry you just created and note that the subvolume is no longer set to the subvolume as created. Further, attempting to set the subvolume back to the original setting is not possible as it is not available in the dropdown. Expected behavior: the Subvolume field should be populated with the current setting of the NFS export. Ron Gage

2 2

Compression confusion
by Ryan Sleeth 11 Jul '25

11 Jul '25

Hello, what are the necessary steps to enable compression on an EC CephFS pool? I have tried the following commands, but copying test data shows little to no savings compared to 1.28x on lz4 ZFS. Is min/max block size required as well? sudo ceph osd pool set cephfs_ec_data compression_mode force sudo ceph osd pool set cephfs_ec_data compression_algorithm lz4 sudo ceph osd pool set cephfs_ec_data compression_required_ratio 0.975 sudo ceph config set osd bluestore_compression_mode force sudo ceph config set osd bluestore_compression_algorithm lz4 sudo ceph config set osd bluestore_compression_required_ratio 0.975 -- Ryan Sleeth

1 0

squid 19.2.2 - troubleshooting pgs in active+remapped+backfill - no pictures
by Steven Vacaroaia 11 Jul '25

11 Jul '25

Hi, I sent another version of this message with pictures that awaits moderation since it is so big - apologies for that In the meantime I got approval to share the output of some of the command - see attached I have a 19.2.2 cluster deployed with cephadm 7 nodes and 2 networks ( cluster (2 x 100GB) and public (2 x 25Gb)) It provides an S3 bucket as a target for my backup The underlying pool (default.rgw.buckets.data) is using an EC 4+2 profile with a storage class for spinning disks All spinning disks are keeping WAl/DB on NVME The amount of data grew pretty fast and , since I started the pool with pg_autoscale_mode = warn,I have decided to increase the number of PG manually ( from 128 to 256) As expected, the backfilling started ...and it never ended ...even now after more than 1 week I still have about 29 pgs backfilling and 13 backfilling_wait What worries me is that the number of backfilling PGs varies very little over time e.g 28 and 12 ALTHOUGH there is constant "recovery" traffic between 250 and 350MiB There is no OSD or capacity issue ( if I enable pg_autoscale_mode the cluster health is OK ) The "recovery" seems to be doing something ( but number of objects remain the same ) Since the recovery should run over the cluster network and the amount of data in the pool is not huge, I am not sure why it takes so many days - it seems stuck actually The only strange thing I noticed is a discrepancy between the number of PG and PGP that the pool currently has ...and what autoscale-status says Any help / suggestions would be very appreciated What I have tried so for : increase recovery speed ( by changing mclock profile to "high_recovery_ops" and overriding various parameters) (recovery_max_active, recovery_max_active_hdd ... etc) redeploying some of the OSDs that were "UP_PRIMARY but part of the backfill_wait PGs query the PGs and look for a "stuck reason" stop scrub and deep-scrub repair the PGs (some) change the pg_autoscale_mode to true check the balancer status Many thanks Steven

3 4

[Urgent suggestion needed] New Prod Cluster Hardware recommendation
by Pripriya 11 Jul '25

11 Jul '25

Hello folks, I am new to this thread would like to get some suggestions to build new external ceph cluster which will backend for proxmox VM's I am planning to start with 5 Nodes(3 Mon & 2 OSD) and I am expecting to start with ~60+ TB usable space. *(*3- Mon will have MGR, OSD &MDS service 2 OSD dedicated nodes) estimated Storage Specs Calculator: RAM: 8GB/OSD Daemon, 16GB OS, 4GB for Mon & MGR, 16GB for MDS cpu: 2 core/osd, 2 core for os, 2 core per services *Dell R7625 5 Node to start with * - RAM: 128G (Plan to increase later as needed) - CPU: 2x AMD EPYC 9224 2.50GHz, 24C/48T, 64M Cache (200W) DDR5-4800 - Chassis Configuration 24x2.5 NVME - 2x1.92TB Data Center NVMe Read Intensive AG Drive U2 Gen4 with carrier ( OS Disk, I need extra space) - 5x 7.68TB Data Center NVMe Read Intensive AG Drive U2 Gen4 with Carrier 24Gbps 512e 2.5in Hot-Plug 1DWPD , AG Drive - 2x Nvidia ConnectX-6 Lx Dual Port 10/25GbE SFP28, No Crypto, PCIe Low Profile - 1G for IPMI Please help me finalize these specs. Thanks

6 11

Experience with high mds_max_caps_per_client
by Kasper Rasmussen 11 Jul '25

11 Jul '25

I have a CephFS where workloads use many small files. I see cache pressure / MDS_CLIENT_RECALL warnings ones in awhile (due to clients exceeding mds_max_caps_per_client) and it seems if they linger to long, it ends up with more warnings e.g. MDS_SLOW_REQUESTS, and some directories locks up. Anyway - Currently I have the mds_max_caps_per_client set to 2M which by looking into the output of sudo ceph tell mds.<name> counter dump .. .. "counters": { "cap_hits": 8122912454, "cap_miss": 497593, "avg_read_latency": 0.000000028, "avg_write_latency": 0.000000000, "avg_metadata_latency": 0.000000000, "dentry_lease_hits": 5630994071, "dentry_lease_miss": 174816044, "opened_files": 65, "opened_inodes": 2106823, "pinned_icaps": 2106823, "total_inodes": 2106823, "total_read_ops": 309938, "total_read_size": 191662499168, "total_write_ops": 371242, "total_write_size": 414398493835 .. .. It Is not enough. However there is not a lot of open files Checking the "ceph_mds_client_metrics_<fs_name>_pinned_icaps" gauge in prometheus tells the same story.. The client is constantly hidden the max caps roof over days (clients have long running jobs) Can anyone share experience in regards to changing mds_max_caps_per_client to accomodate for such workloads. When changing this, should other config variables be taken into account like - mds_cache_memory_limit - currently: 36GB mds_cache_trim_decay_rate - currently: 0.9 mds_cache_trim_threshold - currently: 288358 mds_recall_max_caps - currently: 33000 mds_recall_max_decay_rate - currently: 1.35 Or should the be tuned on a observe-and-change-as-needed basis. Thanks in advance.

2 1

Recovery Not happening..
by Devender Singh 11 Jul '25

11 Jul '25

Hello all Need some help why nothing is happening… version is 19.2.2. Tried repairing the pgs, repair and force-backfill and force-recovery but nothing initiating backfills. # ceph -v ceph version 19.2.2 # ceph -s cluster: id: 79d4fade-ffa9-11eb-a57d-f398f000cfc1 health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore nodeep-scrub flag(s) set Degraded data redundancy: 6114281/137592495 objects degraded (4.444%), 124 pgs degraded, 124 pgs undersized 270 pgs not deep-scrubbed in time 1 subtrees have overcommitted pool target_size_bytes services: mon: 3 daemons, quorum ld-converged03n,ld-converged01n,ld-converged02n (age 5d) mgr: ld-converged01n.gnxoye(active, since 5d), standbys: ld-converged02n.odvopy mds: 4/4 daemons up, 2 standby, 2 hot standby osd: 24 osds: 24 up (since 29h), 24 in (since 29h); 292 remapped pgs flags nodeep-scrub rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 4/4 healthy pools: 24 pools, 1345 pgs objects: 32.22M objects, 98 TiB usage: 196 TiB used, 83 TiB / 279 TiB avail pgs: 6114281/137592495 objects degraded (4.444%) 14349193/137592495 objects misplaced (10.429%) 864 active+clean 237 active+clean+remapped 124 active+undersized+degraded 55 active+clean+scrubbing 45 active+clean+remapped+scrubbing 10 active+clean+remapped+scrubbing+deep 10 active+clean+scrubbing+deep io: client: 50 KiB/s rd, 1.4 MiB/s wr, 8 op/s rd, 205 op/s wr # ceph health detail HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore; nodeep-scrub flag(s) set; Degraded data redundancy: 6114283/137592537 objects degraded (4.444%), 124 pgs degraded, 124 pgs undersized; 270 pgs not deep-scrubbed in time; 1 subtrees have overcommitted pool target_size_bytes [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in BlueStore osd.21 observed slow operation indications in BlueStore [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set [WRN] PG_DEGRADED: Degraded data redundancy: 6114283/137592537 objects degraded (4.444%), 124 pgs degraded, 124 pgs undersized pg 26.3 is stuck undersized for 94m, current state active+undersized+degraded, last acting [31,14,NONE,0,8] pg 26.4 is stuck undersized for 94m, current state active+undersized+degraded, last acting [2,8,NONE,24,35] pg 26.7 is stuck undersized for 94m, current state active+undersized+degraded, last acting [2,15,31,7,NONE] pg 26.d is stuck undersized for 94m, current state active+undersized+degraded, last acting [21,12,30,NONE,9] pg 26.11 is stuck undersized for 94m, current state active+undersized+degraded, last acting [13,8,30,NONE,3] pg 26.3c is stuck undersized for 94m, current state active+undersized+degraded, last acting [0,7,35,5,NONE] pg 26.3d is stuck undersized for 94m, current state active+undersized+degraded, last acting [9,NONE,14,6,31] pg 27.0 is stuck undersized for 94m, current state active+undersized+degraded, last acting [2,31,7,15,NONE] pg 27.2 is stuck undersized for 94m, current state active+undersized+degraded, last acting [0,35,11,NONE,31] pg 27.3 is stuck undersized for 94m, current state active+undersized+degraded, last acting [35,31,4,1,NONE] pg 27.b is active+undersized+degraded, acting [2,8,NONE,5,12] pg 27.c is stuck undersized for 94m, current state active+undersized+degraded, last acting [2,8,12,31,NONE] pg 27.10 is stuck undersized for 94m, current state active+undersized+degraded, last acting [23,8,14,NONE,0] pg 27.14 is stuck undersized for 94m, current state active+undersized+degraded, last acting [30,16,9,NONE,6] pg 27.17 is stuck undersized for 94m, current state active+undersized+degraded, last acting [23,4,14,NONE,1] pg 27.1b is stuck undersized for 94m, current state active+undersized+degraded, last acting [12,10,NONE,2,30] pg 27.26 is stuck undersized for 94m, current state active+undersized+degraded, last acting [0,35,10,24,NONE] pg 27.3c is stuck undersized for 94m, current state active+undersized+degraded, last acting [32,9,2,13,NONE] pg 33.1a is stuck undersized for 94m, current state active+undersized+degraded, last acting [0,NONE,31,35,4] pg 48.22 is stuck undersized for 94m, current state active+undersized+degraded, last acting [30,0,7,NONE,12] pg 48.24 is stuck undersized for 94m, current state active+undersized+degraded, last acting [5,1,NONE,13,9] pg 48.26 is stuck undersized for 94m, current state active+undersized+degraded, last acting [9,3,35,NONE,30] pg 48.28 is stuck undersized for 94m, current state active+undersized+degraded, last acting [9,NONE,3,14,5] pg 48.2b is stuck undersized for 94m, current state active+undersized+degraded, last acting [15,4,5,NONE,0] pg 48.2e is stuck undersized for 94m, current state active+undersized+degraded, last acting [15,7,31,3,NONE] pg 48.30 is stuck undersized for 94m, current state active+undersized+degraded, last acting [15,31,21,NONE,9] pg 48.31 is stuck undersized for 9M, current state active+undersized+degraded, last acting [4,1,13,30,NONE] pg 48.32 is stuck undersized for 94m, current state active+undersized+degraded, last acting [4,0,5,13,NONE] pg 48.41 is stuck undersized for 94m, current state active+undersized+degraded, last acting [8,32,NONE,0,16] pg 48.43 is stuck undersized for 94m, current state active+undersized+degraded, last acting [30,0,12,7,NONE] pg 48.4b is stuck undersized for 94m, current state active+undersized+degraded, last acting [13,6,NONE,10,24] pg 48.4f is stuck undersized for 94m, current state active+undersized+degraded, last acting [30,15,2,7,NONE] pg 48.50 is stuck undersized for 94m, current state active+undersized+degraded, last acting [2,9,NONE,13,24] pg 48.53 is stuck undersized for 94m, current state active+undersized+degraded, last acting [4,30,14,3,NONE] pg 48.54 is stuck undersized for 94m, current state active+undersized+degraded, last acting [9,6,14,NONE,30] pg 48.56 is stuck undersized for 94m, current state active+undersized+degraded, last acting [35,9,5,1,NONE] pg 48.59 is stuck undersized for 94m, current state active+undersized+degraded, last acting [30,14,7,NONE,3] pg 48.5d is stuck undersized for 94m, current state active+undersized+degraded, last acting [10,35,NONE,3,5] pg 48.5f is stuck undersized for 94m, current state active+undersized+degraded, last acting [5,13,8,0,NONE] pg 48.62 is stuck undersized for 94m, current state active+undersized+degraded, last acting [4,2,32,35,NONE] pg 48.70 is stuck undersized for 94m, current state active+undersized+degraded, last acting [3,10,5,15,NONE] pg 48.72 is stuck undersized for 94m, current state active+undersized+degraded, last acting [1,31,11,15,NONE] pg 48.75 is stuck undersized for 94m, current state active+undersized+degraded, last acting [3,24,NONE,9,35] pg 48.7d is stuck undersized for 94m, current state active+undersized+degraded, last acting [5,10,14,1,NONE] pg 48.7e is stuck undersized for 94m, current state active+undersized+degraded, last acting [32,NONE,15,9,0] pg 48.82 is stuck undersized for 94m, current state active+undersized+degraded, last acting [24,13,3,9,NONE] pg 48.86 is stuck undersized for 94m, current state active+undersized+degraded, last acting [23,9,12,2,NONE] pg 48.8e is stuck undersized for 94m, current state active+undersized+degraded, last acting [2,7,12,NONE,32] pg 48.90 is stuck undersized for 94m, current state active+undersized+degraded, last acting [11,2,NONE,12,30] pg 48.97 is stuck undersized for 94m, current state active+undersized+degraded, last acting [0,8,16,5,NONE] pg 48.98 is stuck undersized for 94m, current state active+undersized+degraded, last acting [23,9,12,2,NONE] [WRN] PG_NOT_DEEP_SCRUBBED: 270 pgs not deep-scrubbed in time pg 48.9b not deep-scrubbed since 2025-06-24T22:11:31.466631+0000 pg 48.97 not deep-scrubbed since 2025-06-28T14:03:27.678577+0000 pg 59.9d not deep-scrubbed since 2025-06-23T02:38:49.854500+0000 pg 48.90 not deep-scrubbed since 2025-06-27T07:20:46.510366+0000 pg 48.8e not deep-scrubbed since 2025-06-20T03:44:19.531953+0000 pg 48.8c not deep-scrubbed since 2025-06-27T04:38:11.342009+0000 pg 48.86 not deep-scrubbed since 2025-06-24T02:01:54.551428+0000 pg 59.8e not deep-scrubbed since 2025-06-22T17:48:29.668256+0000 pg 48.82 not deep-scrubbed since 2025-06-28T06:27:01.586778+0000 pg 59.76 not deep-scrubbed since 2025-06-26T04:36:17.368921+0000 pg 48.7d not deep-scrubbed since 2025-06-26T14:26:17.237509+0000 pg 48.7c not deep-scrubbed since 2025-06-23T18:57:42.886560+0000 pg 48.7b not deep-scrubbed since 2025-06-25T04:40:51.163596+0000 pg 59.71 not deep-scrubbed since 2025-06-23T22:57:42.807731+0000 pg 59.73 not deep-scrubbed since 2025-06-23T15:06:04.642059+0000 pg 58.72 not deep-scrubbed since 2025-06-26T08:28:44.995587+0000 pg 48.75 not deep-scrubbed since 2025-06-25T13:33:41.368882+0000 pg 48.74 not deep-scrubbed since 2025-06-22T18:15:30.731614+0000 pg 48.72 not deep-scrubbed since 2025-06-22T18:07:01.392496+0000 pg 48.71 not deep-scrubbed since 2025-06-23T15:40:23.134466+0000 pg 48.70 not deep-scrubbed since 2025-06-22T13:17:09.447156+0000 pg 59.66 not deep-scrubbed since 2025-06-24T21:58:12.342232+0000 pg 48.6d not deep-scrubbed since 2025-06-23T18:59:57.769702+0000 pg 58.66 not deep-scrubbed since 2025-06-22T18:23:49.451971+0000 pg 59.60 not deep-scrubbed since 2025-06-28T11:19:15.679226+0000 pg 58.60 not deep-scrubbed since 2025-06-27T14:13:20.771028+0000 pg 58.62 not deep-scrubbed since 2025-06-23T01:20:37.872422+0000 pg 58.6f not deep-scrubbed since 2025-06-27T13:40:17.031152+0000 pg 58.6b not deep-scrubbed since 2025-06-26T20:17:57.399397+0000 pg 59.6b not deep-scrubbed since 2025-06-25T07:55:45.671580+0000 pg 59.54 not deep-scrubbed since 2025-06-26T12:38:31.713680+0000 pg 48.5f not deep-scrubbed since 2025-06-27T07:22:23.643527+0000 pg 59.53 not deep-scrubbed since 2025-06-24T06:40:33.169587+0000 pg 48.58 not deep-scrubbed since 2025-06-27T17:31:04.072180+0000 pg 48.57 not deep-scrubbed since 2025-06-26T03:20:16.605485+0000 pg 48.56 not deep-scrubbed since 2025-06-28T05:41:54.098155+0000 pg 58.5f not deep-scrubbed since 2025-06-28T12:19:16.661402+0000 pg 48.52 not deep-scrubbed since 2025-06-24T14:49:29.295937+0000 pg 48.50 not deep-scrubbed since 2025-06-28T01:09:37.227387+0000 pg 48.4f not deep-scrubbed since 2025-06-20T01:16:09.369233+0000 pg 58.46 not deep-scrubbed since 2025-06-26T19:20:06.436318+0000 pg 59.40 not deep-scrubbed since 2025-06-28T13:21:24.493082+0000 pg 59.4d not deep-scrubbed since 2025-06-24T14:10:53.198832+0000 pg 58.4c not deep-scrubbed since 2025-06-22T18:24:20.137028+0000 pg 48.43 not deep-scrubbed since 2025-06-18T23:08:02.129787+0000 pg 59.49 not deep-scrubbed since 2025-06-26T03:05:00.546786+0000 pg 48.41 not deep-scrubbed since 2025-06-20T09:38:33.831563+0000 pg 48.40 not deep-scrubbed since 2025-06-22T21:41:26.468848+0000 pg 47.b not deep-scrubbed since 2025-06-28T11:03:41.301804+0000 pg 46.a not deep-scrubbed since 2025-06-26T14:59:32.312819+0000 220 more pgs... [WRN] POOL_TARGET_SIZE_BYTES_OVERCOMMITTED: 1 subtrees have overcommitted pool target_size_bytes Regards Dev

3 3

Community Manager Updates
by Anthony Middleton 10 Jul '25

10 Jul '25

Hey Ceph Users, I hope this message finds you well. I wanted to share a few ongoing activities with you. 1. Ceph Community Survey - https://forms.gle/2JTxcub6tMVLsDfh6 1. Your responses to this survey will help shape the Ceph Governing Board's strategies and the storytelling of the Ceph Communities as we approach Cephalocon 2025 and beyond. It should take no more than five minutes to complete. 2. Cephalocon 2025 Shirt Design Competition - https://ceph.io/en/news/blog/2025/cephalocon25-tshirt-contest/ 1. Don't miss the chance to feature your design on this year's Cephalocon conference t-shirt. 3. Last call for Cephalocon CFPs - https://sessionize.com/cephalocon-2025/ 1. Since I started working with the Ceph Community, one of my main goals has been to increase Ceph Users' involvement and visibility within the organization. Cephalocon offers an excellent platform for Ceph Users to share their experiences and inspire others. Please submit your talk by Saturday's deadline at 11:59 pm PDT. Feel free to reach out to me with any questions on Slack or via email at community-manager(a)ceph.io. Best, Anthony Middleton Ceph Community Manager community-manager(a)ceph.i

1 0

Re: "BUG: soft lockup" with MDS
by Janek Bevendorff 10 Jul '25

10 Jul '25

Yes, they are older Dell PowerEdges. I have to check whether there's newer firmware, but we've been running Ceph for years without these problems. I checked the logs on the host on which I had a lockup just an hour ago, but there's nothing besides the expected hardreset messages. There are two older watchdog messages, but they are from March: -------------------------------------------------------------------------------- SeqNumber = 2089 Message ID = ASR0000 Category = System AgentID = SEL Severity = Critical Timestamp = 2025-03-27 07:16:03 Message = The watchdog timer expired. RawEventData = 0x03,0x00,0x02,0x33,0xFB,0xE4,0x67,0x20,0x00,0x04,0x23,0x71,0x6F,0xC0,0x04,0xFF FQDD = WatchdogTimer.iDRAC.1 -------------------------------------------------------------------------------- SeqNumber = 2088 Message ID = ASR0000 Category = System AgentID = SEL Severity = Critical Timestamp = 2025-03-27 07:06:41 Message = The watchdog timer expired. RawEventData = 0x02,0x00,0x02,0x01,0xF9,0xE4,0x67,0x20,0x00,0x04,0x23,0x71,0x6F,0xC0,0x04,0xFF FQDD = WatchdogTimer.iDRAC.1 -------------------------------------------------------------------------------- I grepped the logs of another host where it happened, but couldn't find any watchtdog messages there. I believe it's also unlikely that suddenly all MDS hosts (we have five active, five hot standbys, and one cold standby) start having hardware issues. I also ran a memtest on one of the hosts last week and couldn't find anything there either. On 16/04/2025 15:14, Anthony D'Atri wrote: > Curious, are your systems Dells? If so you might see some improvement from running DSU to update all the firmware. It might also be illuminating to run `racadm lclog view` > >> On Apr 16, 2025, at 8:32 AM, Janek Bevendorff <janek.bevendorff(a)uni-weimar.de> wrote: >> >> Hi, >> >> Since the latest Reef update I have the problem that some of my hosts suddenly go into a state where all CPUs are stuck in kernel mode causing all daemons on that host to become unresponsive. When I connect to the IPMI console, I see a lot of messages like: >> >> watchdog: BUG: soft lockup - CPU#8 stuck for 47s! [cron:868840] >> >> (it's basically a list of all processes running on the machine). >> >> Usually, this resolves itself after several minutes, but sometimes I have to hardreset the host. When this happens, all daemons are marked as down and I cannot interact with the host at all. I don't know what causes this but, I think it happens primarily on the hosts where my MDS run and it seems to be triggered by events such as cluster rebalances, MDS restarts, or just randomly. >> >> I found a few reports about similar issues on the bug tracker and mailing list, but they are all very unspecific, unanswered, or more than 6 years old. >> >> Is there any way I can debug this? I upgraded to Squid already, but that didn't solve the problem. I also had massive issues with this during the upgrade. Particularly at the end when the MDS were upgraded, I had constant struggles with it. I had to set the noout flag and then literally sit next to it to resume the upgrade every few minutes until it finally went through, because random MDS hosts went intermittently dark all the time. >> >> All hosts run Ubuntu 22.04 with kernel 6.8.0. >> >> Any ideas? Thanks! >> Janek >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 13

RBD quota per namespace
by Eugen Block 10 Jul '25

10 Jul '25

Hi, I asked this question [0] five years ago, I haven't noticed anything wrt rbd quotas in the last releases. Has anyone an update on this topic? I'd appreciate it! Thanks, Eugen [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/ZCZ6MTFS645…

1 0

pg repair starts (endless loop ?)
by Dietmar Rieder 10 Jul '25

10 Jul '25

Hi, our ceph cluster reported an inconsistent pg, so we set it to repair: # ceph pg repair 4.b10 # ceph health detail HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent [ERR] OSD_SCRUB_ERRORS: 1 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 4.b10 is active+clean+scrubbing+deep+inconsistent+repair, acting [109,10,214,148,60,58,199,129,326,165,35] # ceph pg stat 6401 pgs: 1 active+clean+scrubbing+deep+inconsistent+repair, 147 active+clean+scrubbing, 146 active+clean+scrubbing+deep, 6107 active+clean; 1.1 PiB data, 1.5 PiB used, 2.4 PiB / 3.9 PiB avail; 19 MiB/s rd, 11 MiB/s wr, 14 op/s When checking the osd logs on the osd server for osd.109 we just see every couple of seconds repeats of the following messages (which I'm not aware of having seen before when we used pg repair on other occasions): [...] 2025-07-04T09:08:05.076+0000 7f3a063a7700 0 log_channel(cluster) log [INF] : osd.109 pg 4.b10s0 Deep scrub errors, upgrading scrub to deep-scrub 2025-07-04T09:08:05.076+0000 7f39f1b7e700 0 log_channel(cluster) log [DBG] : 4.b10 repair starts 2025-07-04T09:08:06.123+0000 7f3a063a7700 0 log_channel(cluster) log [INF] : osd.109 pg 4.b10s0 Deep scrub errors, upgrading scrub to deep-scrub 2025-07-04T09:08:06.123+0000 7f39f1b7e700 0 log_channel(cluster) log [DBG] : 4.b10 repair starts 2025-07-04T09:08:10.173+0000 7f3a063a7700 0 log_channel(cluster) log [INF] : osd.109 pg 4.b10s0 Deep scrub errors, upgrading scrub to deep-scrub 2025-07-04T09:08:10.196+0000 7f39f1b7e700 0 log_channel(cluster) log [DBG] : 4.b10 repair starts 2025-07-04T09:08:12.162+0000 7f3a063a7700 0 log_channel(cluster) log [INF] : osd.109 pg 4.b10s0 Deep scrub errors, upgrading scrub to deep-scrub 2025-07-04T09:08:12.162+0000 7f39f1b7e700 0 log_channel(cluster) log [DBG] : 4.b10 repair starts [...] Is this something to worry about? Best Dietmar

2 3

2025-Q3: Stable release recommendation for production clusters
by Özkan Göksu 10 Jul '25

10 Jul '25

Hello. I'm one of the oldest Nautilus 14.2.16 user in the community. It served with honor and great stability for more than 5 years. I want to thank you for its awesome developers. But the day has come and I decided to select a new stable and trusted release/version for the next 3-4 years. I will be so grateful if you share your advice and benchmark or test results to help me select the right Ceph release/version for my 7/24 running production clusters. I have 2 different clusters for specific usages: RBD Cluster: (Win10 VM use cases) - 30 nodes (15node <2x100Gbe 100m> 15node). - I have 2000++ rbd volumes of 50-100GB each. - I use daily snapshot with monthly cycle - I use KVM and libvirt drivers - SSD Pool-RBD: Replication 3 (%95) - SSD Pool-Cephfs: Replication 3 (%5) = For internal cluster usages -------------------------------------- RGW+CephFS Cluster: (S3 Muti-side) - 20 nodes (10node <1Gbit 400km> 10node). - HDD Pool-RGW-Data: 8+2 RGW-S3 use case (%70) - SSD Pool-RGW-MetaData: Replication 3 (%5) - SSD Pool-Cephfs: Replication 3 (%25) -------------------------------------- Let's check our options and what I'm thinking about them. 1. Quincy 17.2.9: is EOL but I never had any issue with it on RBD use in my test clusters. I have a trust feeling for RBD and it is a stable completed release. 2. Reef 18.2.7: Active development for one last month but I never tested it. I have trust issues but also I want to use the latest stable release for better RGW features and stability. 3. Squid 19.2.2: I never use the latest development version for my production but if there are important RGW, S3 or Multiside improvements I should not ignore then I might consider it. . Best regards.

4 5

Managing RGW container logs filling up disk space
by Sinan Polat 10 Jul '25

10 Jul '25

Hi all, I am running a Ceph Octopus cluster deployed with cephadm. I have three RGW nodes, the RGW containers are writing their logs to /var/lib/docker/containers/<container_id>/<container_id>-json.log Over time this causes the mount point to become 100% utilized. What is the recommended way to manage or limit these logs? - Should I configure a log driver in /etc/docker/daemon.json? If so, can this be done through Ceph/cephadm, or should it be configured manually on each node? - Would setting up a logrotate for /var/lib/docker/containers/*/*-json.log be appropriate? - Are there other best practices to control container log growth in this setup? Any advice or examples from others managing this would be greatly appreciated. Thanks! Sinan

2 1

System update on RL9 kills Cephadm host daemons
by Nicola Mori 09 Jul '25

09 Jul '25

Dear Ceph users, I noticed that updating a Rocky Linux 9 host with dnf in my cluster results in some of the Cephadm daemons vanishing. Before dnf update: $ sudo systemctl list-units | grep ceph ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)alertmanager.kili.service loaded active running Ceph alertmanager.kili for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)crash.kili.service loaded active running Ceph crash.kili for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)grafana.kili.service loaded active running Ceph grafana.kili for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)node-exporter.kili.service loaded active running Ceph node-exporter.kili for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.117.service loaded active running Ceph osd.117 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.118.service loaded active running Ceph osd.118 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.119.service loaded active running Ceph osd.119 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.120.service loaded active running Ceph osd.120 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.121.service loaded active running Ceph osd.121 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.122.service loaded active running Ceph osd.122 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.123.service loaded active running Ceph osd.123 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)osd.62.service loaded active running Ceph osd.62 for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 system-ceph\x2db1029256\x2d7bb3\x2d11ec\x2da8ce\x2dac1f6b627b45.slice loaded active active Slice /system/ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45.target loaded active active Ceph cluster b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph.target loaded active active All Ceph clusters and services After dnf update: $ sudo systemctl list-units | grep ceph ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45(a)node-exporter.kili.service loaded active running Ceph node-exporter.kili for b1029256-7bb3-11ec-a8ce-ac1f6b627b45 system-ceph\x2db1029256\x2d7bb3\x2d11ec\x2da8ce\x2dac1f6b627b45.slice loaded active active Slice /system/ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph-b1029256-7bb3-11ec-a8ce-ac1f6b627b45.target loaded active active Ceph cluster b1029256-7bb3-11ec-a8ce-ac1f6b627b45 ceph.target I guess this is due to dnf updating several Docker packages (docker-ce, docker-ce-cli etc.), but the same does not occur on the Rocky Linux 8 hosts, where after system updates all the Cephadm daemons are still running. Does anybody have an idea about why this might happen? Thanks, Nicola

1 0

reload SSL certificate in radosgw
by Boris 09 Jul '25

09 Jul '25

Hi, is there a way to reload the ceritificate in rgw without downtime? Or if I have multiple rgw daemons to do it one by one and wait for the last one to be active again? -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

5 8

Squid and Ganesha and ingress options?
by Nigel Williams 08 Jul '25

08 Jul '25

We deployed a new Cephadm Squid cluster (400 OSDs) and are happy with everything except this time we tried Ganesha again with the ingress option (HAproxy), but soon we ran into a failed daemon event. And there is a github issue for it: https://github.com/nfs-ganesha/nfs-ganesha/issues/1158 Our questions are: Is a hotfix pending for this github issue? Is anyone using Ganesha with or without ingress, is it reliable for you? How is the relative performance between Ganesha and kernel NFS? We seem to try Ganesha every release but return to the usual kernel NFS server setup because Ganesha is unreliable, we would be interested to know what others in the community think about the Ganesha option. thanks.

2 1

Re: Heads up: bad drive for Ceph Western Digital Ultrastar DC HC560
by Konstantin Shalygin 07 Jul '25

07 Jul '25

Hi, > On 4 Jul 2025, at 13:14, Marc <Marc(a)f1-outsourcing.eu> wrote: > > How is it worse than any other hdd of that size? At the moment we have just under 3000pcs of Toshiba MG10ACA and we have not registered any such issues k

2 1

RGW: is it possible to restrict a user access to a realm?
by Michel Jouvin 07 Jul '25

07 Jul '25

Hi, We have several RGW realms hosted in the same Ceph cluster. Looking at how to restrict access to one realm to some users (among all existing ones), I don't find the information. Looking at user/realm/zonegroup/zone parameters, I don't see anything that would allow this. I saw in https://docs.ceph.com/en/latest/radosgw/account/#radosgw-account a few words about tenant isolation but it is not clear for me if it is the same thing and how you achieve it. Thanks in advance for any hint! Best regards, Michel

1 0

Information
by Wissem MIMOUNA - Ceph Users 07 Jul '25

07 Jul '25

Hello , I'm wondering if you guys have some good knowledges about the Clay plugin for EC pools ? And what it compare to Jerasure for stability and performances ? https://docs.ceph.com/en/squid/rados/operations/erasure-code-clay/ Regards

2 1

Separate Pool Configuration for DR Zone in Ceph Multisite
by Vignesh Varma 07 Jul '25

07 Jul '25

Hi Team, I would like to know if it's possible to configure a separate pool for the DR zone in a Ceph multisite setup. I have two clusters, Zone A and Zone B. Currently, data synced from Zone A (CBE) to Zone B (TUP) resides in the default pool of Zone B. Instead, I want the replicated data from Zone A to be stored in its own separate pool in Zone B. Here are the steps I've used to set up the multisite configuration: Zone A: radosgw-admin realm create --rgw-realm=IN --default radosgw-admin zonegroup create --rgw-zonegroup=TN --master --default radosgw-admin zone create --rgw-zone=CBE --rgw-zonegroup=TN --master --default radosgw-admin user create --uid=replica --display-name="replica_user" --system radosgw-admin period update --commit Zone B: radosgw-admin realm pull --url=10.100.1.78 --access-key=xxx--secret=xxxx radosgw-admin realm default --rgw-realm=IN radosgw-admin zone create --rgw-zonegroup=TN --rgw-zone=TUP --access-key=xxx--secret=xxxx--endpoints=10.100.1.79 radosgw-admin period update --commit Could anyone guide me on how to achieve the separate pool configuration for the replicated data in Zone B? Thanks in advance! Best Regards, Vignesh Varma G Cloud Engineer www.stackbill.com

1 0

Heads up: bad drive for Ceph Western Digital Ultrastar DC HC560
by Konstantin Shalygin 04 Jul '25

04 Jul '25

Hi, When deploying new project, we discovered poor performance [1] of hard drives Western Digital Ultrastar DC HC560 (20TB, WUH722020BLE6L4). It consists in the fact that with an object size of less than 128k, the speed of the Ceph pool is equivalent to a USB flash drive. Be vigilant when make order. If anyone has such devices with a different firmware, please share the info Model Family: Western Digital Ultrastar DC HC560 Device Model: WDC WUH722020BLE6L4 Serial Number: 8LK364ER LU WWN Device Id: 5 000cca 2c7ebbaaf Firmware Version: PQGNW540 User Capacity: 20,000,588,955,648 bytes [20.0 TB] Sector Size: 4096 bytes logical/physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database 7.3/5625 ATA Version is: ACS-5 (minor revision not indicated) SATA Version is: SATA 3.5, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Thu Jul 3 08:18:32 2025 UTC SMART support is: Available - device has SMART capability. SMART support is: Enabled AAM feature is: Unavailable APM feature is: Disabled Rd look-ahead is: Enabled Write cache is: Disabled DSN feature is: Unavailable ATA Security is: Disabled, NOT FROZEN [SEC1] [1] https://tracker.ceph.com/issues/71927 Thanks, k

1 0

Where are you running Ceph?
by Anthony Fecarotta 04 Jul '25

04 Jul '25

Hello, I was wondering if there are any statistics on which platforms users/organizations are running Ceph on? We have been running Proxmox since day one, but are interested in migrating to something else that supports Ceph. This choice is mainly because of the lack of RGW UI, as well as the incompatibility with the Ceph dashboard for the last few versions. If there aren't any statistics available, I would be interested in hearing anyone's feedback and recommendations. Thank you. Regards, [image] Anthony Fecarotta Founder & President [image] anthony(a)linehaul.ai <mailto:anthony@linehaul.ai> [image] 224-339-1182 [image] (855) 625-0300 [image] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image] www.linehaul.ai <http://www.linehaul.ai/> [image] <http://www.linehaul.ai/> [image] <https://www.linkedin.com/in/anthony-fec/>

8 9

Question about object maps and index rebuilding.
by Gary Molenkamp 03 Jul '25

03 Jul '25

We use ceph rbd as a volume service for both an Openstack deployment and a series of Proxmox servers. This ceph deployment started as a Hammer release and has been upgraded over the years to where it is now running Quincy. It has been fairly solid over that time, even through upgrades from filestore to bluestore, and many transparent hardware replacements/improvements. One concern we have is that when we have a hypervisor that unexpectedly dies/crashes, the volumes must always have the object maps rebuilt. If we don't rebuild the object maps, the VMs will either not boot, or we will have other side-effects that render the volume unusable. (ie cannot mount root). Is this to be expected during this type of event or have I missed a setting during one of the many upgrade on our deployment? Thankfully the above does not happen regularly, but we would like to make use of the HA features of proxmox to ensure some VMs are always available. Requiring the rebuild step limits what can be done automatically, and how quickly it can be recovered. Any advice on ceph configuration, or how others may adapt to the requirements in HA situations would be appreciated. Cheers, Gary -- Gary Molenkamp Science Technology Services Systems Engineer University of Western Ontario molenkam(a)uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566

3 7

Notes from Ceph Steering Committee Weekly 2025-05-12
by Nizamudeen A 03 Jul '25

03 Jul '25

Hey Everyone, Sorry for the delay in sending this out but here's the summary of the short CSC weekly call that happened on Monday. *[Patrick [RFC] redmine upkeep script* Patrick updated everyone about the upkeep bot which automates several redmine tasks. A detailed mail has been sent out to the dev list which you can read here: https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/53LEWWH347Q4H7IYJC… *Status of v19.2.3* Was waiting on RGW team's approval. Full note can be seen here: https://pad.ceph.com/p/csc-weekly-minutes Regards, Nizamudeen A Sr. Software Engineer - IBM Partner Engineer IBM and Red Hat Ceph Storage Red Hat <https://www.redhat.com/> <https://www.redhat.com/>

1 1

RBD or CephFS / Docker Volumes
by Anthony Fecarotta 03 Jul '25

03 Jul '25

Hello -- i'm writing a docker compose file to launch a container inside a Proxmox VM. This is an open-source repo i found on github that i want to test, not our own software. It's going to use an external PSQL Database, that i have running on another LXC, but for the */app/hrms/* & */app/media* Docker Volumes, should I mount those to Ceph RBD or CephFS? Regards, * Anthony Fecarotta* Founder & President [image: phone-icon] anthony(a)linehaul.ai [image: phone-icon] 224-339-1182 [image: phone-icon] (855) 625-0300 [image: phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image: phone-icon] www.linehaul.ai [image: linehaul.ai logo] <http://www.linehaul.ai/> [image: linkedin icon] <https://www.linkedin.com/in/anthony-fec/>

2 2

Disk Utilisation Graph
by Devender Singh 03 Jul '25

03 Jul '25

Hello All I need to understand what below graph expalining and which disk utilisation it is? I have 8 NVME and 16 SSD and their weight is 1 with 20% disk utilised as per osd df. If keeping the weight of NVME and reweight SSD to some .8 and .9 will worth anything to lower this graph utilisation? Below is the graph under each host performance on CEPH dashboard. Regards Dev

4 4

Re: ceph squid - huge difference between capacity reported by "ceph -s" and "ceph df "
by Steven Vacaroaia 02 Jul '25

02 Jul '25

Hi Anthony Appreciate you taking the time to provide so much guidance All your messages in this mailing list are well documented and VERY helpful I have attached a text file with the output of the commands you mentioned You are right , there is no /dev/bluefs_/bluefsstore_bdev pointing to the NVME namespaces No specific reason to use separate WAL , just remembered it was recommended back in the "mimic" days ( yes, I am a bit "rusty") I am using 4+2 EC I have 2 separate NVMEs ( 1.6 TB each) dedicated to DB/WAL for HDD I want to use one for the 6 HDD that I currently have , and save the other for when I will be adding more HDDs The servers are SuperMicro SSG-641E with 2 x Intel GOLD 6530 ( 32 cores each) and 1TB RAM My plan was/is to use the 3 x 15TB NVME on each server for high performance pools (like metadata for cephfs or index for RGW) I have carved them in 3 ( 5 TB each) so I can deploy more PGs hence increase performance The SSD are meant to be used for RBD ( proxmox ) The HDD are meant to be used for archiving data using S3 and cephfs. I can redeploy - please clarify 2 things about osd spec file 1. can I use NVME namespaces ( created with nvme command - see below) or should I let ceph partition the NVME disk ? in either case, how do you specify WHICH device/NVME to use for DB/WAL ( as I have 2 identical ones and want to keep the second one for adding more HDD in the future ) nvme create-ns $device --nsze=$per_ns_blocks --ncap=$per_ns_blocks --flbas=0 --dps=0 nvme attach-ns $device --namespace-id=$i --controllers=`nvme id-ctrl $device 2. to deploy 3 different types of OSDs using the spec I have with the advice from you , I am guessing this is the correct spec service_type: osd service_id: hdd_osd crush_device_class: hdd_class placement: host_pattern: * spec: data_devices rotational: 1 db_devices: rotational: 0 size: 1000G:1600G filter_logic: AND objectstore: bluestore service_id: ssd_osd crush_device_class: ssd_class placement: host_pattern: * spec: data_devices rotational: 0 size: 6000G:8000G service_id: nvme_osd crush_device_class: nvme_class placement: host_pattern: * spec: data_devices rotational: 0 size: 4000G:5500G Many thanks Steven On Sun, 29 Jun 2025 at 17:45, Anthony D'Atri <aad(a)dreamsnake.net> wrote: > So you have NVMe SSD OSDs, SATA SSD OSDs, and HDD OSDs with offload onto > NVNe SSDs. > > Did you have a specific reason to explicitly specify wal_devices_. It’s > usually fine to just run with the default WAL size, with the WAL colocated > with the DB. And thus give your DB partitions a bit more space. > > What are your use-cases for these three classes of OSDs? Looks like you > have 42x 20T HDD OSDs, 63x NVMe OSDs, and 84x 7.6T SATA SSD OSDs? > Apparently with the 15T SSDs divided into 3x OSDs each? How much CPU do > you have on these nodes? Any specific reason to have chopped up the NVMe > SSDs into thirds? > > It looks to me as though your .mgr pool is using the default > replicated_rule, which does not specify a device class. This will confound > the balancer and if enabled the pg_autoscaler. > I recommend changing the .mgr pool to use the CRUSH rule that the > non-buckets.data pools use, which should be one that specifies > 3-replication constrained to one of the SSD device classes. As it is the > .mgr pool may be placed on any of the three device classes, which is > trivial with respect to space, but confounds as I mentioned. > > Or you could manually edit the CRUSH map and change the #0 replicated_rule > to specify nvme_class but it sounds like you’re new to Ceph and I don’t > want to frighten you with that process which unfortunately is still > old-school. Changing the rule as I suggested will be much safer. > > The numbers look like you have all of the RGW pools except buckets.data on > the nvme_class SSDs, which is fine, but you won’t begin to use all their > capacity, the index pool will maybe use 5-10% of the capacity used by your > buckets.data pool over time, depending on your distribution of object sizes > and the replication strategy of your buckets.data pool. Doing the math > I’ll speculate that your buckets.data pool is using a … EC 5+2 profile? > True? If so I might suggest rebuilding if/while you still can. There are > distinct advantages to having EC K+M < the number of OSD nodes. > > > > > Hi, > > Yes, I have separate NVME namespaces allocated for WAL and DB to each > spinning disk > > > Namespaces, or partitions? > > > Does that mean I still have to hunt for the 8TB culprit ? > > > Okay so `ceph cf` shows 8.2 TB of raw space used on the hdd_class OSDs, > that’s your concern, right? > > Please share outputs of the following: > > `ceph osd df` (showing a few of each device class) > `ceph osd dump | grep pool` > `ceph osd metadata NNNN | egrep /dev\|bluefs_\|bluestore_bdev` for at > least one OSD of each device class. And run it yourself without specifying > an OSD ID so it captures all, and see if all OSDs in each device class look > the same. > `ceph osd device ls-by-host ceph-host-1 > > It’s entirely possible that your WAL+DB aren’t actually offloaded to SSDs > as you intended. Advanced OSD service specs can be tricky. > > That’s my suspicion, that the WAL+DB are actually still on your HDDs. > Which can be migrated in-situ, or you can nuke the site from orbit and > redeploy. > > A note about your OSD specs. Specifying the models as you’re doing is > totally supported. But think about what happens if you add nodes in the > future that have different drive SKUs, or you RMA a drive and they send you > a different SKU as the replacement. > > It’s usually more future-proof to use a size range in the spec for each > osd service instead of `model`, with a bit of margin to account for base 2 > units vs base 10 units. > > Here’s an example that creates OSDs on SSDs between 490 and 1200 GB, this > is on systems that have ~ 1TB nominal drives. The systems also have 2TB > SATA SSDs that are used for WAL+DB offload, which are above the 1200GB > limit specified so they aren’t matched. > > service_type: osd > service_id: dashboard-admin-1705602677615 > service_name: osd.dashboard-admin-1705602677615 > placement: > host_pattern: * > spec: > data_devices: > rotational: 0 > size: 490G:1200G > filter_logic: AND > objectstore: bluestore > > And here is a spec that matches any HDD larger than 18T and deploys OSDs > on them without offload. This cluster has 20TB HDDs, so the range of 18+ > TB matches both the SEAGATE_ST20000NM007H and SEAGATE_ST20000NM002D drives > present. > > service_type: osd > service_id: cost_capacity > service_name: osd.cost_capacity > placement: > host_pattern: noactuallyusedanymore > spec: > data_devices: > rotational: 1 > size: '18T:' > filter_logic: AND > objectstore: bluestore > > Oh, and make sure that your HDDs and SSDs are all updated to the most > recent firmware. If you have Dell chassis, run DSU on the nodes and update > all firmware, but skip the OS drivers. If you have HP chassis, you can get > firmware update scripts from their web site, but I suspect these aren’t > HP. If anyone else, they’re likely generic drives and you can get firmware > updaters from the mfgs respective web sites. > > Then reboot nodes one at a time to effect the firmware, letting the > cluster completely recover between each reboot. > > > > > If yes , what would be the most efficient way of finding out what takes > the space ? > > Apologies for sending pictures but we are operating in an air gapped > environment > > I used this spec file to create the OSDs > > <image.png> > > Here is the osd tree of one of the servers > all the other 6 are similar > > <image.png> > > Steven > > > On Sun, 29 Jun 2025 at 14:25, Anthony D'Atri <aad(a)dreamsnake.net> wrote: > >> WAL by default rides along with the DB and rarely warrants a separate or >> larger allocation. >> >> Since you say you’ve allocated DB space, does that mean that you have >> WAL+DB offloaded onto SSDs? If so they don’t contribute to the space used >> on the hdd device class. >> >> >> > On Jun 29, 2025, at 1:56 PM, Steven Vacaroaia <stef97(a)gmail.com> wrote: >> > >> > Hi Janne >> > >> > Thanks >> > That make sense since I have allocated 196GB for DB and 5 GB for WALL >> for >> > all 42 spinning OSDs >> > Again, thanks >> > Steveb >> > >> > On Sun, 29 Jun 2025 at 12:02, Janne Johansson <icepic.dz(a)gmail.com> >> wrote: >> > >> >> Den sön 29 juni 2025 kl 17:22 skrev Steven Vacaroaia <stef97(a)gmail.com >> >: >> >> >> >>> Hi, >> >>> >> >>> I just built a new CEPH squid cluster with 7 nodes >> >>> Since this is brand new, there is no actuall data on it except few >> test >> >>> files in the S3 data.bucket >> >>> >> >>> Why is "ceph -s" reporting 8 TB of used capacity ? >> >>> >> >> >> >> Because each OSD will have GBs of preallocated data for the RocksDB, >> >> write-ahead-logs and other structures, and this counts against "raw >> >> available space", even if you don't have objects of this size put into >> the >> >> pools, the creation of the DBs and other things happened at osd >> creation, >> >> or when the first object was made, and are there even if you delete the >> >> object later. >> >> >> >> -- >> >> May the most significant bit of your life be positive. >> >> >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >

3 4

doc: Is there a ceph-users list archive that archives before June 2019?
by Zac Dover 02 Jul '25

02 Jul '25

Here is the archive of the ceph-users mailing list that docs.ceph.com and ceph.io link to: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/ This archive reaches back in time only to June of 2019. But I would like to refer to an email that was sent in 2017. Is anyone aware of an archive of ceph-users that reaches that far back in time? Zac

2 1

CephFS with Ldap
by gagan tiwari 01 Jul '25

01 Jul '25

Hi Guys, We have a Ldap server with all users login details. We have to mount data stored in Ceph to several client nodes via CephFS so that users can access that data and start using that data in their processes. But we need to grant permission / ownership to users to enable them to access that data. like chown user:group /dirs ( on Linux ) How will cephfs recognize users , groups that are in Ldap ? Will I need to set-up Ldap authentication on all nodes in Ceph cluster for this purpose ( ceph mgr , ceph mons , ceph mds and all ods nodes ) Please advise. Thanks, Gagan

5 4

orchestrator not behaving strangely
by Holger Naundorf 01 Jul '25

01 Jul '25

Hello, we are running a ceph cluster at version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable) and since a few weeks the orchestrator started to misbehave - up to now we could not identify any root cause, so I am fishing in the community to see if there are any hints. Problems: An OSD removal (for disk replacement) gets stuck in the 'purge' step: ceph orch osd rm 406 --replace root@aadm01:~# ceph orch osd rm status OSD HOST STATE PGS REPLACE FORCE ZAP DRAIN STARTED AT 406 acn07 done, waiting for purge 0 True False True 2025-06-25 09:18:07.650734+00:00 (now for more than 24h in this state) At the same time the orchestrator is not restarting OSD daemons - i.e. an 'ceph orch daemon restart osd.xxx' claims its queuing uo the restart, but it never happens. Other services continue to be controlled correctly via 'ceph orch ...' If anyone has an idea where to poke around or can match this to some known problem - I would appreciate any pointers. Regards, Holger -- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naundorf(a)rz.uni-kiel.de

3 8

RADOS Gateway IAM for Production Environments ( reef to squid upgrade )
by ramodav＠yahoo.com 01 Jul '25

01 Jul '25

Hi Everyone ! I am planning to upgrade ceph from reef to squid to benefit from RGW IAM .Does anybody know is it still a tech-preview ?Would an update be a nice solution to have this feature ?

3 3

Cephfs performance on a small configuration
by Patrick Begou 01 Jul '25

01 Jul '25

Hi everybody, I've setup a small ceph cluster for a cephfs service. - 5 nodes in Almalinux9 and Squid version of Ceph - 4 HDD per node (so 20 HDD in the cluster) - Erasure coding in k=6+m=2 (ceph osd erasure-code-profile set ec-62-profile-isa k=6 m=2 crush-failure-domain=host plugin=isa) - two chunks per node (I add un "virtual" host bucket on each nodes and move 2 HDD to it) - only one network interface (ethernet 25Gb/s) I'm testing I/Os from another cluster (for computation) with a workflow similar to our needs: several mpi processes writing each in a different file. Node is connected to the same switch, 25Gb ethernet, same VLAN (no router). 1 process writing/reading 20GB shows (around) *700MB/s writing and 600MB/s reading* so far from the maximum network bandwidth 4 processes writing/reading each 5GB (so 20GB aggregated) show (around) *900MB/s *(225MB/s per process)*writing and 800MB/s reading* (200MB/s per process) Is this what I can expect from the actual setup or cluster architecture (I do not find any reference) or is something wrong in my first Ceph cluster setup. Of course the goal is to add HDD (for capacitive storage) and servers in the next years, this is the starting point of a Ceph deployment in the laboratory. I'm checking against the performance of my old NFS server (10Gb ethernet, two cluster of 8 HDDS in HW raid6, stripped) and the throughput improvement is small with my test. Thanks Patrick [ceph: root@whitaker01-ceph /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 254.66797 root default -11 25.46680 host whitaker01-ceph 4 hdd 12.73340 osd.4 up 1.00000 1.00000 13 hdd 12.73340 osd.13 up 1.00000 1.00000 -13 25.46680 host whitaker01-virt 7 hdd 12.73340 osd.7 up 1.00000 1.00000 18 hdd 12.73340 osd.18 up 1.00000 1.00000 -7 25.46680 host whitaker02-ceph 0 hdd 12.73340 osd.0 up 1.00000 1.00000 11 hdd 12.73340 osd.11 up 1.00000 1.00000 -15 25.46680 host whitaker02-virt 8 hdd 12.73340 osd.8 up 1.00000 1.00000 17 hdd 12.73340 osd.17 up 1.00000 1.00000 -5 25.46680 host whitaker03-ceph 2 hdd 12.73340 osd.2 up 1.00000 1.00000 12 hdd 12.73340 osd.12 up 1.00000 1.00000 -17 25.46680 host whitaker03-virt 5 hdd 12.73340 osd.5 up 1.00000 1.00000 15 hdd 12.73340 osd.15 up 1.00000 1.00000 -3 25.46680 host whitaker04-ceph 3 hdd 12.73340 osd.3 up 1.00000 1.00000 10 hdd 12.73340 osd.10 up 1.00000 1.00000 -19 25.46680 host whitaker04-virt 6 hdd 12.73340 osd.6 up 1.00000 1.00000 16 hdd 12.73340 osd.16 up 1.00000 1.00000 -9 25.46680 host whitaker05-ceph 1 hdd 12.73340 osd.1 up 1.00000 1.00000 14 hdd 12.73340 osd.14 up 1.00000 1.00000 -21 25.46680 host whitaker05-virt 9 hdd 12.73340 osd.9 up 1.00000 1.00000 19 hdd 12.73340 osd.19 up 1.00000 1.00000

1 0

Loki retention policy missing?
by Stefan Kooman 01 Jul '25

01 Jul '25

Hi, Yesterday we got a disk space warning notification on one of the monitor nodes. Most space consumption was in use by a running Loki container. Looking into online documentation / trackers to find info on Ceph's default Loki retention policy left me empty handed. I did fine some trackers for Prometheus retention policy in the past, so I figured there might be none yet. The documentation states that you can find the jinja2 templates used for the service here: src/pybind/mgr/cephadm/templates So I changed this template to include a retention policy and a compactor like: ... orig template ... compactor: working_directory: /tmp/loki/compactor shared_store: filesystem compaction_interval: 10m retention_enabled: true retention_delete_delay: 2h retention_delete_worker_count: 150 limits_config: retention_period: 2160h split_queries_by_interval: 24h ceph config-key set mgr/cephadm/services/loki/loki.yml -i $PWD/loki_with_retention_policy.yml.j2 and redeployed loki (ceph orch daemon redeploy loki.mon1). That gained a lot of free space ... as in ... all Loki logs where gone and it had started from scratch. This "/tmp/loki" working directory is ... odd? How is this supposed to work in Ceph? Thanks, Gr. Stefan

1 0