June 2025 - ceph-users - lists.ceph.io

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

CephFS: no MDS does join the filesystem
by Robert Sander 24 Mar '26

24 Mar '26

Hi, we are having an issue at a customer site where a 3PB CephFS is in failed state. The cluster itself is unhealthy and awaits replacements disks: # ceph -s cluster: id: 28ca2bfa-d87e-11ed-83a3-1070fddda30f health: HEALTH_ERR 4 failed cephadm daemon(s) There are daemons running an older version of ceph 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged 8 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 46 pgs backfill_toofull Possible data damage: 4 pgs inconsistent Degraded data redundancy: 6427646/15858772167 objects degraded (0.041%), 8 pgs degraded, 8 pgs undersized 6 pool(s) nearfull (muted: OSDMAP_FLAGS OSD_SCRUB_ERRORS(2d) PG_NOT_DEEP_SCRUBBED PG_NOT_SCRUBBED) services: mon: 3 daemons, quorum sn01,sn03,sn02 (age 3w) mgr: sn03.crlpzh(active, since 33h), standbys: sn01.tegfya, sn02.mzvgcr mds: 18/19 daemons up, 1 standby osd: 181 osds: 174 up (since 4d), 172 in (since 4d); 206 remapped pgs flags nodeep-scrub data: volumes: 2/3 healthy, 1 recovering; 1 damaged pools: 12 pools, 3585 pgs objects: 1.93G objects, 1.3 PiB usage: 2.5 PiB used, 501 TiB / 3.0 PiB avail pgs: 6427646/15858772167 objects degraded (0.041%) 293845758/15858772167 objects misplaced (1.853%) 2844 active+clean 532 active+clean+scrubbing 147 active+remapped+backfill_wait 28 active+remapped+backfill_toofull 11 active+remapped+backfill_wait+backfill_toofull 10 active+remapped+backfilling 6 active+undersized+degraded+remapped+backfill_toofull 2 active+clean+inconsistent 1 active+clean+scrubbing+deep+inconsistent+repair 1 active+undersized+remapped+backfilling 1 active+undersized+degraded+remapped+backfilling 1 active+recovering+degraded+remapped 1 active+remapped+inconsistent+backfill_toofull io: recovery: 183 MiB/s, 312 objects/s The CephFS metadata pool is not affected by the inconsistent PGs. The MDSs have this line in their logfile: "Monitors have assigned me to become a standby." The filesystem is joinable: # ceph fs lsflags storage_cluster joinable allow_snaps allow_multimds_snaps refuse_client_session But no MDS joins: # ceph fs status storage_cluster - 0 clients =============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed POOL TYPE USED AVAIL cephfs_metadata metadata 490G 12.9T cephfs_data data 970T 54.1T shared_data data 1351T 22.5T STANDBY MDS storage_cluster.sn04.cbvzzu MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) Why? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

6 13

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Ceph orchestrator not refreshing device list
by Bob Gibson 14 Sep '25

14 Sep '25

Hi, We recently converted a legacy cluster running Quincy v17.2.7 to cephadm. The conversion went smoothly and left all osds unmanaged by the orchestrator as expected. We’re now in the process of converting the osds to be managed by the orchestrator. We successfully converted a few of them, but then the orchestrator somehow got confused. `ceph health detail` reports a “stray daemon” for the osd we’re trying to convert, and the orchestrator is unable to refresh its device list so it doesn’t see any available devices. From the perspective of the osd node, the osd has been wiped and is ready to be reinstalled. We’ve also rebooted the node for good measure. `ceph osd tree` shows that the osd has been destroyed, but the orchestrator won’t reinstall it because it thinks the device is still active. The orchestrator device information is stale, but we’re unable to refresh it. The usual recommended workaround of failing over the mgr hasn’t helped. We’ve also tried `ceph orch device ls —refresh` to no avail. In fact after running that command subsequent runs of `ceph orch device ls` produce no output until the mgr is failed over again. Is there a way to force the orchestrator to refresh its list of devices when in this state? If not, can anyone offer any suggestions on how to fix this problem? Cheers, /rjg P.S. Some additional information in case it’s helpful... We’re using the following command to replace existing devices so that they’re managed by the orchestrator: ``` ceph orch osd rm <osd> --replace —zap ``` and we’re currently stuck on osd 88. ``` ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.88 on host ceph-osd31 not managed by cephadm ``` `ceph osd tree` shows that the osd has been destroyed and is ready to be replaced: ``` ceph osd tree-from ceph-osd31 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -46 34.93088 host ceph-osd31 84 ssd 3.49309 osd.84 up 1.00000 1.00000 85 ssd 3.49309 osd.85 up 1.00000 1.00000 86 ssd 3.49309 osd.86 up 1.00000 1.00000 87 ssd 3.49309 osd.87 up 1.00000 1.00000 88 ssd 3.49309 osd.88 destroyed 0 1.00000 89 ssd 3.49309 osd.89 up 1.00000 1.00000 90 ssd 3.49309 osd.90 up 1.00000 1.00000 91 ssd 3.49309 osd.91 up 1.00000 1.00000 92 ssd 3.49309 osd.92 up 1.00000 1.00000 93 ssd 3.49309 osd.93 up 1.00000 1.00000 ``` The cephadm log shows a claim on node `ceph-osd31` for that osd: ``` 2024-09-25T14:15:45.699348-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims -> {'ceph-osd31': ['88']} 2024-09-25T14:15:45.699534-0400 mgr.ceph-mon3.qzjgws [INF] Found osd claims for drivegroup ceph-osd31 -> {'ceph-osd31': ['88']} ``` `ceph orch device ls` shows that the device list isn’t refreshing: ``` ceph orch device ls ceph-osd31 HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS ceph-osd31 /dev/sdc ssd INTEL_SSDSC2KG038T8_PHYG039603PE3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdd ssd INTEL_SSDSC2KG038T8_PHYG039600AY3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sde ssd INTEL_SSDSC2KG038T8_PHYG039600CW3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdf ssd INTEL_SSDSC2KG038T8_PHYG039600CM3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdg ssd INTEL_SSDSC2KG038T8_PHYG039600UB3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdh ssd INTEL_SSDSC2KG038T8_PHYG039603753P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdi ssd INTEL_SSDSC2KG038T8_PHYG039603R63P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdj ssd INTEL_SSDSC2KG038TZ_PHYJ4011032M3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdk ssd INTEL_SSDSC2KG038TZ_PHYJ3234010J3P8DGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ceph-osd31 /dev/sdl ssd INTEL_SSDSC2KG038T8_PHYG039603NS3P8EGN 3576G No 22h ago Insufficient space (<10 extents) on vgs, LVM detected, locked ``` `ceph node ls` thinks the osd still exists ``` ceph node ls osd | jq -r '."ceph-osd31"' [ 84, 85, 86, 87, 88, <— this shouldn’t exist 89, 90, 91, 92, 93 ] ``` Each osd node has 10x 3.8 TB ssd drives for osds. On `ceph-osd31`, cephadm doesn’t see osd.88 as expected: ``` cephadm ls --no-detail [ { "style": "cephadm:v1", "name": "osd.93", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.93" }, { "style": "cephadm:v1", "name": "osd.85", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.85" }, { "style": "cephadm:v1", "name": "osd.90", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.90" }, { "style": "cephadm:v1", "name": "osd.92", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.92" }, { "style": "cephadm:v1", "name": "osd.89", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.89" }, { "style": "cephadm:v1", "name": "osd.87", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.87" }, { "style": "cephadm:v1", "name": "osd.86", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.86" }, { "style": "cephadm:v1", "name": "osd.84", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.84" }, { "style": "cephadm:v1", "name": "osd.91", "fsid": "9b3b3539-59a9-4338-8bab-3badfab6e855", "systemd_unit": "ceph-9b3b3539-59a9-4338-8bab-3badfab6e855(a)osd.91" } ] ``` `lsblk` shows that `/dev/sdg` has been wiped. ``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 223.6G 0 disk |-sda1 8:1 0 94M 0 part `-sda2 8:2 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdb 8:16 0 223.6G 0 disk |-sdb1 8:17 0 94M 0 part `-sdb2 8:18 0 223.5G 0 part `-md0 9:0 0 223.4G 0 raid1 / sdc 8:32 1 3.5T 0 disk `-ceph--03782b4c--9faa--49f5--b554--98e7b8515834-osd--block--ba272724--daa6--45f5--9f69--789cc0bda077 253:3 0 3.5T 0 lvm `-keCkP2-o6h8-jKkw-RKiD-UBFf-A8EL-JDJGPR 253:9 0 3.5T 0 crypt sdd 8:48 1 3.5T 0 disk `-ceph--c07907d8--4a75--4ba3--b5e1--2ebf49ecbdf6-osd--block--58d1d50d--6228--4e6f--9a52--2a305ba00700 253:7 0 3.5T 0 lvm `-WB8Mxn-qCHI-4T01-imiG-hNBR-by60-YuxgfD 253:11 0 3.5T 0 crypt sde 8:64 1 3.5T 0 disk `-ceph--6f9d4df4--7ce6--44a4--a7b1--62c85af8cfe0-osd--block--aabcb30d--0084--490a--969b--78f7af6e94da 253:8 0 3.5T 0 lvm `-g9qErH-vTXY-JQbs-eh61-W0Mn-TAV8-gof4zy 253:12 0 3.5T 0 crypt sdf 8:80 1 3.5T 0 disk `-ceph--d6b728f8--e365--46db--b30f--6c00805c752b-osd--block--88426db7--2322--4807--ac2e--b49929e170d6 253:6 0 3.5T 0 lvm `-LNG2gB-pa0w-gl2v-hVQ3-6qTd-aXsR-Lenri3 253:10 0 3.5T 0 crypt sdg 8:96 1 3.5T 0 disk sdh 8:112 1 3.5T 0 disk `-ceph--de2cfee6--8e0a--4aa0--9e6b--90c09025768c-osd--block--a3b86251--2799--4243--a857--f218fa90f29a 253:2 0 3.5T 0 lvm sdi 8:128 1 3.5T 0 disk `-ceph--30dee450--0fdd--46ea--9eec--6a4c7706df9c-osd--block--bfc090db--dde4--47dd--a1c9--1cd838ea43b3 253:4 0 3.5T 0 lvm sdj 8:144 1 3.5T 0 disk `-ceph--78febcf5--43f4--4820--8dc7--0f6c22816c9f-osd--block--da1e69c7--6427--4562--8290--90bcb9526747 253:0 0 3.5T 0 lvm sdk 8:160 1 3.5T 0 disk `-ceph--fe210281--b1f5--4d5e--9ab0--2f226612af00-osd--block--6bb9f308--e853--4303--83ea--553c3a3513e1 253:1 0 3.5T 0 lvm sdl 8:176 1 3.5T 0 disk `-ceph--9f21c916--f211--4d1b--8214--6ad1cecac810-osd--block--572d850c--c201--4af4--ac42--0ed2a6ed73ed 253:5 0 3.5T 0 lvm ```

4 22

SuSE stops building Ceph packages for its distributions
by Robert Sander 15 Aug '25

15 Aug '25

Hi, SuSE will not build even the Ceph client packages for its distributions any more: https://lists.opensuse.org/archives/list/packaging@lists.opensuse.org/threa… https://www.suse.com/releasenotes/x86_64/SUSE-SLES/15-SP6/index.html#jsc-PE… This means that this distribution cannot consume Ceph storage any more, if no other party will build these packages. If you are using SuSE Linux Enterprise Server or OpenSuSE as a client for Ceph storage you need to plan for this. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

4 4

Announcing go-ceph v0.32.0
by Sven Anderson 12 Aug '25

12 Aug '25

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.32.0 The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Sven

1 1

Debugging OSD cache thrashing
by Hector Martin 11 Aug '25

11 Aug '25

Hi all, I have a small 3-node cluster (4 HDD + 1 SSD OSD per node, ceph version 19.2.2) and noticed that during snaptrim ops (?), the OSDs max out CPU usage and cluster performance plummets. Most of the CPU usage was accounted as "system", while actual disk I/O usage was low, so that didn't sound right. I perf traced the system, and found that most of the usage is in __arch_copy_to_user in the kernel, in read() syscalls. That sounds like the OSD is thrashing buffered block device reads which the kernel satisfies from the page cache (hence no real I/O load, most of the CPU usage is from data copies), so it's repeatedly reading the same disk blocks, which doesn't sound right. I increased osd_memory_target to 2.4G (from 1.2G) live with `ceph config`, and CPU usage immediately dropped to near zero. However, waiting a bit eventually the CPU thrashing returns after memory usage increases. Restarting an OSD has a similar effect. I believe that something is wrong with the OSD bluestore cache allocation/flush policy, and when the cache becomes full it starts thrashing reads instead of evicting colder cached data (or perhaps some cache bucket is starving another cache bucket of space). I would appreciate some hints on how to debug this. Are there any cache stats I should be looking at, or info on how the cache is partitioned? Here is a perf call trace of the thrashing: > - OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*) > - 95.04% ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&) > - OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&) > - 95.03% PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&) > - PGBackend::handle_message(boost::intrusive_ptr<OpRequest>) > - ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>) > - 88.86% ECBackend::handle_sub_write(pg_shard_t, boost::intrusive_ptr<OpRequest>, ECSubWrite&, ZTracer::Trace const&, ECListener&) > - 88.78% non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>) > - BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusiv > - 88.75% BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*) > - 88.71% BlueStore::_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&) > - BlueStore::_do_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&) > - 88.69% BlueStore::_do_truncate(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&, unsigned long, std > - 88.68% BlueStore::_wctx_finish(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&, BlueStore::Writ > - 88.65% BlueStore::Collection::load_shared_blob(boost::intrusive_ptr<BlueStore::SharedBlob>) > - 88.64% RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::ch > - 88.61% rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*) > - rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::DBImpl::GetImplOptions&) > - 88.43% rocksdb::Version::Get(rocksdb::ReadOptions const&, rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::PinnableWideColumns*, std > - 88.41% rocksdb::TableCache::Get(rocksdb::ReadOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, rocksdb::Sl > - rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::SliceTransform const*, bo > - 88.32% rocksdb::BlockBasedTable::FullFilterKeyMayMatch(rocksdb::FilterBlockReader*, rocksdb::Slice const&, bool, rocksdb::SliceTransfor > - rocksdb::FullFilterBlockReader::MayMatch(rocksdb::Slice const&, bool, rocksdb::GetContext*, rocksdb::BlockCacheLookupContext*, rocks > - 88.29% rocksdb::FilterBlockReaderCommon<rocksdb::ParsedFullFilterBlock>::GetOrReadFilterBlock(bool, rocksdb::GetContext*, rocksdb > - rocksdb::FilterBlockReaderCommon<rocksdb::ParsedFullFilterBlock>::ReadFilterBlock(rocksdb::BlockBasedTable const*, rocksdb::Fi > - rocksdb::Status rocksdb::BlockBasedTable::RetrieveBlock<rocksdb::ParsedFullFilterBlock>(rocksdb::FilePrefetchBuffer*, rocks > - rocksdb::Status rocksdb::BlockBasedTable::MaybeReadBlockAndLoadToCache<rocksdb::ParsedFullFilterBlock>(rocksdb::FilePref > - 88.25% rocksdb::BlockFetcher::ReadBlockContents() [ note: split, the rest is + 19.42% rocksdb::VerifyBlockChecksum] > - 68.80% rocksdb::RandomAccessFileReader::Read(rocksdb::IOOptions const&, unsigned long, unsigned long, rocksdb::Sli [ note: at this level the call trace splits into 4, but it all leads to the same place ] > - 35.99% 0xaaaac0992e40 > BlueRocksRandomAccessFile::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const > BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long, char*) > - KernelDevice::read_random(unsigned long, unsigned long, char*, bool) > - 35.98% __libc_pread > - el0_svc > - invoke_syscall > - 35.97% __arm64_sys_pread64 > - 35.96% vfs_read > - blkdev_read_iter > - 35.93% filemap_read > - 35.28% copy_page_to_iter > 35.01% __arch_copy_to_user - Hector

7 17

Squid 19.2.1 dashboard javascript error
by Chris Palmer 09 Aug '25

09 Aug '25

Firstly thank you so much for the 19.2.1 release. Initial testing suggests that the blockers that we had in 19.2.0 have all been resolved, so we are proceeding with further testing. We have noticed one small problem in 19.2.1 that was not present in 19.2.0 though. We use the older-style dashboard (mgr/dashboard/FEATURE_TOGGLE_DASHBOARD false). The problem happens on the Dashboard screen when health changes to WARN. If you click on WARN you get a small empty dropdown instead of the list of warnings. A javascript error is logged, and using browser inspection there is the additional bit of info that it happens in polyfill: 2025-02-07T15:59:00.970+0000 7f1d63877640 0 [dashboard ERROR frontend.error] (https://<redacted>:8443/#/dashboard): NG0901 Error: NG0901 at d.find (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3342365) at le.ngDoCheck (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3173112) at Qe (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225586) at bt (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225341) at cs (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3225051) at $m (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259043) at jf (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3266563) at S1 (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259790) at $m (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3259801) at fg (https://<redacted>:8443/main.7869bccdd1b73f3c.js:3:3267248) Also, after this happens, no dropdowns work again until the page is forcibly refreshed. Environment is RPM install on Centos 9 Stream. I've created issue [0]. Thanks, Chris [0] https://tracker.ceph.com/issues/69867 <https://tracker.ceph.com/issues/69867?next_issue_id=69865&prev_issue_id=90>

4 9

RBD Mirror - Failed to unlink peer
by Scott Cairns 01 Aug '25

01 Aug '25

Hi, Following the introduction of an additional node to our Ceph cluster, we've started to see unlink errors when taking a rbd mirror snapshot. We've had RBD mirroring configured for over a year now and it's been working flawlessly, however after we created OSD's on a new node we've receiving the following error: librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f60c80056f0 handle_unlink_peer: failed to unlink peer: (2) No such file or directory This seemed to appear on around 3 of 150 snapshots on the first night and over the weeks has progressed to almost every snapshot. What's odd, is that the snapshot appears to be taken without any issues and does mirror to the DR site - we can see the snapshot ID taken on the source side is mirrored to the destination side when checking the rbd snap ls, and we've tested promoting an image on the DR site to ensure the snapshot does include up to date data, which it does. I can't see any other errors generated when the snapshot is taken to identify what file/directory isn't found - everything appears to be working okay it's just generating an error during the snapshot. I've also tried disabling mirroring on the disk and re-enabling however it doesn't appear to make any difference - there's no error on the initial mirror image, or the first snapshot taken after that, but every subsequent snapshot shows the error again. Any ideas? Thanks, Scott ________________________________ The content of this e-mail and any attachment is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of Tecnica Limited. If you have received this e-mail in error please notify the sender. Any use, dissemination, forwarding, printing, or copying of this e-mail or any attachments thereto, in whole or part, without permission is strictly prohibited. Tecnica Limited Registered office: 5 Castle Court, Carnegie Campus, Dunfermline, Fife, KY11 8PB. Registered in Scotland No. SC250307. VAT No. 827 5110 42. This footnote also confirms that this email message has been swept for the presence of computer viruses.

4 8

How to create buckets in secondary zonegroup?
by Adam Prycki 29 Jul '25

29 Jul '25

Hello, I have a question about ceph multisite theory. How to create buckets in non-master zonegroup? I'm trying to create crate a test enviroment with 2 regions. I've crated deployment which looks like this REALM region-1 (master zonegroup) region-1-zone-a (master zone) region-1-zone-b region-2 region-2-zone-a (master zone) The goal would be to have 2 regions, each with separate data and 2 active-active zones in each. Just after creating region-2-zone-a I've decided to create a bucket there in region-2 and upload something. But I cannot create a bucket in new region. Rgw allows you to create buckets only on master zone. If I try to use s3cmd to create bucket on region-1-zone-b endpoint I get error 503. (as expected) Trying to list region-1 bucket on region-2 zone gives me 301 (moved permanently) so region-2 has metadata and working authentication. But I cannot figure out how to make a bucket in region-2. s3cmd .s3cfg-region-2 mb s3://bucket gives me error 503 like if I were trying to create a bucket on non-master zone. Creating bucket on region-1 endpoint with s3cmd flags specifying LocationConstraint region-2 creates bucket in region-1 I see in region-1-zone-a logs that rgw sees region-2 in LocationConstraint argument but creates bucket in region-1 I'm working on ceph 19.2.2 Best regards Adam Prycki

2 5

CephFS limp mode when fullest OSD is between nearfull & backfillfull value
by Jeremi-Ernst Avenant 28 Jul '25

28 Jul '25

Good day We've been struggling with this issue since we've upgraded post 16.2.11 to 16.2.15 and now up to Reef 18.2.7. We didn't have this issue prior to upgrading from 16.2.11 to 16.2.15. The moment the fullest OSD % is between the nearful and backfillful value, CephFS goes into a limp mode. Meaning our Client IO goes from ~30 *GiB*/s to 100 *MiB*/s. Doesn't matter what the threshold values are set. If my nearful% is 75% and backfillfull 80%, with the highest OSD 75.01% it will limp. If my nearful% is 85% and backfillfull 90%, with the highest OSD 85.01% it will go into limp mode. The only way to get CephFS operational again, is to set both ratios to either far below or above the highest OSD value. e.g. if the fullest is 72% , I need to set it to ceph osd set-nearfull-ratio 0.76 & ceph osd set-backfillfull-ratio 0.81 When I deploy a brand new cluster from scratch on our testbed I get the same issue, Cephadm / Reef 18.2.4. Back in the days I was able to manipulate these settings with the following as well, but I think since Pacific they are no longer tuneable: ceph tell osd.1231 config show | grep -E 'osd_nearfull_ratio|osd_backfillfull_ratio|osd_full_ratio' "mon_osd_backfillfull_ratio": "0.900000", "mon_osd_full_ratio": "0.950000", "mon_osd_nearfull_ratio": "0.850000", ceph tell osd.$osd injectargs '--osd_nearfull_ratio=0.85' ceph tell osd.$osd injectargs '--osd_backfillfull_ratio=0.90' ceph tell osd.$osd injectargs '--osd_full_ratio=0.95' URL to the issue: https://tracker.ceph.com/issues/70129 Any ideas would be greatly appreciated. -- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi(a)idia.ac.za <mfundo(a)idia.ac.za> Rondebosch, Cape Town, 7600, South Africa

2 2

2 MDSs behind on trimming on my Ceph Cluster since the upgrade from 18.2.6 (reef) to 19.2.2 (squid)
by Edouard FAZENDA 25 Jul '25

25 Jul '25

Dear all, I have the following issue on my Ceph cluster MDSs behind on trimming on my Ceph Cluster since the upgrade using cephadm from 18.2.6 to 19.2.2. Here some cluster logs : 8/5/25 09:00 AM[WRN]overall HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:50 AM[WRN]overall HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:40 AM[WRN]mds.cephfs.node2.isqjza(mds.0): Behind on trimming (326/128) max_segments: 128, num_segments: 326 8/5/25 08:40 AM[WRN]mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming (326/128) max_segments: 128, num_segments: 326 8/5/25 08:40 AM[WRN][WRN] MDS_TRIM: 2 MDSs behind on trimming 8/5/25 08:40 AM[WRN]Health detail: HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:33 AM[WRN]Health check update: 2 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:33 AM[WRN]Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:30 AM[INF]overall HEALTH_OK 8/5/25 08:22 AM[INF]Cluster is now healthy 8/5/25 08:22 AM[INF]Health check cleared: MDS_TRIM (was: 1 MDSs behind on trimming) 8/5/25 08:22 AM[INF]MDS health message cleared (mds.?): Behind on trimming (525/128) 8/5/25 08:22 AM[WRN]Health check update: 1 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:22 AM[INF]MDS health message cleared (mds.?): Behind on trimming (525/128) 8/5/25 08:20 AM[WRN]overall HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:10 AM[WRN]mds.cephfs.node2.isqjza(mds.0): Behind on trimming (332/128) max_segments: 128, num_segments: 332 8/5/25 08:10 AM[WRN]mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming (332/128) max_segments: 128, num_segments: 332 8/5/25 08:10 AM[WRN][WRN] MDS_TRIM: 2 MDSs behind on trimming 8/5/25 08:10 AM[WRN]Health detail: HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:03 AM[WRN]Health check update: 2 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:03 AM[WRN]Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:00 AM[INF]overall HEALTH_OK #ceph fs status cephfs - 50 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephfs.node1.ojmpnk Reqs: 10 /s 305k 294k 91.8k 6818 0-s standby-replay cephfs.node2.isqjza Evts: 0 /s 551k 243k 90.6k 0 POOL TYPE USED AVAIL cephfs_metadata metadata 2630M 2413G cephfs_data data 12.7T 3620G STANDBY MDS cephfs.node3.vdicdn MDS version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable) # ceph versions { "mon": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "mgr": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 2 }, "osd": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 18 }, "mds": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "rgw": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 6 }, "overall": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 32 } } #ceph orch ps --daemon-type mds NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mds.cephfs.node1.ojmpnk rke-sh1-1 running (18h) 4m ago 19M 1709M - 19.2.2 4892a7ef541b 8dd8db30a1de mds.cephfs.node2.isqjza rke-sh1-2 running (18h) 2m ago 3y 1720M - 19.2.2 4892a7ef541b 7b9d5b692764 mds.cephfs.node3.vdicdn rke-sh1-3 running (18h) 108s ago 18M 27.9M - 19.2.2 4892a7ef541b d2de22a15e18 root@node1:~# ceph config show-with-defaults mds.cephfs.rke-sh1-3.vdicdn | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate" mds_cache_memory_limit 4294967296 default mds_cache_trim_decay_rate 1.000000 default mds_cache_trim_threshold 262144 default mds_recall_max_caps 30000 default mds_recall_max_decay_rate 1.500000 default root@node2:~# ceph config show-with-defaults mds.cephfs.rke-sh1-2.isqjza | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate" mds_cache_memory_limit 4294967296 default mds_cache_trim_decay_rate 1.000000 default mds_cache_trim_threshold 262144 default mds_recall_max_caps 30000 default mds_recall_max_decay_rate 1.500000 default root@node3:~# ceph config show-with-defaults mds.cephfs.rke-sh1-1.ojmpnk | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate" mds_cache_memory_limit 4294967296 default mds_cache_trim_decay_rate 1.000000 default mds_cache_trim_threshold 262144 default mds_recall_max_caps 30000 default mds_recall_max_decay_rate 1.500000 default # ceph mds stat cephfs:1 {0=cephfs.node1.ojmpnk=up:active} 1 up:standby-replay 1 up:standby Do you have an idea on what could happen ? Should I increate mds_cache_trim_decay_rate ? I saw the folloing issue : <https://tracker.ceph.com/issues/66948> Bug #66948: mon.a (mon.0) 326 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log - CephFS - Ceph ( <https://github.com/ceph/ceph/pull/60838> squid: mds: trim mdlog when segments exceed threshold and trim was idle by vshankar · Pull Request #60838 · ceph/ceph · GitHub ) maybe related ? Thanks for the help 😊 Best Regards, Edouard Fazenda. <https://www.csti.ch/> Swiss Cloud Provider Edouard Fazenda Technical Support <https://www.csti.ch/> Chemin du Curé-Desclouds, 2 CH-1226 Thonex +41 22 869 04 40 www.csti.ch

4 6

Re: "BUG: soft lockup" with MDS
by Janek Bevendorff 10 Jul '25

10 Jul '25

Yes, they are older Dell PowerEdges. I have to check whether there's newer firmware, but we've been running Ceph for years without these problems. I checked the logs on the host on which I had a lockup just an hour ago, but there's nothing besides the expected hardreset messages. There are two older watchdog messages, but they are from March: -------------------------------------------------------------------------------- SeqNumber = 2089 Message ID = ASR0000 Category = System AgentID = SEL Severity = Critical Timestamp = 2025-03-27 07:16:03 Message = The watchdog timer expired. RawEventData = 0x03,0x00,0x02,0x33,0xFB,0xE4,0x67,0x20,0x00,0x04,0x23,0x71,0x6F,0xC0,0x04,0xFF FQDD = WatchdogTimer.iDRAC.1 -------------------------------------------------------------------------------- SeqNumber = 2088 Message ID = ASR0000 Category = System AgentID = SEL Severity = Critical Timestamp = 2025-03-27 07:06:41 Message = The watchdog timer expired. RawEventData = 0x02,0x00,0x02,0x01,0xF9,0xE4,0x67,0x20,0x00,0x04,0x23,0x71,0x6F,0xC0,0x04,0xFF FQDD = WatchdogTimer.iDRAC.1 -------------------------------------------------------------------------------- I grepped the logs of another host where it happened, but couldn't find any watchtdog messages there. I believe it's also unlikely that suddenly all MDS hosts (we have five active, five hot standbys, and one cold standby) start having hardware issues. I also ran a memtest on one of the hosts last week and couldn't find anything there either. On 16/04/2025 15:14, Anthony D'Atri wrote: > Curious, are your systems Dells? If so you might see some improvement from running DSU to update all the firmware. It might also be illuminating to run `racadm lclog view` > >> On Apr 16, 2025, at 8:32 AM, Janek Bevendorff <janek.bevendorff(a)uni-weimar.de> wrote: >> >> Hi, >> >> Since the latest Reef update I have the problem that some of my hosts suddenly go into a state where all CPUs are stuck in kernel mode causing all daemons on that host to become unresponsive. When I connect to the IPMI console, I see a lot of messages like: >> >> watchdog: BUG: soft lockup - CPU#8 stuck for 47s! [cron:868840] >> >> (it's basically a list of all processes running on the machine). >> >> Usually, this resolves itself after several minutes, but sometimes I have to hardreset the host. When this happens, all daemons are marked as down and I cannot interact with the host at all. I don't know what causes this but, I think it happens primarily on the hosts where my MDS run and it seems to be triggered by events such as cluster rebalances, MDS restarts, or just randomly. >> >> I found a few reports about similar issues on the bug tracker and mailing list, but they are all very unspecific, unanswered, or more than 6 years old. >> >> Is there any way I can debug this? I upgraded to Squid already, but that didn't solve the problem. I also had massive issues with this during the upgrade. Particularly at the end when the MDS were upgraded, I had constant struggles with it. I had to set the noout flag and then literally sit next to it to resume the upgrade every few minutes until it finally went through, because random MDS hosts went intermittently dark all the time. >> >> All hosts run Ubuntu 22.04 with kernel 6.8.0. >> >> Any ideas? Thanks! >> Janek >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 13

Squid and Ganesha and ingress options?
by Nigel Williams 08 Jul '25

08 Jul '25

We deployed a new Cephadm Squid cluster (400 OSDs) and are happy with everything except this time we tried Ganesha again with the ingress option (HAproxy), but soon we ran into a failed daemon event. And there is a github issue for it: https://github.com/nfs-ganesha/nfs-ganesha/issues/1158 Our questions are: Is a hotfix pending for this github issue? Is anyone using Ganesha with or without ingress, is it reliable for you? How is the relative performance between Ganesha and kernel NFS? We seem to try Ganesha every release but return to the usual kernel NFS server setup because Ganesha is unreliable, we would be interested to know what others in the community think about the Ganesha option. thanks.

2 1

Question about object maps and index rebuilding.
by Gary Molenkamp 03 Jul '25

03 Jul '25

We use ceph rbd as a volume service for both an Openstack deployment and a series of Proxmox servers. This ceph deployment started as a Hammer release and has been upgraded over the years to where it is now running Quincy. It has been fairly solid over that time, even through upgrades from filestore to bluestore, and many transparent hardware replacements/improvements. One concern we have is that when we have a hypervisor that unexpectedly dies/crashes, the volumes must always have the object maps rebuilt. If we don't rebuild the object maps, the VMs will either not boot, or we will have other side-effects that render the volume unusable. (ie cannot mount root). Is this to be expected during this type of event or have I missed a setting during one of the many upgrade on our deployment? Thankfully the above does not happen regularly, but we would like to make use of the HA features of proxmox to ensure some VMs are always available. Requiring the rebuild step limits what can be done automatically, and how quickly it can be recovered. Any advice on ceph configuration, or how others may adapt to the requirements in HA situations would be appreciated. Cheers, Gary -- Gary Molenkamp Science Technology Services Systems Engineer University of Western Ontario molenkam(a)uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566

3 7

Re: ceph squid - huge difference between capacity reported by "ceph -s" and "ceph df "
by Steven Vacaroaia 02 Jul '25

02 Jul '25

Hi Anthony Appreciate you taking the time to provide so much guidance All your messages in this mailing list are well documented and VERY helpful I have attached a text file with the output of the commands you mentioned You are right , there is no /dev/bluefs_/bluefsstore_bdev pointing to the NVME namespaces No specific reason to use separate WAL , just remembered it was recommended back in the "mimic" days ( yes, I am a bit "rusty") I am using 4+2 EC I have 2 separate NVMEs ( 1.6 TB each) dedicated to DB/WAL for HDD I want to use one for the 6 HDD that I currently have , and save the other for when I will be adding more HDDs The servers are SuperMicro SSG-641E with 2 x Intel GOLD 6530 ( 32 cores each) and 1TB RAM My plan was/is to use the 3 x 15TB NVME on each server for high performance pools (like metadata for cephfs or index for RGW) I have carved them in 3 ( 5 TB each) so I can deploy more PGs hence increase performance The SSD are meant to be used for RBD ( proxmox ) The HDD are meant to be used for archiving data using S3 and cephfs. I can redeploy - please clarify 2 things about osd spec file 1. can I use NVME namespaces ( created with nvme command - see below) or should I let ceph partition the NVME disk ? in either case, how do you specify WHICH device/NVME to use for DB/WAL ( as I have 2 identical ones and want to keep the second one for adding more HDD in the future ) nvme create-ns $device --nsze=$per_ns_blocks --ncap=$per_ns_blocks --flbas=0 --dps=0 nvme attach-ns $device --namespace-id=$i --controllers=`nvme id-ctrl $device 2. to deploy 3 different types of OSDs using the spec I have with the advice from you , I am guessing this is the correct spec service_type: osd service_id: hdd_osd crush_device_class: hdd_class placement: host_pattern: * spec: data_devices rotational: 1 db_devices: rotational: 0 size: 1000G:1600G filter_logic: AND objectstore: bluestore service_id: ssd_osd crush_device_class: ssd_class placement: host_pattern: * spec: data_devices rotational: 0 size: 6000G:8000G service_id: nvme_osd crush_device_class: nvme_class placement: host_pattern: * spec: data_devices rotational: 0 size: 4000G:5500G Many thanks Steven On Sun, 29 Jun 2025 at 17:45, Anthony D'Atri <aad(a)dreamsnake.net> wrote: > So you have NVMe SSD OSDs, SATA SSD OSDs, and HDD OSDs with offload onto > NVNe SSDs. > > Did you have a specific reason to explicitly specify wal_devices_. It’s > usually fine to just run with the default WAL size, with the WAL colocated > with the DB. And thus give your DB partitions a bit more space. > > What are your use-cases for these three classes of OSDs? Looks like you > have 42x 20T HDD OSDs, 63x NVMe OSDs, and 84x 7.6T SATA SSD OSDs? > Apparently with the 15T SSDs divided into 3x OSDs each? How much CPU do > you have on these nodes? Any specific reason to have chopped up the NVMe > SSDs into thirds? > > It looks to me as though your .mgr pool is using the default > replicated_rule, which does not specify a device class. This will confound > the balancer and if enabled the pg_autoscaler. > I recommend changing the .mgr pool to use the CRUSH rule that the > non-buckets.data pools use, which should be one that specifies > 3-replication constrained to one of the SSD device classes. As it is the > .mgr pool may be placed on any of the three device classes, which is > trivial with respect to space, but confounds as I mentioned. > > Or you could manually edit the CRUSH map and change the #0 replicated_rule > to specify nvme_class but it sounds like you’re new to Ceph and I don’t > want to frighten you with that process which unfortunately is still > old-school. Changing the rule as I suggested will be much safer. > > The numbers look like you have all of the RGW pools except buckets.data on > the nvme_class SSDs, which is fine, but you won’t begin to use all their > capacity, the index pool will maybe use 5-10% of the capacity used by your > buckets.data pool over time, depending on your distribution of object sizes > and the replication strategy of your buckets.data pool. Doing the math > I’ll speculate that your buckets.data pool is using a … EC 5+2 profile? > True? If so I might suggest rebuilding if/while you still can. There are > distinct advantages to having EC K+M < the number of OSD nodes. > > > > > Hi, > > Yes, I have separate NVME namespaces allocated for WAL and DB to each > spinning disk > > > Namespaces, or partitions? > > > Does that mean I still have to hunt for the 8TB culprit ? > > > Okay so `ceph cf` shows 8.2 TB of raw space used on the hdd_class OSDs, > that’s your concern, right? > > Please share outputs of the following: > > `ceph osd df` (showing a few of each device class) > `ceph osd dump | grep pool` > `ceph osd metadata NNNN | egrep /dev\|bluefs_\|bluestore_bdev` for at > least one OSD of each device class. And run it yourself without specifying > an OSD ID so it captures all, and see if all OSDs in each device class look > the same. > `ceph osd device ls-by-host ceph-host-1 > > It’s entirely possible that your WAL+DB aren’t actually offloaded to SSDs > as you intended. Advanced OSD service specs can be tricky. > > That’s my suspicion, that the WAL+DB are actually still on your HDDs. > Which can be migrated in-situ, or you can nuke the site from orbit and > redeploy. > > A note about your OSD specs. Specifying the models as you’re doing is > totally supported. But think about what happens if you add nodes in the > future that have different drive SKUs, or you RMA a drive and they send you > a different SKU as the replacement. > > It’s usually more future-proof to use a size range in the spec for each > osd service instead of `model`, with a bit of margin to account for base 2 > units vs base 10 units. > > Here’s an example that creates OSDs on SSDs between 490 and 1200 GB, this > is on systems that have ~ 1TB nominal drives. The systems also have 2TB > SATA SSDs that are used for WAL+DB offload, which are above the 1200GB > limit specified so they aren’t matched. > > service_type: osd > service_id: dashboard-admin-1705602677615 > service_name: osd.dashboard-admin-1705602677615 > placement: > host_pattern: * > spec: > data_devices: > rotational: 0 > size: 490G:1200G > filter_logic: AND > objectstore: bluestore > > And here is a spec that matches any HDD larger than 18T and deploys OSDs > on them without offload. This cluster has 20TB HDDs, so the range of 18+ > TB matches both the SEAGATE_ST20000NM007H and SEAGATE_ST20000NM002D drives > present. > > service_type: osd > service_id: cost_capacity > service_name: osd.cost_capacity > placement: > host_pattern: noactuallyusedanymore > spec: > data_devices: > rotational: 1 > size: '18T:' > filter_logic: AND > objectstore: bluestore > > Oh, and make sure that your HDDs and SSDs are all updated to the most > recent firmware. If you have Dell chassis, run DSU on the nodes and update > all firmware, but skip the OS drivers. If you have HP chassis, you can get > firmware update scripts from their web site, but I suspect these aren’t > HP. If anyone else, they’re likely generic drives and you can get firmware > updaters from the mfgs respective web sites. > > Then reboot nodes one at a time to effect the firmware, letting the > cluster completely recover between each reboot. > > > > > If yes , what would be the most efficient way of finding out what takes > the space ? > > Apologies for sending pictures but we are operating in an air gapped > environment > > I used this spec file to create the OSDs > > <image.png> > > Here is the osd tree of one of the servers > all the other 6 are similar > > <image.png> > > Steven > > > On Sun, 29 Jun 2025 at 14:25, Anthony D'Atri <aad(a)dreamsnake.net> wrote: > >> WAL by default rides along with the DB and rarely warrants a separate or >> larger allocation. >> >> Since you say you’ve allocated DB space, does that mean that you have >> WAL+DB offloaded onto SSDs? If so they don’t contribute to the space used >> on the hdd device class. >> >> >> > On Jun 29, 2025, at 1:56 PM, Steven Vacaroaia <stef97(a)gmail.com> wrote: >> > >> > Hi Janne >> > >> > Thanks >> > That make sense since I have allocated 196GB for DB and 5 GB for WALL >> for >> > all 42 spinning OSDs >> > Again, thanks >> > Steveb >> > >> > On Sun, 29 Jun 2025 at 12:02, Janne Johansson <icepic.dz(a)gmail.com> >> wrote: >> > >> >> Den sön 29 juni 2025 kl 17:22 skrev Steven Vacaroaia <stef97(a)gmail.com >> >: >> >> >> >>> Hi, >> >>> >> >>> I just built a new CEPH squid cluster with 7 nodes >> >>> Since this is brand new, there is no actuall data on it except few >> test >> >>> files in the S3 data.bucket >> >>> >> >>> Why is "ceph -s" reporting 8 TB of used capacity ? >> >>> >> >> >> >> Because each OSD will have GBs of preallocated data for the RocksDB, >> >> write-ahead-logs and other structures, and this counts against "raw >> >> available space", even if you don't have objects of this size put into >> the >> >> pools, the creation of the DBs and other things happened at osd >> creation, >> >> or when the first object was made, and are there even if you delete the >> >> object later. >> >> >> >> -- >> >> May the most significant bit of your life be positive. >> >> >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >

3 4

CephFS with Ldap
by gagan tiwari 01 Jul '25

01 Jul '25

Hi Guys, We have a Ldap server with all users login details. We have to mount data stored in Ceph to several client nodes via CephFS so that users can access that data and start using that data in their processes. But we need to grant permission / ownership to users to enable them to access that data. like chown user:group /dirs ( on Linux ) How will cephfs recognize users , groups that are in Ldap ? Will I need to set-up Ldap authentication on all nodes in Ceph cluster for this purpose ( ceph mgr , ceph mons , ceph mds and all ods nodes ) Please advise. Thanks, Gagan

5 4

orchestrator not behaving strangely
by Holger Naundorf 01 Jul '25

01 Jul '25

Hello, we are running a ceph cluster at version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable) and since a few weeks the orchestrator started to misbehave - up to now we could not identify any root cause, so I am fishing in the community to see if there are any hints. Problems: An OSD removal (for disk replacement) gets stuck in the 'purge' step: ceph orch osd rm 406 --replace root@aadm01:~# ceph orch osd rm status OSD HOST STATE PGS REPLACE FORCE ZAP DRAIN STARTED AT 406 acn07 done, waiting for purge 0 True False True 2025-06-25 09:18:07.650734+00:00 (now for more than 24h in this state) At the same time the orchestrator is not restarting OSD daemons - i.e. an 'ceph orch daemon restart osd.xxx' claims its queuing uo the restart, but it never happens. Other services continue to be controlled correctly via 'ceph orch ...' If anyone has an idea where to poke around or can match this to some known problem - I would appreciate any pointers. Regards, Holger -- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naundorf(a)rz.uni-kiel.de

3 8

RADOS Gateway IAM for Production Environments ( reef to squid upgrade )
by ramodav＠yahoo.com 01 Jul '25

01 Jul '25

Hi Everyone ! I am planning to upgrade ceph from reef to squid to benefit from RGW IAM .Does anybody know is it still a tech-preview ?Would an update be a nice solution to have this feature ?

3 3

Failure to build on a devuan.
by robin hammond 01 Jul '25

01 Jul '25

I just attempted to build the lasted tarbar on a freshly installed and updated devuan daedalus system. Linux devuan-ceph 6.1.0-37-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.140-1 (2025-05-22) x86_64 GNU/Linux I ran ./install-deps.sh as per https://docs.ceph.com/en/latest/install/build-ceph/ but ./do_cmake.sh resulted in an error. Has anyone any experience with this error ? root@devuan-ceph:/usr/local/src/ceph-19.2.2# cat /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeError.log Determining if the function memset_s exists failed with the following output: Change Dir: /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-7e7NF3 Run Build Command(s):/usr/bin/ninja cmTC_3bf24 && [1/2] Building C object CMakeFiles/cmTC_3bf24.dir/CheckFunctionExists.c.o [2/2] Linking C executable cmTC_3bf24 FAILED: cmTC_3bf24 : && /usr/bin/gcc-12 -DCHECK_FUNCTION_EXISTS=memset_s CMakeFiles/cmTC_3bf24.dir/CheckFunctionExists.c.o -o cmTC_3bf24 && : /usr/bin/ld: CMakeFiles/cmTC_3bf24.dir/CheckFunctionExists.c.o: in function `main': CheckFunctionExists.c:(.text+0x10): undefined reference to `memset_s' collect2: error: ld returned 1 exit status ninja: build stopped: subcommand failed. Determining if the function pthread_set_name_np exists failed with the following output: Change Dir: /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-swTR94 Run Build Command(s):/usr/bin/ninja cmTC_2fec2 && [1/2] Building C object CMakeFiles/cmTC_2fec2.dir/CheckFunctionExists.c.o [2/2] Linking C executable cmTC_2fec2 FAILED: cmTC_2fec2 : && /usr/bin/gcc-12 -DCHECK_FUNCTION_EXISTS=pthread_set_name_np CMakeFiles/cmTC_2fec2.dir/CheckFunctionExists.c.o -o cmTC_2fec2 -lpthread && : /usr/bin/ld: CMakeFiles/cmTC_2fec2.dir/CheckFunctionExists.c.o: in function `main': CheckFunctionExists.c:(.text+0x10): undefined reference to `pthread_set_name_np' collect2: error: ld returned 1 exit status ninja: build stopped: subcommand failed. Determining if the function pthread_get_name_np exists failed with the following output: Change Dir: /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-6fkgqm Run Build Command(s):/usr/bin/ninja cmTC_13767 && [1/2] Building C object CMakeFiles/cmTC_13767.dir/CheckFunctionExists.c.o [2/2] Linking C executable cmTC_13767 FAILED: cmTC_13767 : && /usr/bin/gcc-12 -DCHECK_FUNCTION_EXISTS=pthread_get_name_np CMakeFiles/cmTC_13767.dir/CheckFunctionExists.c.o -o cmTC_13767 -lpthread && : /usr/bin/ld: CMakeFiles/cmTC_13767.dir/CheckFunctionExists.c.o: in function `main': CheckFunctionExists.c:(.text+0x10): undefined reference to `pthread_get_name_np' collect2: error: ld returned 1 exit status ninja: build stopped: subcommand failed. Determining if the function getprogname exists failed with the following output: Change Dir: /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-CZhew8 Run Build Command(s):/usr/bin/ninja cmTC_6c988 && [1/2] Building C object CMakeFiles/cmTC_6c988.dir/CheckFunctionExists.c.o [2/2] Linking C executable cmTC_6c988 FAILED: cmTC_6c988 : && /usr/bin/gcc-12 -DCHECK_FUNCTION_EXISTS=getprogname CMakeFiles/cmTC_6c988.dir/CheckFunctionExists.c.o -o cmTC_6c988 && : /usr/bin/ld: CMakeFiles/cmTC_6c988.dir/CheckFunctionExists.c.o: in function `main': CheckFunctionExists.c:(.text+0x10): undefined reference to `getprogname' collect2: error: ld returned 1 exit status ninja: build stopped: subcommand failed. Determining if files valgrind/helgrind.h exist failed with the following output: Change Dir: /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-gqBwr4 Run Build Command(s):/usr/bin/ninja cmTC_c5529 && [1/2] Building C object CMakeFiles/cmTC_c5529.dir/HAVE_VALGRIND_HELGRIND_H.c.o FAILED: CMakeFiles/cmTC_c5529.dir/HAVE_VALGRIND_HELGRIND_H.c.o /usr/bin/gcc-12 -o CMakeFiles/cmTC_c5529.dir/HAVE_VALGRIND_HELGRIND_H.c.o -c /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-gqBwr4/HAVE_VALGRIc /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-gqBwr4/HAVE_VALGRIND_HELGRIND_H.c:2:10: fatal error: valgrind/helgrind.h: No such file or directory 2 | #include <valgrind/helgrind.h> | ^~~~~~~~~~~~~~~~~~~~~ compilation terminated. ninja: build stopped: subcommand failed. Source: /* */ #include <valgrind/helgrind.h> int main(void){return 0;} Performing C SOURCE FILE Test HAVE_STAT_ST_MTIMESPEC_TV_NSEC failed with the following output: Change Dir: /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-sKMpCv Run Build Command(s):/usr/bin/ninja cmTC_b96bc && [1/2] Building C object CMakeFiles/cmTC_b96bc.dir/src.c.o FAILED: CMakeFiles/cmTC_b96bc.dir/src.c.o /usr/bin/gcc-12 -DHAVE_STAT_ST_MTIMESPEC_TV_NSEC -o CMakeFiles/cmTC_b96bc.dir/src.c.o -c /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-sKMpCv/c /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-sKMpCv/src.c: In function ‘main’: /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-sKMpCv/src.c:6:34: error: ‘struct stat’ has no member named ‘st_mtimespec’ 6 | (void)sizeof(((struct stat *)0)->st_mtimespec.tv_nsec); | ^~ ninja: build stopped: subcommand failed. Source file was: #include <sys/stat.h> int main() { (void)sizeof(((struct stat *)0)->st_mtimespec.tv_nsec); return 0; } Performing C++ SOURCE FILE Test HAVE_CXX11_ATOMIC failed with the following output: Change Dir: /usr/local/src/ceph-19.2.2/build/CMakeFiles/CMakeScratch/TryCompile-Qzk4En Run Build Command(s):/usr/bin/ninja cmTC_ba8d6 && [1/2] Building CXX object CMakeFiles/cmTC_ba8d6.dir/src.cxx.o [2/2] Linking CXX executable cmTC_ba8d6 FAILED: cmTC_ba8d6 : && /usr/bin/g++-12 -std=c++11 CMakeFiles/cmTC_ba8d6.dir/src.cxx.o -o cmTC_ba8d6 && : /usr/bin/ld: CMakeFiles/cmTC_ba8d6.dir/src.cxx.o: in function `std::atomic<tagged_ptr>::store(tagged_ptr, std::memory_order)': src.cxx:(.text._ZNSt6atomicI10tagged_ptrE5storeES0_St12memory_order[_ZNSt6atomicI10tagged_ptrE5storeES0_St12memory_order]+0x62): undefined reference to `__atomic_stor' /usr/bin/ld: CMakeFiles/cmTC_ba8d6.dir/src.cxx.o: in function `std::atomic<tagged_ptr>::load(std::memory_order) const': src.cxx:(.text._ZNKSt6atomicI10tagged_ptrE4loadESt12memory_order[_ZNKSt6atomicI10tagged_ptrE4loadESt12memory_order]+0x2d): undefined reference to `__atomic_load_16' /usr/bin/ld: CMakeFiles/cmTC_ba8d6.dir/src.cxx.o: in function `std::atomic<tagged_ptr>::compare_exchange_strong(tagged_ptr&, tagged_ptr, std::memory_order, std::memor: src.cxx:(.text._ZNSt6atomicI10tagged_ptrE23compare_exchange_strongERS0_S0_St12memory_orderS3_[_ZNSt6atomicI10tagged_ptrE23compare_exchange_strongERS0_S0_St12memory_or' collect2: error: ld returned 1 exit status ninja: build stopped: subcommand failed. Source file was: #include <atomic> #include <cstdint> #include <cstddef> #if defined(__SIZEOF_INT128__) // Boost needs 16-byte atomics for tagged pointers. // These are implemented via inline instructions on the platform // if 16-byte alignment can be proven, and are delegated to libatomic // library routines otherwise. Whether or not alignment is provably // OK for a std::atomic unfortunately depends on compiler version and // optimization levels, and also on the details of the expression. // We specifically test access via an otherwise unknown pointer here // to ensure we get the most complex case. If this access can be // done without libatomic, then all accesses can be done. struct tagged_ptr { int* ptr; std::size_t tag; }; void atomic16(std::atomic<tagged_ptr> *ptr) __attribute__ ((used)); void atomic16(std::atomic<tagged_ptr> *ptr) { tagged_ptr p{nullptr, 1}; ptr->store(p); tagged_ptr f = ptr->load(); tagged_ptr new_tag{nullptr, 0}; ptr->compare_exchange_strong(f, new_tag); } #endif int main() { #if defined(__SIZEOF_INT128__) std::atomic<tagged_ptr> ptr; atomic16(&ptr); #endif std::atomic<uint8_t> w1; std::atomic<uint16_t> w2; std::atomic<uint32_t> w4; std::atomic<uint64_t> w8; return w1 + w2 + w4 + w8; }

2 1

Re: Suspiciously low PG count for CephFS with many small files
by Niklas Hambüchen 30 Jun '25

30 Jun '25

(Adding back the list) > https://docs.clyso.com/docs/kb/rados/#osd---improved-procedure-for-adding-h… <https://docs.clyso.com/docs/kb/rados/#osd---improved-procedure-for-adding-h…> Is the upstream source for `upmap-remapped.py` the `ceph-scripts` repo? If yes, is it known why there are 2 different places for `ceph-scripts` (Gitlab and Github), with diverged histories, both of which are recently edited, by the same individuals? https://github.com/cernceph/ceph-scripts/commits/master/tools/upmap/upmap-r… https://gitlab.cern.ch/ceph/ceph-scripts/-/commits/master/tools/upmap/upmap… > If I understand what you’re getting at, I don’t think that’s the case. I *think* recent releases give priority to undersized / degraded PGs over remapped PGs. It's not really `degraded` vs `misplaced` I'm getting at. But instead, consider this scneario: We have * osd.1 (very full, 100 GB free) * osd.2 (very full, 100 GB free) * osd.30 (empty) and we have * PGa on osd.1, which Ceph now wants to be on osd.2 * PGb on osd.2, which Ceph now wants to be on osd.30 The simple thing that could theoretically be done is that simultaneously be done is that the PGa (osd.1 -> osd.2) move, and the PGb (osd2. -> osd.30) move, are started immediately and run in parallel. But I suspect that what Ceph does is to make decisions on PG granularity: 1. See that there's no space on osd.2 for PGa, thus not start the move. 2. Move PGb from osd.2 to osd.30. Wait until that's done (takes long if PGs are large). 3. See that now there enough space on osd.2 for PGa, thus only now start the move. Is this really what happens? Thanks!

2 1

Latest stable version of Ceph
by gagan tiwari 30 Jun '25

30 Jun '25

Hi Guys, I am going to set-up a new Ceph Cluster on nodes with OS RHEL8/9. Please let me know which is the most stable version of ceph with all bug fixes that I should go for. Thanks, Gagan

3 4

Suspiciously low PG count for CephFS with many small files
by Niklas Hambüchen 30 Jun '25

30 Jun '25

I have 2 clusters; both have HDDs and SSDs. Reporting only the HDDs which have their own pools: "rep-cluster": hdd-pool 3-replication, 86 OSDs (16 TiB each), 1024 PGs, 78 %RAW USED, 100 M objects "ec-cluster": hdd-pool erasure k=4 m=2, 58 OSDs (16 TiB each), 256 PGs, 60 %RAW USED, 450 M objects Both are Ceph 18.2.1, Bluestore, and have the autoscaler enabled. As you can see, I have many small objects. My PGs-copies-per-OSD seem far off from the recommendation of 100 PGs per OSD (`mon_target_pg_per_osd`): rep-cluster: 35 PGs/OSD (= 1024*3/86) ec-cluster: 26 PGs/OSD (= 256*6/58) So I'm at least 3x-4x off. Why? Should the autoscaler not have increased the PGs here? `ceph osd pool autoscale-status`: rep-cluster: POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK data 349.4T 3.0 1343T 0.7802 1.0 1024 on False ec-cluster: data_ec 347.3T 1.5 876.4T 0.5944 1.0 256 on False I believe that because of this I suffer some drawbacks: * On ec-cluster, a PG contains ~2 TiB and ~2 M objects, causing rebalances to happen in coarse, slow steps. Should I take some steps to force the autoscaler to increase PGs, and if yes, which approach would be best here? Thanks for your tips! Niklas

3 9

Code of Conduct
by Anthony Middleton 30 Jun '25

30 Jun '25

Hello Ceph Community, I'm reaching out regarding a recent dispute on our public mailing lists. In line with our commitment to uphold the Ceph Code of Conduct ( https://ceph.io/en/code-of-conduct/) I've started a resolution process in close collaboration with the Ceph Steering Committee and Ceph Executive Council. As part of this process, I have contacted those involved to understand their perspectives. The review is ongoing, and no final decisions have been made. I want to assure you that I am handling this matter with impartiality and care, guided by the principles and expectations outlined in our Code of Conduct. The Ceph Executive Council and I value every community member deeply. Our goal is not only to maintain a safe and respectful environment but also to foster a collaborative culture that inspires all of us to contribute toward our shared mission: advancing open-source storage for everyone's benefit. Thank you for your ongoing engagement and for being part of what makes the Ceph community strong. If you have thoughts or concerns you'd like to share, please feel free to contact me directly at community-manager(a)ceph.io. Best, Anthony Middleton Ceph Community Manager

1 0

Re-uploading existing s3 objects slower than initial upload
by Sinan Polat 30 Jun '25

30 Jun '25

Hi all, I am investigating a performance issue related to uploading s3 objects and would appreciate any insights. I have a bucket with ~200 million objects. I noticed that re-uploading files that already exist in the bucket appears to be slower than uploading them for the first time. To verify this behavior, I created a small test case with a new empty bucket: - 500 files with random sizes between 5 KB and 50 KB. - First time upload took 8 seconds: s3cmd put files/* s3://my-bucket/ - Second upload (same files, no changes) took 10 seconds I had expected the second upload to be faster or at least the same. Is this expected behavior? Any guidance on this would be appreciated. Thanks! Sinan

2 1

Get past epoch pg map
by Darrell Enns 30 Jun '25

30 Jun '25

Is there a way to get a pg dump for a previous epoch? I have a situation where I want to make sure all of the PGs that used to be on a specific OSD are deep scrubbed. However, since that OSD is now down and all the PGs have been relocated, I don't know what the PG numbers are. I know the epoch number I want to look at, and I can "ceph osd getmap <epoch>". But I'm not sure if there's a way to use this to get/view the pg map.

3 3

ceph squid - huge difference between capacity reported by "ceph -s" and "ceph df "
by Steven Vacaroaia 29 Jun '25

29 Jun '25

Hi, I just built a new CEPH squid cluster with 7 nodes Since this is brand new, there is no actuall data on it except few test files in the S3 data.bucket Why is "ceph -s" reporting 8 TB of used capacity ? Many thanks Steven [image: image.png]

2 2

Reef: RGW logrotate "breaks" log_to_file
by Eugen Block 27 Jun '25

27 Jun '25

Hi, after upgrading multiple clusters from 18.2.4. some weeks ago, I noticed that the RGWs stop logging to file after the nightly logrotate. Other daemons don't seem to be affected, they continue logging to file. Restarting an RGW daemon helps until the next logrotate. I could reproduce this in a lab cluster, after upgrading from 18.2.4 to 18.2.5, the daemon stops logging to file after a logrotate. Is this a known issue? I can't seem to find anything on tracker. Thanks! Eugen ceph config dump | grep log_to_file global basic log_to_file true global advanced mon_cluster_log_to_file true

5 10

cephfs + SELinux: .snap folder is unlabeled
by Fabian Wenzel 27 Jun '25

27 Jun '25

Hi everyone, when setting up ceph fs (using reef) on my system, I have noticed that there is an issue with an SELinux environment for an smb-gateway: SELinux requires samba shares to be of type samba_share_t. While this can be achieved with e.g. chcon on the mounted ceph fs (since it supports xattr), it is not possible to assign an SELinux type to the .snap subfolder, since it is read-only. Therefore, Is -lZ .snap returns a question mark, and samba access results in an access denied error to the type "unlabeled_t". While it is possible (as a workaround) to implement a rule that allows samba to access folders of "unlabeled_t", I think a proper solution would be to assign a label to the .snap folder as well. For samba, accessing the .snap folder is essential in order to enable "previous versions" functionality of the Windows Explorer with the samba shadow_copy2 vfs, as it can be found in a couple of youtube tutorials covering ceph fs + samba. How could this be done? Thanks, sophonet

2 1

First time configuration advice
by Ryan Sleeth 26 Jun '25

26 Jun '25

I am setting up my first cluster of 9-nodes each with 8x 20T HDDs and 2x 2T NVMes. I plan to partition the NVMes into 5x 300G so that one partition can be used by cephfs_metadata (SSD only), while the other 4x partitions will be paired as db devices for 4x of the HDDs. The cluster will only be used for cephfs and data will only be stored on its EC 4+2 HDD-only pool. Just a simple and large file server, so performance isn't a primary concern. Each node has 2x 10Gb network connections (one public, one cluster). All disks encrypted (encrypted=true on OSD creation on top of LVM). I am new to Ceph, and was hoping to get advice on if these initial configurations are sensible or if I am doing something incorrect that might have a negative impact in the future: https://pastebin.com/yiQS4g3q Thanks!

7 6

Incomplete PG's
by Devender Singh 26 Jun '25

26 Jun '25

Hello Seeking some help to recover these incomplete pgs… I made osd, down then tried but nothing seems working …. Ceph version 18.2.7, we have multiple disk failed in ceph cluster on different nodes… for i in 19.5a 19.5d 19.5e 19.6b 19.6f 19.74 19.7b 34.48 34.69; do ceph pg map $i | grep -q "acting.*1" && \ ceph-objectstore-tool --data-path /var/lib/ceph/15688cb4-044a-11ec-942e-516035adea04/osd.17 --op mark-complete --pgid $i --force done # for i in 19.5a 19.5d 19.5e 19.6b 19.6f 19.74 19.7b 34.48 34.69; do ceph pg map $i | grep -q "acting.*1" && \ ceph-objectstore-tool --data-path /var/lib/ceph/15688cb4-044a-11ec-942e-516035adea04/osd.17 --op repair --pgid $i --force done & pg 16.48 is incomplete, acting [20,9,17] pg 16.50 is incomplete, acting [20,27,5] pg 16.68 is incomplete, acting [2,5,27] pg 16.78 is incomplete, acting [23,14,29] pg 17.4f is incomplete, acting [10,30,18] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.7a is incomplete, acting [8,28,27] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 19.44 is incomplete, acting [10,4,21,29,0] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.45 is stuck inactive since forever, current state incomplete, last acting [10,28,24,4,0] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.49 is incomplete, acting [28,10,26,0,15] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.4a is incomplete, acting [21,1,23,0,26] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.4b is incomplete, acting [6,12,17,30,7] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.4c is incomplete, acting [5,0,8,21,26] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.4f is incomplete, acting [26,14,5,7,9] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.51 is incomplete, acting [20,14,28,26,23] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.52 is incomplete, acting [10,26,22,30,7] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.53 is incomplete, acting [12,24,25,23,22] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.54 is incomplete, acting [10,4,20,15,11] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.57 is incomplete, acting [15,22,9,0,14] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.58 is incomplete, acting [15,0,12,28,20] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.59 is incomplete, acting [6,9,0,21,26] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.5a is incomplete, acting [11,31,20,17,24] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.5d is incomplete, acting [30,19,4,17,15] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.5e is incomplete, acting [3,9,10,17,16] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.5f is remapped+incomplete, acting [22] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.60 is incomplete, acting [25,27,23,4,22] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.61 is remapped+incomplete, acting [23] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.62 is remapped+incomplete, acting [23] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.63 is incomplete, acting [15,4,1,22,7] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.64 is incomplete, acting [4,7,5,29,31] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.65 is incomplete, acting [3,9,15,10,19] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.6b is remapped+incomplete, acting [1] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.6f is incomplete, acting [30,16,17,18,10] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.72 is remapped+incomplete, acting [19] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.73 is incomplete, acting [12,27,13,16,11] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.74 is incomplete, acting [17,18,19,4,29] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.75 is incomplete, acting [8,31,20,15,7] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.76 is incomplete, acting [5,26,18,30,21] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.78 is incomplete, acting [13,26,14,10,16] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.7a is incomplete, acting [15,1,16,12,6] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.7b is incomplete, acting [2,17,15,29,9] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.7c is incomplete, acting [28,19,6,4,26] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.7f is remapped+incomplete, acting [23] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 22.4b is incomplete, acting [14,7,23] pg 22.51 is incomplete, acting [11,8,2] pg 22.5a is incomplete, acting [31,15,4] pg 34.48 is incomplete, acting [31,19,17] pg 34.51 is incomplete, acting [15,12,20] pg 34.65 is incomplete, acting [30,18,20] pg 34.6d is incomplete, acting [30,12,21] pg 34.7c is incomplete, acting [31,9,3] # ceph -s cluster: id: 15688cb4-044a-11ec-942e-516035adea04 health: HEALTH_ERR 4 OSD(s) experiencing slow operations in BlueStore 1 failed cephadm daemon(s) 1 filesystem is degraded 1 MDSs report slow metadata IOs 1/16510107 objects unfound (0.000%) Reduced data availability: 449 pgs inactive, 444 pgs incomplete Possible data damage: 1 pg recovery_unfound Degraded data redundancy: 3/60411723 objects degraded (0.000%), 1 pg degraded 343 pgs not deep-scrubbed in time 63 slow ops, oldest one blocked for 262107 sec, daemons [osd.1,osd.15,osd.16,osd.18,osd.19,osd.2,osd.21,osd.25,osd.26,osd.27]... have slow ops. services: mon: 7 daemons, quorum test-host04n,test-host05n,test-host02n,test-host03n,test-host06n,test-host07n,test-host08n (age 5h) mgr: test-host04n.azmqik(active, since 5h), standbys: test-host05n.lybhho mds: 3/3 daemons up, 11 standby osd: 32 osds: 32 up (since 18m), 32 in (since 18m); 36 remapped pgs rbd-mirror: 2 daemons active (2 hosts) data: volumes: 1/2 healthy, 1 recovering pools: 32 pools, 1505 pgs objects: 16.51M objects, 55 TiB usage: 155 TiB used, 246 TiB / 401 TiB avail pgs: 0.332% pgs unknown 29.502% pgs not active 3/60411723 objects degraded (0.000%) 1250071/60411723 objects misplaced (2.069%) 1/16510107 objects unfound (0.000%) 1022 active+clean 438 incomplete 16 active+remapped+backfilling 14 active+remapped+backfill_wait 6 remapped+incomplete 5 unknown 3 active+clean+scrubbing+deep 1 active+recovery_unfound+degraded io: recovery: 1.3 GiB/s, 345 objects/s progress: Global Recovery Event (4h) [===================.........] (remaining: 2h) Regards Dev

2 1

ceph health mute behavior
by Eugen Block 25 Jun '25

25 Jun '25

Hi, I'm trying to understand the "ceph health mute" behavior. In this case, I'm referring to the warning PG_NOT_DEEP_SCRUBBED. If you mute it for a week and the cluster continues deep-scrubbing, the "mute" will clear at some point although there are still PGs not deep-scrubbed in time warnings. I could verify this in a tiny lab with 19.2.2, setting osd_deep_scrub_interval to 10 minutes, the warning pops up. Then I mute that warning, issue deep-scrubs for several pools, and at some point I see this in the mon log: Jun 25 08:53:28 host1 ceph-mon[823315]: log_channel(cluster) log [WRN] : Health check update: 61 pgs not deep-scrubbed in time (PG_NOT_DEEP_SCRUBBED) Jun 25 08:53:28 host1 ceph-mon[823315]: Health check update: 61 pgs not deep-scrubbed in time (PG_NOT_DEEP_SCRUBBED) Jun 25 08:53:29 host1 ceph-mon[823315]: pgmap v164176: 389 pgs: 389 active+clean; 428 MiB data, 57 GiB used, 279 GiB / 336 GiB avail ... Jun 25 08:53:31 host1 ceph-mon[823315]: log_channel(cluster) log [INF] : Health alert mute PG_NOT_DEEP_SCRUBBED cleared (count increased from 60 to 61) Jun 25 08:53:31 host1 ceph-mon[823315]: Health alert mute PG_NOT_DEEP_SCRUBBED cleared (count increased from 60 to 61) I don't really understand what the code does [0] (I'm not a dev): ---snip--- if (!p->second.sticky) { auto q = all.checks.find(p->first); if (q == all.checks.end()) { mon.clog->info() << "Health alert mute " << p->first << " cleared (health alert cleared)"; p = pending_mutes.erase(p); changed = true; continue; } if (p->second.count) { // count-based mute if (q->second.count > p->second.count) { mon.clog->info() << "Health alert mute " << p->first << " cleared (count increased from " << p->second.count << " to " << q->second.count << ")"; p = pending_mutes.erase(p); changed = true; continue; ---snip--- Could anyone shed some light what I'm not understanding? Why would the mute clear although there are still PGs not deep-scrubbed? Thanks! Eugen [0] https://github.com/ceph/ceph/blob/d78ffd1247d6cef5cbd829e77204185dc0d3a8ba/…

4 5

RadosGW buckets stuck in lifecycle "PROCESSING"
by Niklaus Hofer 25 Jun '25

25 Jun '25

Dear all I managed to "CTRL+C" a `radosgw-admin lc process`. And now I have a bucket stuck in PROCESSING. | radosgw-admin lc list | jq '.[] | select(.status == "PROCESSING")' > { > "bucket": ":XXXXXXX:ost-dp1-tier3.7955243979.1209", > "started": "Sun, 22 Jun 2025 16:13:42 GMT", > "status": "PROCESSING" > } It's been like that for > 36 hours now. Is it possible that the "CTRL+C"-ed process has left behind some kind of lock? Can I check that? Maybe even remove the lock? That same question has been asked before on this ML [1] but in that thread, the bucket reverted back to UNINITIAL. I was hoping that maybe it would be the same here, but after waiting for a good while, I've lost hope on that... Many thanks in advance Niklaus Hofer Links: [1] https://www.mail-archive.com/ceph-users@lists.ceph.com/msg36977.html -- stepping stone AG Wasserwerkgasse 7 CH-3011 Bern Telefon: +41 31 332 53 63 www.stepping-stone.ch niklaus.hofer(a)stepping-stone.ch

1 1

Re: First time configuration advice
by Robert Sander 25 Jun '25

25 Jun '25

Hi, Am 6/24/25 um 19:21 schrieb Ryan Sleeth: > Do you think there would be any performance benefits to using namespaces > over LVM partitions? I do not think there is a big difference. > Currently I am setting up the nodes with this: > > sudo ceph orch daemon add osd ceph01:/dev/mapper/ceph--nvme--01-block--01,encrypted=true > sudo ceph orch daemon add osd ceph01:data_devices=/dev/mapper/ceph--hdd--01-block--02,db_devices=/dev/mapper/ceph--nvme--01-db--02,encrypted=true > sudo ceph orch daemon add osd ceph01:data_devices=/dev/mapper/ceph--hdd--02-block--03,db_devices=/dev/mapper/ceph--nvme--01-db--03,encrypted=true This is very manual and will not benefit from the automation possibilities of the orchestrator. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Question: What's special about subvolumes?
by James Tocknell 24 Jun '25

24 Jun '25

Hi All Possibly a dumb question, but given with cephfs you can apply quotas to arbitrary directories, snapshot arbitrary directories, assign arbitrary directories to pools etc. what is special/different about subvolumes, and why would I want to use them (and more importantly, not want to use them)? I guess the only thing I can see could be around where the extended attributes are assigned (with the two-level directory structure enabling setting a quota while not allowing read access to the directory above, based on looking at the cephfs CSI?), but I feel I'm missing something obvious. The specific usecase I'm looking at is evaluating using just quotas vs subvolumes for team shares (to ensure that each team only uses the amount of storage they require), but in general I'm trying to understand subvolumes. Regards James

4 3

Force remove undeletable image on RBD pool?
by Gaël THEROND 24 Jun '25

24 Jun '25

Hi folks, I've a quick question. On one of our pool we found out an image that doesn't exist anymore physically (This image doesn't exist, have no snap attached, is not parent of another image) but is still listed when performing a `rbd -p pool ls`. However, it error with a nice "Error opening image <image_name>: (2) No such file or directory" when we try to delete it using `rbd -p pool rm <image_name>`. This pool is a EC based pool and neither the metadata nor the data pool have any data of that image remaining. So my question is, is there a cli way to force ceph to forget/abandon this image? A command which isn't involving manually manipulating the various maps would be preferred but if I had to dig that deep I can. Thanks!

7 12

CEPH upgrade from 18.2.7 to 19.2.2 -- Hung from last 24h at 66%
by Devender Singh 24 Jun '25

24 Jun '25

Hello all I have a cluster where my cluster is in hung state, Some back fills are there but I reduced it to 1 but still upgrade not progressing… Please help… ```# ceph health detail HEALTH_WARN 8 OSD(s) experiencing slow operations in BlueStore; Failed to apply 2 service(s): osd.all-available-devices,osd.iops_optimized; 1 failed cephadm daemon(s); failed to probe daemons or devices; noscrub,nodeep-scrub flag(s) set; Degraded data redundancy: 1600150/39365198 objects degraded (4.065%), 93 pgs degraded, 103 pgs undersized; 127 pgs not deep-scrubbed in time [WRN] BLUESTORE_SLOW_OP_ALERT: 8 OSD(s) experiencing slow operations in BlueStore osd.5 observed slow operation indications in BlueStore osd.9 observed slow operation indications in BlueStore osd.18 observed slow operation indications in BlueStore osd.36 observed slow operation indications in BlueStore osd.59 observed slow operation indications in BlueStore osd.66 observed slow operation indications in BlueStore osd.106 observed slow operation indications in BlueStore osd.110 observed slow operation indications in BlueStore [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 2 service(s): osd.all-available-devices,osd.iops_optimized osd.all-available-devices: Command timed out on host cephadm deploy (osd daemon) (default 900 second timeout) osd.iops_optimized: Command timed out on host cephadm deploy (osd daemon) (default 900 second timeout) [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) daemon osd.19 on phl-prod-host04n.example.comis in error state [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices Command "cephadm ceph-volume -- inventory" timed out on host phl-prod-converged03n.phl.netskrt.org (default 900 second timeout) [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set [WRN] PG_DEGRADED: Degraded data redundancy: 1600150/39365198 objects degraded (4.065%), 93 pgs degraded, 103 pgs undersized pg 25.0 is stuck undersized for 21h, current state active+undersized+degraded+remapped+backfill_wait, last acting [90,77,NONE,38,3] pg 25.1 is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfill_wait, last acting [131,1,66,NONE,72] pg 25.c is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfill_wait, last acting [1,135,110,28,NONE] pg 25.e is active+undersized+degraded+remapped+backfill_wait, acting [18,20,108,101,NONE] pg 25.14 is stuck undersized for 19h, current state active+undersized+degraded+remapped+backfill_wait, last acting [32,13,NONE,65,97] pg 25.17 is stuck undersized for 19h, current state active+undersized+degraded+remapped+backfill_wait, last acting [32,65,110,23,NONE] pg 25.1a is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfill_wait, last acting [97,66,NONE,112,139] pg 25.1c is stuck undersized for 2h, current state active+undersized+degraded+remapped+backfill_wait, last acting [102,136,97,NONE,45] pg 25.1f is stuck undersized for 8h, current state active+undersized+degraded+remapped+backfill_wait, last acting [NONE,66,7,39,109] pg 26.45 is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfill_wait, last acting [32,47] pg 26.48 is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfill_wait, last acting [113,55] pg 26.4b is stuck undersized for 8h, current state active+undersized+degraded+remapped+backfill_wait, last acting [105,7] pg 26.59 is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfill_wait, last acting [35,40] pg 26.6a is stuck undersized for 7h, current state active+undersized+degraded+remapped+backfill_wait, last acting [25,45] pg 26.74 is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfill_wait, last acting [112,131] pg 26.77 is stuck undersized for 2h, current state active+undersized+degraded+remapped+backfill_wait, last acting [17,105] pg 26.9c is stuck undersized for 19h, current state active+undersized+degraded+remapped+backfill_wait, last acting [47,76] pg 26.bf is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [32,26] pg 26.c2 is stuck undersized for 2h, current state active+undersized+degraded+remapped+backfill_wait, last acting [97,135] pg 26.ec is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [26,80] pg 26.f7 is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [49,105] pg 31.12 is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [47,110,131,37,NONE] pg 31.19 is stuck undersized for 2h, current state active+undersized+degraded+remapped+backfill_wait, last acting [NONE,87,3,137,39] pg 31.1b is stuck undersized for 2h, current state active+undersized+degraded+remapped+backfill_wait, last acting [108,NONE,42,102,97] pg 31.1c is stuck undersized for 19h, current state active+undersized+degraded+remapped+backfill_wait, last acting [113,66,NONE,45,72] pg 31.41 is stuck undersized for 2h, current state active+undersized+remapped+backfill_wait, last acting [14,101,NONE,67,18] pg 31.42 is stuck undersized for 7h, current state active+undersized+degraded+remapped+backfill_wait, last acting [37,134,NONE,82,8] pg 31.44 is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [49,NONE,101,3,62] pg 31.46 is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfilling, last acting [95,38,NONE,25,102] pg 31.47 is stuck undersized for 18h, current state active+undersized+degraded+remapped+backfill_wait, last acting [60,NONE,130,72,110] pg 31.4a is stuck undersized for 4h, current state active+undersized+degraded+remapped+backfilling, last acting [66,NONE,135,82,14] pg 31.4c is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [34,NONE,1,82,18] pg 31.4d is stuck undersized for 2h, current state active+undersized+degraded+remapped+backfill_wait, last acting [101,13,65,30,NONE] pg 31.52 is stuck undersized for 4m, current state active+undersized+remapped+backfill_wait, last acting [26,112,66,NONE,135] pg 31.53 is stuck undersized for 17h, current state active+undersized+degraded+remapped+backfill_wait, last acting [NONE,24,11,42,110] pg 31.55 is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [116,NONE,27,4,117] pg 31.57 is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [17,15,110,NONE,1] pg 31.5a is stuck undersized for 19h, current state active+undersized+degraded+remapped+backfill_wait, last acting [13,24,124,26,NONE] pg 31.5b is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [NONE,31,8,27,42] pg 31.5c is stuck undersized for 2h, current state active+undersized+degraded+remapped+backfill_wait, last acting [NONE,60,28,18,119] pg 31.5d is stuck undersized for 17h, current state active+undersized+degraded+remapped+backfill_wait, last acting [124,NONE,85,6,11] pg 31.5e is stuck undersized for 2h, current state active+undersized+degraded+remapped+backfill_wait, last acting [NONE,4,1,23,138] pg 31.60 is stuck undersized for 18h, current state active+undersized+degraded+remapped+backfilling, last acting [37,11,102,NONE,133] pg 31.64 is stuck undersized for 4m, current state active+undersized+remapped+backfill_wait, last acting [26,106,45,34,NONE] pg 31.65 is stuck undersized for 4h, current state active+undersized+degraded+remapped+backfill_wait, last acting [NONE,14,127,62,3] pg 31.66 is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [34,NONE,35,23,59] pg 31.67 is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [67,24,127,8,NONE] pg 31.68 is stuck undersized for 4m, current state active+undersized+degraded+remapped+backfill_wait, last acting [32,41,18,17,NONE] pg 31.70 is stuck undersized for 19h, current state active+undersized+degraded+remapped+backfilling, last acting [106,38,1,NONE,97] pg 31.75 is stuck undersized for 2h, current state active+undersized+degraded+remapped+backfill_wait, last acting [26,31,34,8,NONE] pg 31.7f is stuck undersized for 2h, current state active+undersized+remapped+backfilling, last acting [NONE,101,110,10,45] [WRN] PG_NOT_DEEP_SCRUBBED: 127 pgs not deep-scrubbed in time pg 26.e5 not deep-scrubbed since 2025-06-07T05:59:22.612553+0000 pg 26.ca not deep-scrubbed since 2025-06-06T18:21:07.435823+0000 pg 26.bf not deep-scrubbed since 2025-06-03T07:29:02.791095+0000 pg 26.be not deep-scrubbed since 2025-06-04T07:55:54.389714+0000 pg 26.a5 not deep-scrubbed since 2025-06-07T07:28:56.878096+0000 pg 26.85 not deep-scrubbed since 2025-06-03T15:18:59.395929+0000 pg 26.7f not deep-scrubbed since 2025-06-04T21:29:28.412637+0000 pg 26.7e not deep-scrubbed since 2025-06-03T09:14:19.585388+0000 pg 26.7d not deep-scrubbed since 2025-06-03T18:37:30.931020+0000 pg 26.7c not deep-scrubbed since 2025-06-04T13:38:00.061488+0000 pg 26.73 not deep-scrubbed since 2025-06-03T06:20:15.111819+0000 pg 26.6f not deep-scrubbed since 2025-06-03T13:45:24.880397+0000 pg 26.6e not deep-scrubbed since 2025-05-26T23:15:32.099862+0000 pg 26.6d not deep-scrubbed since 2025-06-04T14:04:10.449101+0000 pg 31.62 not deep-scrubbed since 2025-06-03T13:34:49.518456+0000 pg 26.65 not deep-scrubbed since 2025-06-04T07:56:25.353411+0000 pg 31.66 not deep-scrubbed since 2025-06-03T10:32:05.364424+0000 pg 26.62 not deep-scrubbed since 2025-06-04T09:35:58.267976+0000 pg 31.65 not deep-scrubbed since 2025-06-03T16:04:40.003140+0000 pg 31.5b not deep-scrubbed since 2025-06-03T14:18:18.835477+0000 pg 26.5d not deep-scrubbed since 2025-06-04T15:14:30.870252+0000 pg 31.58 not deep-scrubbed since 2025-06-03T03:09:27.568605+0000 pg 26.5c not deep-scrubbed since 2025-06-03T01:57:27.644129+0000 pg 31.5f not deep-scrubbed since 2025-06-03T05:53:20.860393+0000 pg 31.52 not deep-scrubbed since 2025-05-27T00:01:27.040861+0000 pg 31.53 not deep-scrubbed since 2025-05-24T09:37:58.964829+0000 pg 26.55 not deep-scrubbed since 2025-06-04T21:25:34.135356+0000 pg 26.54 not deep-scrubbed since 2025-06-04T06:07:12.978734+0000 pg 31.56 not deep-scrubbed since 2025-06-04T12:58:17.599712+0000 pg 31.57 not deep-scrubbed since 2025-06-03T07:02:16.859990+0000 pg 26.51 not deep-scrubbed since 2025-06-03T05:42:22.435483+0000 pg 26.4f not deep-scrubbed since 2025-06-03T09:10:22.617328+0000 pg 31.4a not deep-scrubbed since 2025-05-28T00:54:55.246532+0000 pg 26.4e not deep-scrubbed since 2025-06-03T11:16:49.278513+0000 pg 31.4b not deep-scrubbed since 2025-06-03T10:24:24.123351+0000 pg 26.4d not deep-scrubbed since 2025-06-04T19:01:44.614410+0000 pg 31.49 not deep-scrubbed since 2025-05-28T04:56:29.368285+0000 pg 31.42 not deep-scrubbed since 2025-05-28T08:38:57.151865+0000 pg 26.41 not deep-scrubbed since 2025-06-03T05:47:35.443867+0000 pg 26.40 not deep-scrubbed since 2025-06-03T05:43:13.283668+0000 pg 25.17 not deep-scrubbed since 2025-06-03T09:26:52.253625+0000 pg 32.2e not deep-scrubbed since 2025-06-05T13:01:06.175389+0000 pg 22.1a not deep-scrubbed since 2025-06-07T01:40:45.063268+0000 pg 31.13 not deep-scrubbed since 2025-06-03T12:21:17.965218+0000 pg 22.1b not deep-scrubbed since 2025-06-04T12:22:44.947751+0000 pg 25.14 not deep-scrubbed since 2025-05-28T06:26:32.552200+0000 pg 26.17 not deep-scrubbed since 2025-06-03T18:37:26.617483+0000 pg 31.12 not deep-scrubbed since 2025-05-28T08:39:23.271194+0000 pg 31.1c not deep-scrubbed since 2025-06-03T08:17:51.230187+0000 pg 25.1f not deep-scrubbed since 2025-05-28T00:19:23.653883+0000 77 more pgs… Regards Dev

6 16

Why is it still so difficult to just dump all config and where it comes from?
by Niklas Hambüchen 22 Jun '25

22 Jun '25

Year after year I find myself confused with ceph's various methods to just show the current configuration. I just want one command that can dump all the config, and where the values come from. I used a mix of the following so far: ceph config dump "dumps the entire Monitor central configuration database" But it shows only values that are different from the default. Outputs a "WHO" column that includes entries for both "mgr" or "osd", and e.g. "osd.123". config show-with-defaults <who> Very useful: Outputs SOURCE/OVERRIDES/IGNORES Oddly, not listed in https://docs.ceph.com/en/squid/rados/configuration/ceph-conf/#commands Unfortunately, cannot dump all config, instead needs to be given <who> e.g. `config show-with-defaults osd.123`. But not all things from the WHO column from "ceph config dump" are accepted, e.g. `config show-with-defaults mgr` does not work. Confusing. Somehow misses some values: `ceph config dump` shows WHO MASK LEVEL OPTION VALUE RO mgr unknown mgr/pg_autoscaler/autoscale_profile scale-up * But `autoscale_profile` is nowhere to be found in any `ceph config show-with-defaults mgr...` outputs. ceph-conf --show-config Legacy, as explained on https://docs.ceph.com/en/squid/man/8/ceph-conf/ But I think https://docs.ceph.com/en/squid/rados/configuration/ceph-conf/#commands should list it so that from there one can easily see it's legacy. This is all very confusing. Can we please have 1 command, that can dump all config (including from config files, monitor central configuration database, all currently running daemons), and nicely point out what's set and overridden where and which value is in effect? Is there any issue I can subscribe to that tracks improving this? Thanks!

2 3

OSD network Issue -- Not using cluster Network
by Devender Singh 22 Jun '25

22 Jun '25

Hello All Found some network issue on OSD’s, why they are not using cluster network? Only using public network? My mons are on public network… How can I resolve it? Changing this network will lead any issue like need to stop all osd’s or restart cluster or rolling restart osd’s & how, please help? # ceph config get mon cluster_network 10.102.5.0/24 root@pl-host01n:/cephfs/bin# ceph config get mon public_network 10.102.1.0/24 root@pl-host01n:/cephfs/bin# ceph osd find osd.8 { "osd": 8, "addrs": { "addrvec": [ { "type": "v2", "addr": "10.102.1.122:6806", "nonce": 4172377551 }, { "type": "v1", "addr": "10.102.1.122:6807", "nonce": 4172377551 } ] }, "osd_fsid": "1b167c95-a873-4c02-aafc-24067246e65d", "host": "pl-host02n.ld6.example.”com, "crush_location": { "host": "pl-host02n", "root": "default" } } Regards Dev

2 1

Adding OSD nodes
by Sinan Polat 21 Jun '25

21 Jun '25

Hello, I am currently managing a Ceph cluster that consists of 3 racks, each with 4 OSD nodes. Each node contains 24 OSDs. I plan to add three new nodes, one to each rack, to help alleviate the high OSD utilization. The current highest OSD utilization is 85%. I am concerned about the possibility of any OSD reaching the osd_full_ratio threshold during the rebalancing process. This would cause the cluster to enter a read-only state, which I want to avoid at all costs. I am planning to execute the following commands: ceph orch host add new-node-1 ceph orch host add new-node-2 ceph orch host add new-node-3 ceph osd crush move new-node-1 rack=rack-1 ceph osd crush move new-node-2 rack=rack-2 ceph osd crush move new-node-3 rack=rack-3 ceph config set osd osd_max_backfills 1 ceph config set osd osd_recovery_max_active 1 ceph config set osd osd_recovery_sleep 0.1 ceph orch apply osd --all-available-devices Before proceeding, I would like to ask if the above steps are safe to execute in a cluster with such high utilization. My main concern is whether the rebalancing could cause any OSD to exceed the osd_full_ratio and result in unexpected failures. Any insights or advice on how to safely add these nodes without impacting cluster stability would be greatly appreciated. Thanks!

7 7

el10 plans
by Christopher Durham 20 Jun '25

20 Jun '25

Hi, When will rpms for Linux EL10 and clones such as Rocky Linux be available, and what ceph release will it be? I am not specifically asking when RedHat or IBMthemselves will release on this version, I am looking for el10 rpms at https://download.ceph.com. Thanks -Chris

2 1

Ceph reef vs squid performance
by Wissem MIMOUNA - Ceph Users 20 Jun '25

20 Jun '25

Hi ceph users, We have two ceph cluster with the same hardware/CPU(ARM)/OS(debian 12)/network , and both deployed with cephadm ( default configuration , 2 osds per device nvme ) , except that one is on reef version and the seco,d is on squid version. We noticed a big difference in our benchmark ( the Reef cluster is far better the Squid cluster ). Does mean that Ceph Reef is more performant than Ceph Squid ? Squid RandRW bench => fio -ioengine=rbd -name=test-2 -direct=1 -bs=4k -iodepth=1 -rw=randrw -rwmixwrite=100 -runtime=10 -pool=nova -rbdname=myimage test-2: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-3.33 Starting 1 process Jobs: 1 (f=1): [w(1)][100.0%][w=2488KiB/s][w=622 IOPS][eta 00m:00s] test-2: (groupid=0, jobs=1): err= 0: pid=331890: Fri Jun 20 11:35:14 2025 write: IOPS=627, BW=2511KiB/s (2571kB/s)(24.5MiB/10002msec); 0 zone resets slat (nsec): min=4400, max=51722, avg=15476.42, stdev=4695.32 clat (usec): min=670, max=7704, avg=1574.43, stdev=341.38 lat (usec): min=676, max=7730, avg=1589.91, stdev=341.83 clat percentiles (usec): | 1.00th=[ 1074], 5.00th=[ 1303], 10.00th=[ 1369], 20.00th=[ 1434], | 30.00th=[ 1483], 40.00th=[ 1516], 50.00th=[ 1549], 60.00th=[ 1582], | 70.00th=[ 1614], 80.00th=[ 1647], 90.00th=[ 1696], 95.00th=[ 1778], | 99.00th=[ 2769], 99.50th=[ 3458], 99.90th=[ 5800], 99.95th=[ 5932], | 99.99th=[ 7701] bw ( KiB/s): min= 2000, max= 2608, per=100.00%, avg=2512.42, stdev=131.03, samples=19 iops : min= 500, max= 652, avg=628.11, stdev=32.76, samples=19 lat (usec) : 750=0.14%, 1000=0.61% lat (msec) : 2=96.21%, 4=2.64%, 10=0.40% cpu : usr=1.34%, sys=0.93%, ctx=6280, majf=0, minf=5 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,6278,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Reef RandRW bench=> | | |fio -ioengine=rbd -name=test -direct=1 -bs=4K -iodepth=1 -rw=randrw -rwmixwrite=100 -runtime=10 -pool=nova -rbdname=image1 test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-3.33 | |Starting 1 process | |Jobs: 1 (f=1): [w(1)][100.0%][w=3931KiB/s][w=982 IOPS][eta 00m:00s] | |test: (groupid=0, jobs=1): err= 0: pid=1866868: Tue Jun 17 15:24:42 2025 | |write: IOPS=964, BW=3860KiB/s (3952kB/s)(37.7MiB/10001msec); 0 zone resets | |slat (nsec): min=3880, max=30402, avg=6499.36, stdev=1597.11 | |clat (usec): min=617, max=3219, avg=1027.83, stdev=161.62 | |lat (usec): min=625, max=3224, avg=1034.33, stdev=161.62 | |clat percentiles (usec): | 1.00th=[ 824], 5.00th=[ 881], 10.00th=[ 906], 20.00th=[ 938], | 30.00th=[ 963], 40.00th=[ 979], 50.00th=[ 1004], 60.00th=[ 1020], | 70.00th=[ 1045], 80.00th=[ 1074], 90.00th=[ 1123], 95.00th=[ 1221], | 99.00th=[ 1860], 99.50th=[ 1942], 99.90th=[ 2147], 99.95th=[ 2343], | 99.99th=[ 3228] | |bw ( KiB/s): min= 3152, max= 3984, per=100.00%, avg=3861.89, stdev=183.90, samples=19 | |iops : min= 788, max= 996, avg=965.47, stdev=45.98, samples=19 | |lat (usec) : 750=0.16%, 1000=49.50% | |lat (msec) : 2=50.07%, 4=0.27% | |cpu : usr=0.91%, sys=0.51%, ctx=9657, majf=0, minf=13 | |IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,9650,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 |

1 0

Cluster Not recovering after multiple disk failure
by Devender Singh 20 Jun '25

20 Jun '25

Hello all Need urgent help on below… I tried reducing min_size but still showing same… cluster: id: 15688cb4-044a-11ec-942e-516035adea04 health: HEALTH_ERR 3 failed cephadm daemon(s) 1 filesystem is degraded 1 MDSs report slow metadata IOs 20/16670718 objects unfound (0.000%) Reduced data availability: 283 pgs inactive, 464 pgs incomplete Possible data damage: 2 pgs recovery_unfound Degraded data redundancy: 42998/61175329 objects degraded (0.070%), 2 pgs degraded, 1 pg undersized 304 pgs not deep-scrubbed in time 1055 slow ops, oldest one blocked for 94042 sec, daemons [osd.1,osd.12,osd.13,osd.15,osd.16,osd.19,osd.20,osd.21,osd.26,osd.27]... have slow ops. services: mon: 5 daemons, quorum van2-converged04n,van2-converged05n,van2-converged01n,van2-converged03n,van2-converged02n (age 17h) mgr: van2-converged05n.lybhho(active, since 22h), standbys: van2-converged04n.azmqik mds: 3/3 daemons up, 11 standby osd: 31 osds: 31 up (since 5h), 31 in (since 5h); 2 remapped pgs rbd-mirror: 2 daemons active (2 hosts) data: volumes: 1/2 healthy, 1 recovering pools: 32 pools, 1505 pgs objects: 16.67M objects, 56 TiB usage: 152 TiB used, 234 TiB / 387 TiB avail pgs: 0.332% pgs unknown 30.897% pgs not active 42998/61175329 objects degraded (0.070%) 21469/61175329 objects misplaced (0.035%) 20/16670718 objects unfound (0.000%) 1034 active+clean 463 incomplete 5 unknown 1 remapped+incomplete 1 active+recovery_unfound+degraded 1 recovery_unfound+undersized+degraded+remapped+peered progress: Global Recovery Event (22h) [===================.........] (remaining: 10h) [WRN] SLOW_OPS: 1055 slow ops, oldest one blocked for 93877 sec, daemons [osd.1,osd.12,osd.13,osd.15,osd.16,osd.19,osd.20,osd.21,osd.26,osd.27]... have slow ops. [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs mds.van2.van2-converged05n.mbbzfj(mds.1): 1 slow metadata IOs are blocked > 30 secs, oldest blocked for 62279 secs [WRN] OBJECT_UNFOUND: 20/16670718 objects unfound (0.000%) pg 28.1f has 9 unfound objects pg 29.2 has 11 unfound objects [WRN] PG_AVAILABILITY: Reduced data availability: 470 pgs inactive, 464 pgs incomplete pg 16.58 is incomplete, acting [7,14,19] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.5c is incomplete, acting [26,3,19] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.5f is incomplete, acting [13,6,21] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.63 is incomplete, acting [9,7,15] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.64 is incomplete, acting [18,19,5] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.68 is incomplete, acting [2,5,27] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.70 is incomplete, acting [19,1,24] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.74 is incomplete, acting [8,25,14] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.78 is incomplete, acting [23,14,29] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.7c is incomplete, acting [24,10,29] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 16.7d is incomplete, acting [12,22,13] (reducing pool lv-r3-for-ec-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.64 is incomplete, acting [20,11,13] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.66 is incomplete, acting [21,13,27] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.6a is incomplete, acting [4,26,13] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.6c is incomplete, acting [13,12,31] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.6e is incomplete, acting [8,31,24] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.7a is incomplete, acting [8,28,27] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 19.54 is stuck inactive since forever, current state incomplete, last acting [10,4,20,15,11] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.58 is incomplete, acting [15,18,12,28,20] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.59 is incomplete, acting [6,9,18,21,26] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.5a is incomplete, acting [11,31,20,17,24] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for ‘incomplceph -s Ceph Any way to make these scrubs complete faster? pg 19.63 not deep-scrubbed since 2025-05-24T08:18:29.738427+0000 pg 34.6d not deep-scrubbed since 2025-06-04T08:33:54.534882+0000 pg 16.5f not deep-scrubbed since 2025-05-21T10:44:09.996254+0000 pg 19.5d not deep-scrubbed since 2025-05-26T09:36:27.064154+0000 pg 19.5e not deep-scrubbed since 2025-06-05T00:52:03.859984+0000 pg 16.5c not deep-scrubbed since 2025-06-06T00:36:22.021390+0000 pg 19.5f not deep-scrubbed since 2025-06-03T16:27:42.356213+0000 pg 22.5a not deep-scrubbed since 2025-06-05T23:00:28.066065+0000 pg 34.69 not deep-scrubbed since 2025-06-03T05:07:58.209808+0000 pg 19.58 not deep-scrubbed since 2025-05-27T23:32:29.963976+0000 pg 19.59 not deep-scrubbed since 2025-05-25T11:50:44.735318+0000 pg 19.5a not deep-scrubbed since 2025-06-06T02:34:05.486126+0000 pg 16.58 not deep-scrubbed since 2025-05-13T14:10:44.570493+0000 Regards Dev

3 4

Ceph 14.2.22 MONs keep crashing in MDSMonitor::maybe_resize_cluster (out of range)
by Miles Goodhew 20 Jun '25

20 Jun '25

Hi, I've been called-in by a client with an ancient SUSE-based Ceph Nautilus (14.2.22) who's MONs keep dieing oddly. Apparently the issue started with MDS daemons not working and eventuallt a MON restart killed the cluster. OS: SLES 15-SP1 (out of support) Ceph: 14.2.22 "Nautilus" (Deployed with Salt... I think) 3 MONs; 5 MDSs; 3 MGRs; 4 RGWs; 336 OSDs on 21 nodes. Client services: "One of everything at least", but RBD/Openstack, S3/RGW and CephFS are big ones. After sorting out some of the logs here are some things I know: Disk space, RAM availability, inodes and network connectivity seem OK to me. After shutting-down all the MONs, MGRs and MDSes, one MON can usually be started, but it sits there spamming-out log messages like "[SERVICE_ID](probing) e6 handle_auth_request failed to assign global_id" (maybe 50 - 100 times per second). All the while the syslog shows 'e6 get_health_metrics reporting [INCREASING_NUMBER] slow ops` fairly often. This is probably due to OSDs and clients being active. If I restart one of the other MONs, the running one will die with a stack trace at (Limiting to C++/library internal calls): ``` 8: (std::__throw_out_of_range(char const*)+0x41) [0x7f2a5983fa07] 9: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0xcf0) [0x55b441e37490] 10: (MDSMonitor::tick()+0xc9) [0x55b441e38ce9] 11: (MDSMonitor::on_active()+0x28) [0x55b441e22fa8] 12: (PaxosService::_active()+0xdd) [0x55b441d7188d] 13: (Context::complete(int)+0x9) [0x55b441c888a9] 14: (void finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > >(CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0xa8) [0x55b441cb2408] 15: (Paxos::finish_round()+0x76) [0x55b441d681b6] 16: (Paxos::handle_last(boost::intrusive_ptr<MonOpRequest>)+0xc1f) [0x55b441d693df] 17: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x233) [0x55b441d69e23] 18: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x1668) [0x55b441c820b8] 19: (Monitor::_ms_dispatch(Message*)+0xa3a) [0x55b441c82b5a] 20: (Monitor::ms_dispatch(Message*)+0x26) [0x55b441cb3646] 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x26) [0x55b441cb00b6] 22: (DispatchQueue::entry()+0x1279) [0x7f2a5b188379] 23: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2a5b238a5d] 24: (()+0x8539) [0x7f2a59db7539] 25: (clone()+0x3f) [0x7f2a58f87ecf] ``` Anyone got any clues about how to diagnose or better-yet repair this? Sorry, I know this is a bit half-baked, but I'm trying to dump this help request at COB to see if I can hook anyone's interest overnight. Thanks for at least reading this far, M0les.

2 14

Securing Public RGW /admin access
by Deep Dish 19 Jun '25

19 Jun '25

Hello, I'm using Ceph Squid, currently 19.2.1. I'd like to use either the built-in ingress service or an approach with the RGW service to restrict /admin access from public endpoints. My setup: Firewall (NGINX reverse proxy - TCP 443 stream passthrough only - no SSL) | Ingress Service (SSL Certificates applied) | RGW Gateways (SSL Certificates applied) All administrative functions interact with RGW Gateways directly. Only public flows pass through the Ingress Service. As I understand /admin API is required for gateway and RGW service management. I want to manage my SSL certs ideally via ceph spec files (we have automation for that). The frontend firewall is designed to act as a simple L3 / L4 ACL. Is there a way to modify the Ingress service spec file to restrict /admin access. According to GROK adding the following to the ingress spec YAML should work, but the custom_configs section throws an error as follows. I can't seem to find any official documentation around this. Feels like it should be something to easily configure. How are others securing this? Thanks. --- service_type: ingress service_id: ingress01 placement: hosts: - ingress01 spec: backend_service: rgw.public virtual_ip: 10.1.1.1/24 frontend_port: 443 monitor_port: 1967 custom_configs: - | frontend frontend acl block_admin path_beg /admin http-request deny if block_admin ssl_cert: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- -----BEGIN PRIVATE KEY----- ... -----END PRIVATE KEY----- Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1862, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 184, in handle_command return dispatch[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 499, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 120, in <lambda> wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 109, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/orchestrator/module.py", line 1697, in apply_misc return self._apply_misc(specs, dry_run, format, no_overwrite) File "/usr/share/ceph/mgr/orchestrator/module.py", line 1700, in _apply_misc completion = self.apply(specs, no_overwrite) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1737, in inner completion = self._oremote(method_name, args, kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1806, in _oremote mgr.log.debug("_oremote {} -> {}.{}(*{}, **{})".format(mgr.module_name, o, meth, args, kwargs)) File "/lib/python3.9/site-packages/ceph/deployment/service_spec.py", line 1109, in __repr__ y = yaml.dump(cast(dict, self), default_flow_style=False) File "/lib64/python3.9/site-packages/yaml/__init__.py", line 290, in dump return dump_all([data], stream, Dumper=Dumper, **kwds) File "/lib64/python3.9/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/lib64/python3.9/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/lib64/python3.9/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/lib/python3.9/site-packages/ceph/deployment/service_spec.py", line 1122, in yaml_representer return dumper.represent_dict(cast(Mapping, data.to_json().items())) File "/lib/python3.9/site-packages/ceph/deployment/service_spec.py", line 1062, in to_json ret['custom_configs'] = [c.to_json() for c in self.custom_configs] File "/lib/python3.9/site-packages/ceph/deployment/service_spec.py", line 1062, in <listcomp> ret['custom_configs'] = [c.to_json() for c in self.custom_configs] AttributeError: 'str' object has no attribute 'to_json' ---

1 0

ceph rbd mirror snapshot-based does not work if mirror-mode is pool and not image
by Julien Laurenceau 19 Jun '25

19 Jun '25

Hi, I have 2 running ceph squid clusters (19.2.2). On each cluster there is an rbd pool named k8s-1 that I want to mirror using the snapshot based mode. if I configure both clusters in two way mirroring using the mode=image everything works fine. ``` rbd mirror pool enable ${mypool} image rbd mirror image enable ${mypool}/${myimage} snapshot ``` When I take a snapshot of myimage it gets mirrored. if I configure both clusters in two way mirroring using the mode=pool nothings happen. The snapshot are never mirrored in the second cluster. Reading the documentation https://docs.ceph.com/en/squid/rbd/rbd-mirroring/, I think that the problem is that the default mode for this command is "journal": rbd mirror image enable {pool-name}/{image-name} {mode} But, when enabling mirror on the pool there is no way to define mode=journal/snapshot: rbd mirror pool enable [--site-name {local-site-name}] {pool-name} {mode} with mode=pool (default) or image. Then I think when I do : rbd mirror pool enable ${mypool} image I think it implicitly do : rbd mirror pool enable ${mypool} image journal # THIS COMMAND DOES NOT WORK. But I would need to have: rbd mirror pool enable ${mypool} image snapshot # THIS COMMAND DOES NOT WORK. In addition, when I am configured on mirror mode = pool, if I try to disable mirroring on an image I get the following error : rbd mirror image disable ${mypool}/${myimage} 2025-06-18T15:37:42.177+0000 7f24ec53ce00 -1 librbd::api::Mirror: image_disable: cannot disable mirroring in the current pool mirroring mode Also error when I am configured on mirror mode = pool, if I try to enable snapshot mirroring on an image : rbd mirror image enable ${mypool}/${myimage} snapshot 2025-06-18T15:39:44.507+0000 7f533df1fe00 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: pool is not in image mirror mode Could someone please confirm, that this is not supported ? What is the best workaround ? Cron cephadm shell script to enable mirror on all images ? PS : I do not want to use the journal based mode since the ceph-csi mounter nbd is not prod-ready.

2 3

RadosGW: Even more large omap objects after resharding
by Niklaus Hofer 19 Jun '25

19 Jun '25

Dear all We are running Ceph Pacific (16.2.15) with RadosGW and have been getting "large omap objects" health warnings on the RadosGW index pool. Indeed we had one bucket in particular that was positively huge with 8'127'198 objects that had just a single shard. But we have been seeing the message on some other buckets, too. Eventually, we activated automatic resharding (rgw_dynamic_resharding = true) and indeed this bucket was resharded to now 167 shards. However, I am now getting even more large omap object warnings. On that same bucket, too. The other buckets have not been resharded at all. They are not in the queue, either: | radosgw-admin reshard list > [] | grep 'Large omap object found' /var/log/ceph/ceph* | grep 'PG: ' | cut -d: -f 10 | cut -d. -f 4-5 | sort | uniq -c 2 7936686773.215 12 7937604172.149 10 7955243979.1209 9 7955243979.2480 13 7955243979.2481 12 7968198782.110 13 7968913553.67 11 7968913553.68 10 7968913553.69 11 7981210604.1 74 7981624399.1 217 7988881492.1 | radosgw-admin metadata list --metadata-key bucket.instance | grep -i 7988881492 > "<bucket1_name>:<pool_name>.7988881492.1", | radosgw-admin bucket stats --bucket <bucket1_name> > { > "bucket": "<bucket1_name>", > "num_shards": 167, > [...] > "usage": { > "rgw.main": { > "size": 9669928611955, > "size_actual": 9692804734976, > "size_utilized": 9669928611955, > "size_kb": 9443289661, > "size_kb_actual": 9465629624, > "size_kb_utilized": 9443289661, > "num_objects": 8134437 > } > }, Let's check another one, too: | radosgw-admin metadata list --metadata-key bucket.instance | grep -i 7968198782.110 > "<bucket2_name>:<pool_name>.7968198782.110", | radosgw-admin bucket stats --bucket <bucket2_name> > [...] > "num_objects": 38690 > [...] According to the documentation in [0], buckets are resharded at a threshold of 100'000 objects per shard. For both of these, that applies nicely, so it makes sense that they are not getting resharded any further. But why then am I getting these warnings? Reading the documentation in [1], I can see that warnings are printed at 200'000 entries per omap object. Can I assume that one object in an RGW bucket means 1 entry in an omap object? Or is that a missconception? Now, here is my working theory. Please let me know if that has any merit or if I'm completely off: The affected buckets have versioning activated. Plus object locking too. They get used by a backup software (Kopia) that uses these features to provide ransomware protection. So my thinking is that maybe with versioning active, each object in a bucket could result in multiple omap entries, maybe one per version or something? If that is the case, then maybe I should reduce `rgw_max_objs_per_shard` from 100'000 to something like 10'000 to have the buckets resharded more aggressively? But then again, that assumes a lot. For example, that assumes that the num_objects counter in the bucket stats does not count up on versioned objects. So my assumption could be completely whack. What do you think? What can I do to get rid of the large omap objects? Is more resharding going to help? What else could I check? Sincerely Niklaus Hofer Links: [0] https://docs.ceph.com/en/latest/radosgw/dynamicresharding/#confval-rgw_max_… [1] https://docs.ceph.com/en/latest/rados/operations/health-checks/#large-omap-… -- stepping stone AG Wasserwerkgasse 7 CH-3011 Bern Telefon: +41 31 332 53 63 www.stepping-stone.ch niklaus.hofer(a)stepping-stone.ch

4 12

Autoscale warnings depite autoscaler being off
by Niklaus Hofer 19 Jun '25

19 Jun '25

Dear all After upgrading to Pacific, we are now getting health warnings from the auto scaler: > 10 pools have too few placement groups > 8 pools have too many placement groups The thing is, I do not particularly care about the recommendations from the auto scaler. So I checked and saw that it was set to warning on all pools: | ceph osd pool autoscale-status > POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK > openstack-glance 902.7G 3.0 618.2T 0.0043 1.0 128 warn False > openstack-cinder 99.48T 3.0 618.2T 0.4827 1.0 2048 warn False > openstack-nova 169.6G 3.0 618.2T 0.0008 1.0 128 warn False > [...] No worries, I just turn it off: | ceph osd pool set noautoscale Check it's off: | ceph osd pool autoscale-status > POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK > openstack-glance 902.7G 3.0 618.2T 0.0043 1.0 128 off False > openstack-cinder 99.48T 3.0 618.2T 0.4827 1.0 2048 off False > openstack-nova 169.6G 3.0 618.2T 0.0008 1.0 128 off False > [...] That looks as expected. But despite that, I'm still getting the health warnings: | ceph --status > [...] > 10 pools have too few placement groups > 8 pools have too many placement groups So how can I get rid of the warnings (without changing the number of PGs on my pools!)? Sincerely Niklaus Hofer -- stepping stone AG Wasserwerkgasse 7 CH-3011 Bern Telefon: +41 31 332 53 63 www.stepping-stone.ch niklaus.hofer(a)stepping-stone.ch

3 3

4k IOPS: miserable performance in All-SSD cluster
by Martin Gerhard Loschwitz 18 Jun '25

18 Jun '25

Folks, I am getting somewhat desperate debugging multiple setups here within the same environment. Three clusters, two SSD-only, one HDD-only, and what they all have in common is abysmal 4k IOPS performance when measuring with „rados bench“. Abysmal means: In an All-SSD cluster I will get roughly 400 IOPS over more than 250 devices. I’ve know SAS-SSDs are not ideal, but 250 looks a bit on the low side of things to me. In the second cluster, also All-SSD based, I get roughly 120 4k IOPS. And the HDD-only cluster delivers 60 4k IOPS. The latter both with substantially fewer devices, granted. But even with 20 HDDs, 68 4k IOPS seems like a very bad value to me. I’ve tried to rule out everything I know of: BIOS misconfigurations, HBA problems, networking trouble (I am seeing comparably bad values with a size=1 pool) and so further and so on. But to no avail. Has anybody dealt with something similar on Dell hardware or in general? What could cause such extremely bad benchmark results? I measure with rados bench and qd=1 at 4k block size. „ceph tell osd bench“ with 4k blocks yields 30k+ IOPS for every single device in the big cluster, and all that leads to is 400 IOPS in total when writing to it? Even with no replication in place? That looks a bit off, doesn't it? Any help will be greatly appreciated, thank you very much in advance. Even a pointer to the right direction would be held in high esteem right now. Thank you very much in advance! Best regards Martin

8 13

ceph rbd mirror snapshot-based does not work if mirror-mode is pool and not image
by Julien Laurenceau 18 Jun '25

18 Jun '25

Hi, I have 2 running ceph squid clusters (19.2.2). On each cluster there is an rbd pool named k8s-1 that I want to mirror using the snapshot based mode. if I configure both clusters in two way mirroring using the mode=image everything works fine. ``` rbd mirror pool enable ${mypool} image rbd mirror image enable ${mypool}/${myimage} snapshot ``` When I take a snapshot of myimage it gets mirrored. if I configure both clusters in two way mirroring using the mode=pool nothings happen. The snapshot are never mirrored in the second cluster. Reading the documentation https://docs.ceph.com/en/squid/rbd/rbd-mirroring/, I think that the problem is that the default mode for this command is "journal": rbd mirror image enable {pool-name}/{image-name} {mode} But, when enabling mirror on the pool there is no way to define mode=journal/snapshot: rbd mirror pool enable [--site-name {local-site-name}] {pool-name} {mode} with mode=pool (default) or image. Then I think when I do : rbd mirror pool enable ${mypool} image I think it implicitly do : rbd mirror pool enable ${mypool} image journal # THIS COMMAND DOES NOT WORK. But I would need to have: rbd mirror pool enable ${mypool} image snapshot # THIS COMMAND DOES NOT WORK. In addition, when I am configured on mirror mode = pool, if I try to disable mirroring on an image I get the following error : rbd mirror image disable ${mypool}/${myimage} 2025-06-18T15:37:42.177+0000 7f24ec53ce00 -1 librbd::api::Mirror: image_disable: cannot disable mirroring in the current pool mirroring mode Also error when I am configured on mirror mode = pool, if I try to enable snapshot mirroring on an image : rbd mirror image enable ${mypool}/${myimage} snapshot 2025-06-18T15:39:44.507+0000 7f533df1fe00 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: pool is not in image mirror mode Could someone please confirm, that this is not supported ? What is the best workaround ? Cron cephadm shell script to enable mirror on all images ? PS : I do not want to use the journal based mode since the ceph-csi mounter nbd is not prod-ready.

1 0

How to clear the "slow operations" warning?
by Jan Kasprzak 18 Jun '25

18 Jun '25

Hello, Ceph users, TL;DR: how to clear/acknowledge the following warning from ceph -s? health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore Details: This is caused by a bad sector on the physical HDD - the time of this warning is the same as the most recent error in the SMART log of that HDD. # ceph health detail HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in BlueStore osd.33 observed slow operation indications in BlueStore When I restart osd.33 using "systemctl restart ceph-osd(a)33.service" on the OSD host, the warning disappears. But is there a "softer" way how to acknowledge this warning? Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

11 15

CephFS scrub resulting in MDS_CACHE_OVERSIZED
by Kasper Rasmussen 18 Jun '25

18 Jun '25

After starting a recursive scrub on a cephfs with alot of files the MDS cache went oversized. Scrub command: ceph... scrub start / recursive,repair,force I kept an eye on the MDS memory usage - since I was warned that it might go crazy.. and after 2-3 hours, I started getting the warning [WRN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache mds.generic-mds.<host>.asddje(mds.0): MDS cache is too large (63GB/36GB); 1250394 inodes in use by clients, 28888 stray files I then paused the scrub, resulting in scrub status { "status": "PAUSED (22837086 inodes in the stack)", "scrubs": { "27f0e32a-bc8c-443d-b1f0-534474798ddf": { "path": "/", "tag": "27f0e32a-bc8c-443d-b1f0-534474798ddf", "options": "recursive,repair,force" } } } and expected the cache size to go down again - but it didn't. After +12 hours with no change, I opted to abort the scrub - again expecting that the inodes in the stack would be offloaded from memory. The status after abort command: { "status": "PAUSED (0 inodes in the stack)", "scrubs": {} } But still no changes to the cache size. Since the status after the abort command had "PAUSED" in it, I resumed the scrub, resulting in status: { "status": "no active scrubs running", "scrubs": {} } Still no changes to the cache size. The log from the MDS in standard log level was: debug 2025-06-03T06:48:24.122+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub start {path=/,prefix=scrub start,scrubops=[recursive,repair,force]} (starting...) debug 2025-06-03T06:48:24.122+0000 7f318864d640 0 log_channel(cluster) log [INF] : scrub queued for path: / debug 2025-06-03T06:48:24.122+0000 7f318864d640 0 log_channel(cluster) log [INF] : scrub summary: idle+waiting paths [/] debug 2025-06-03T06:48:24.122+0000 7f318864d640 0 log_channel(cluster) log [INF] : scrub summary: active paths [/] debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) mismatch between head items and fnode.fragstat! printing dentries debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) get_num_head_items() = 38; fnode.fragstat.nfiles=28 fnode.fragstat.nsubdirs=11 debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) mismatch between child accounted_rstats and my rstats! debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) total of child dentries: n(v0 rc2025-06-03T06:48:11.042059+0000 b1661845634 127=95+32) debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) my rstats: n(v544237 rc2025-06-03T06:48:11.042059+0000 b1661845650 128=96+32) debug 2025-06-03T06:49:38.689+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status {prefix=scrub status} (starting...) debug 2025-06-03T06:51:49.782+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status {prefix=scrub status} (starting...) debug 2025-06-03T06:55:39.654+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status {prefix=scrub status} (starting...) debug 2025-06-03T07:00:56.205+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status .. .. From here it's either - asok_command: scrub status {prefix=scrub status} (starting...) - Updating MDS map to version xxxxxx from mon.3 Until I pause the scrub. Extracts from the perf dump from the MDS: "mds": { .. .. .. "inodes": 23121955, "inodes_top": 3684, "inodes_bottom": 1728, "inodes_pin_tail": 23116543, "inodes_pinned": 23116691, "inodes_expired": 39049803601, "inodes_with_caps": 84593, .. .. } .. .. "mds_mem": { "ino": 23114378, "ino+": 38966647328, "ino-": 38943532950, "dir": 513065, "dir+": 130921896, "dir-": 130408831, "dn": 23121954, "dn+": 39349549680, "dn-": 39326427726, "cap": 87280, "cap+": 6964477825, "cap-": 6964390545, "rss": 79730620, "heap": 223508 }, I've have been reluctant to just fail the MDS, to clear the memory, but when I finally came around to do so I got the error "Error EPERM: MDS has one of two health warnings which could extend recovery: MDS_TRIM or MDS_CACHE_OVERSIZED. MDS failover is not recommended since it might cause unexpected file system unavailability. If you wish to proceed, pass --yes-i-really-mean-it" At this moment the number strays reported in the MDS_CACHE_OVERSIZED warning, are now up with a factor 10 (approx. 280000) Which made me pause. This seems like a bug.. But To be honest I don't know quite what to expect, if I just execute with "--yes-i-really-mean-it".. Will the MDS eat huge amount of RAM during replay? (I've seen this before during failover - where MDS ate almost 200GB ram, even though the cache was not oversized.) Any advice on how to proceed? BR. Kasper

2 3

db_slots in service spec. is ignored
by Kasper Rasmussen 17 Jun '25

17 Jun '25

Hi I have a service spec for osds with the db_slots parameter set. However it's being ignored. After doing some code review, it seems to me that the reason is, that it's not implemented in translate.py - e.g. https://github.com/ceph/ceph/blob/v18.2.7/src/python-common/ceph/deployment… (or other versions) The translate.py simply does not have translation to "--block-db-slots" that is used by the ceph-volume command as specified here https://docs.ceph.com/en/reef/ceph-volume/lvm/batch/#implicit-sizing Something along the lines of: if db_slots: cmd += " --block-db-slots {}".format(self.spec.db_slots) Should probably be added after line 119 of v.18.2.7 at least - placement will defer depending on version - Any devs in here that can confirm this? Seems like a quick fix. * found this 2y old track for it as well - https://tracker.ceph.com/issues/57939

1 0

Use local reads with rados_replica_read_policy
by Eugen Block 17 Jun '25

17 Jun '25

Hi *, I have a question regarding the upcoming feature to optimize read performance [0] by reading from the nearest OSD, especially in a stretch cluster across two sites (or more). Anthony pointed me to [1], looks like a new config option will be introduced in Tentacle: rados_replica_read_policy Will this config option be limited to the "official" stretch mode? Or will it be possible to utilize it independent of the cluster layout? Thanks! Eugen [0] https://ceph.io/en/news/blog/2025/stretch-cluuuuuuuuusters-part2/ [1] https://github.com/ceph/ceph/blob/d28e5fe890016235e302122f955fc910c96f2d43/…

3 6

19.2.2: large number of scrubs suddenly not completed in time
by Michel Jouvin 17 Jun '25

17 Jun '25

Hi, We upgraded one of our production cluster (480 OSDs, 13,5 K PGs, most pools EC 9+6) from 18.2.7 to 19.2.2 on May 26. It was healthy when we upgraded it and remained so until last Thrusday (June 5) where, doing a `ceph -s`, I saw that there were 1 deep scrub and ~1000 scrubs late: 1 pgs not deep-scrubbed in time 1168 pgs not scrubbed in time Looking again Friday morning (~16 hours later), this number increased a lot: 27 pgs not deep-scrubbed in time 3252 pgs not scrubbed in time Checking our configuration on Friday, we found that osd_max_scrubs was set to 1 instead of using the new default since Reef which is 3 (probably a leftover of a config change after a problem 18 monts ago). We unset the specific value and this led to a reduction of these numbers in the next 24 hours (~2700 scrubs late) but since then (2 days) it remained stable initially and is now increasing slowly. This morning the situation is: 294 pgs not deep-scrubbed in time 3013 pgs not scrubbed in time My guess is that the real issue is the late scrubs that results in many OSD reaching the limit of 3 concurrent scrubs and that it has the consequences that some deep scrubs cannot run too (I've in mind that the limit applies both to shallow scubs and deep scrubs, am I right?). I checked the main cluster logs and didn't find any error or warning, related to OSDs like slow ops, slow requests...The only things we have spotted through our monitoring system is a dramatic decrease (~75%) of IOPS on each OSD server right after the 19.2.2 upgrade but it is not necessarily the sign of a problem. I guess it may in particular be a consequence of the new deep scrubs parameters, osd_shallow_scrub_chunk_min/max, which a probably intended to reduced the deep scrub IOPS load. The release notes for Squid also mention a change in osd_op_num_shards_hdd and osd_op_num_threads_per_shard_hdd, I don't know if they may also have impact. Up to now, no users has reported any issue so it seems to be a problem only with scrubs. I'm wondering where to start looking for an issue or anything related to 19.2.2 is already known. We increased the deep scrub interval from 10 to 14 days a few days before the upgrade (we saw that there was permanently 1 deep scrub late, a different PG all the time) and kept the standard 7 day interval for scrubs. Looking at the number of scrubs and deep scrubs per day, it doesn't look weird (see below). I guess that if we retart all OSDs we'll clear the problem but we'd like to understand what happened and be sure it is not something related to Squid, before upgrade our other production cluster. Any hint/advice will be highly appreciated. I took a snapshot of `ceph pg dump pgs_brief` regularly and I'll try to identify if there are some stucked scrubs and what are the OSDs involved but with 500 OSDs and 18 OSD servers, it may not be obvious... Best regards, Michel _Distribution of scrubs_ (first number is the number of scrubs during the day) (number increases on June 6 after setting osd_max_scrubs=3) 295 "2025-05-25 1578 "2025-05-26 300 "2025-05-27 392 "2025-05-28 578 "2025-05-29 707 "2025-05-30 611 "2025-05-31 819 "2025-06-01 679 "2025-06-02 724 "2025-06-03 698 "2025-06-04 726 "2025-06-05 1577 "2025-06-06 1393 "2025-06-07 1962 "2025-06-08 645 "2025-06-09 _Distribution of deep scrubs per day_ (number increases on June 6 after setting osd_max_scrubs=3 and starts to decrease again on June 8, when the number of late subs increases again, probably because we hit the limit of 3 scrubs per OSD) 22 "2025-05-12 63 "2025-05-13 101 "2025-05-14 127 "2025-05-15 173 "2025-05-16 238 "2025-05-17 305 "2025-05-18 387 "2025-05-19 450 "2025-05-20 564 "2025-05-21 675 "2025-05-22 716 "2025-05-23 871 "2025-05-24 1071 "2025-05-25 801 "2025-05-26 188 "2025-05-27 292 "2025-05-28 335 "2025-05-29 409 "2025-05-30 371 "2025-05-31 514 "2025-06-01 440 "2025-06-02 504 "2025-06-03 478 "2025-06-04 546 "2025-06-05 1132 "2025-06-06 1022 "2025-06-07 662 "2025-06-08 227 "2025-06-09

2 4

18.2.7 - IOPS issue
by Devender Singh 17 Jun '25

17 Jun '25

Dear All I upgrade my clusters from 18.2.4 & 18.2.6 to 18.2.7. 1. I started facing multiple disks went down.. 2. Fio test showing 5-35 IOPS on cephfs? $ fio --name=latency-test --ioengine=libaio --rw=randread --bs=4k --size=512M --numjobs=1 --iodepth=1 --direct=1 --runtime=60 --time_based --group_reporting latency-test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.28 Starting 1 process Jobs: 1 (f=1): [r(1)][100.0%][r=100KiB/s][r=25 IOPS][eta 00m:00s] latency-test: (groupid=0, jobs=1): err= 0: pid=3684386: Mon Jun 16 23:16:51 2025 read: IOPS=142, BW=572KiB/s (586kB/s)(33.5MiB/60001msec) slat (usec): min=46, max=250, avg=75.77, stdev=19.79 clat (usec): min=571, max=716047, avg=6913.78, stdev=33009.42 lat (usec): min=630, max=716107, avg=6989.89, stdev=33009.90 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 857], 10.00th=[ 914], 20.00th=[ 979], | 30.00th=[ 1037], 40.00th=[ 1074], 50.00th=[ 1123], 60.00th=[ 1172], | 70.00th=[ 1254], 80.00th=[ 1696], 90.00th=[ 4424], 95.00th=[ 22938], | 99.00th=[160433], 99.50th=[233833], 99.90th=[446694], 99.95th=[492831], | 99.99th=[717226] bw ( KiB/s): min= 8, max= 1824, per=100.00%, avg=581.15, stdev=438.28, samples=118 iops : min= 2, max= 456, avg=145.29, stdev=109.57, samples=118 lat (usec) : 750=0.99%, 1000=22.51% lat (msec) : 2=62.04%, 4=3.73%, 10=4.09%, 20=1.40%, 50=2.11% lat (msec) : 100=1.47%, 250=1.24%, 500=0.38%, 750=0.05% cpu : usr=0.12%, sys=1.18%, ctx=8922, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=8580,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=572KiB/s (586kB/s), 572KiB/s-572KiB/s (586kB/s-586kB/s), io=33.5MiB (35.1MB), run=60001-60001msec 3. RBD vis are even showing same number of IOS? Did anyone also face similar issue? 4. I have another version 19.2.2 where it shows almost 9000 IOPS… 5. On version 18.2.4 also facing same less numbers of IOPS. Any solution if someone can help.. Regards Dev

1 0

CephFS mirroring alternative?
by Ralf Schulze 16 Jun '25

16 Jun '25

Hi, we run two clusters with Squid 19.2.2 and would like to use CephFS mirroring. Unfortunately, the feature is still not working correctly and cannot be used. So we are looking for an alternative for the synchronization. One option would be to synchronize the snapshots to the second cluster with rsync. What alternative do you use for CephFS mirroring? What tools and processes do you use? Should CephFS mirroring be fixed? Do you know anything new about this? Thanks and best regards, Ralf

6 7

Re: v18.2.7 Reef released
by Erwin Bogaard 16 Jun '25

16 Jun '25

It looks like there is an issue with the package-based 18.2.7 release, when upgrading from 18.2.6 on el9. There seems to be a new (unfulfilled) dependency that prevents the packages from installing: Problem 1: cannot install the best update candidate for package librgw2-2:18.2.6-0.el9.x86_64 - problem with installed package librgw2-2:18.2.6-0.el9.x86_64 - nothing provides libcrypto.so.3(OPENSSL_3.4.0)(64bit) needed by librgw2-2:18.2.7-0.el9.x86_64 from ceph-reef This issue prevents almost all ceph packages from installing. 18.2.6 is installed and fully functional on this machine, so it seems something happened between 18.2.6 and 18.2.6. Hope someone can confirm this is happening at their severs as well and/or confirm or deny this is a packaging problem. Regards, Erwin

11 17

partitioning OSD Storage drive
by filip Mutterer 16 Jun '25

16 Jun '25

Is it possible to partition an OSD storage drive? As I only have 2 S-ATA ports it sounds like a better choice to use a small part of the two SSDs for the operating system and leave the rest to Ceph. Can I do this? I already have a setup like this with ZFS and it can have significant downsides. but we can live with slow and save data due to this combination.

3 2

radosgw enforce bucket encryption with sse-s3
by Boris 13 Jun '25

13 Jun '25

This is a follow up question to the sse-kms thread, because the KMS team is now working on the transit engine and we will POC with openbao. Is there a way to enforce the bucket encryption for every bucket? As how I understand it, I need to enable the bucket encryption after creating. If we could enforce that, I could a header check in the haproxy to make it transparent for the user. cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 2

Infiniband (RDMA) connectivity problems
by Jan Marek 12 Jun '25

12 Jun '25

Hello, we had sometimes problem with OSD in our all-flash NVMe CEPH cluster. Our cluster is made by cephadm, using Debian bookworm as a "basic" system and podman as a container solution. We are using two-ports Mellanox 100Gbps adapters: # lspci | grep Mellanox 43:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] 43:00.1 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] We are using TCP/IP for front-end communication and RDMA for internal communication. Sometimes (after cca 2-3 month of normal working) one OSD container from whole machine have a problem to connect to other OSD processes, but it is still connected to MON process, which goes to the situation, when it is "half-dead" - it cannot connects to other OSD, other OSD cannot connect to it, but it reports to the MON process, that it is still alive. We have experience, that this situation can resolve by restart container of this OSD. We have script, which is running every 5 minute and which searching in logs for this pattern: start_waiting_for_healthy If is this pattern found, we restart of container with appropriate OSD daemon. After restart this container, CEPH cluster goes quickly to HEALTHY state. I'm adding log, where this situation occured, up to restarting container. It was an osd.12 container with PID 3920171. In my opinion there is some problem to open infiniband connection - some problems in infiniband stack... :-( I'm open to you questions, although I cannot do some special operations, because it is our production cluster... Many thanks for this software :-) Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

3 4

When is osd_mclock_capacity_iops_hdd/ssd updated?
by Michel Jouvin 11 Jun '25

11 Jun '25

Hi, We have a production cluster homogeneous in terms of HDD devices and I'm a little bit surprised that the osd_mclock_capacity_iops_hdd computed for each OSD is so different (basically a x2 between the lowest and highest values). I understand that sometimes the OSD bench is perturbated by external factors leading to an inappropriate value and if it is too low, it may impact in particular deep scrub duration. The documentation [1] explains that the OSD bench is run at OSD initialization, which is, I guess, startup time. It also suggests that it can be run manually but I guess it makes no sense if the OSD is up because of the concurrent activity. Am I right? How is the bench command supposed to be run as I guess it doesn't work if the OSD is not up? Also, the documentation explains that you can define a value that you measured and seems to suggest that once defined, it will not be updated. Am I right? If yes, does it mean that once the automatic bench has determined a value the only way to update it is to delete it from the config and restart the OSD (if you want the automatic bench to update/redefine it)? Best regards, Michel [1] https://docs.ceph.com/en/squid/rados/configuration/mclock-config-ref/

4 4

pool create via cli ignores size parameter
by Eugen Block 11 Jun '25

11 Jun '25

Hi, without having checked the tracker, does anyone have an explanation why the size parameter is not applied when creating a pool via CLI? According to the help output for 'ceph osd pool create -h' you can specify expected_num_objects (btw. I don't understand what impact that has, all I could find was related to filestore OSDs and PG splitting) and then the size: profile>] [<rule>] [<expected_num_objects:int>] [<size:int>] [<pg_num_min:int>] [<pg_num_max: If I specify both parameters, the pool gets its default size according to osd_pool_default_size (this is a single node cluster, hence default size 2): soc9-ceph:~ # ceph osd pool create temp5 4 replicated replicated_rule 0 4 pool 4 'temp5' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 4 pgp_num 4 autoscale_mode off last_change 260 flags hashpspool stripe_width 0 read_balance_score 1.00 I get the same result on a larger test cluster with multiple hosts. Of course I know that I could just set the size afterwards, but is the command broken? Has anyone looked into this? Thanks! Eugen

1 3

Limits on the number of cephx keys used
by James Tocknell 11 Jun '25

11 Jun '25

Hi All As far as I can see, there is no guidance on the number of cephx keys that can be in use at one time. Is there a number at which ceph becomes much slower e.g. 100, 10000, 1000000? I'm wondering how best to manage keys across many clients (let's say 1000s for now), most of which won't actually be connected at the same time. Regards James

4 4

rgw + LDAP
by Albert Shih 11 Jun '25

11 Jun '25

Hi, I'm currently running ceph 18.2.7 and I try to connect my RGW to my LDAP After many hours to battle with that I end up to turn on every debug flag I can find. It seem the RGW try to bind anonymously to my ldap server, here the log on my ldap server (openldap) Jun 10 16:32:02 ldaps2-m2 slapd[453]: conn=836633 op=1 SRCH base="dc=obspm,dc=fr" scope=2 deref=0 filter="(&(&(objectClass=inetOrgPerson)(memberOf=cn=s3storage,ou=groups,ou=services_centraux,dc=obspm,dc=fr))(uid=jas))" Jun 10 16:32:02 ldaps2-m2 slapd[453]: conn=836633 op=1 SRCH attr=uid Jun 10 16:32:02 ldaps2-m2 slapd[453]: ==> limits_get: conn=836633 op=1 self="[anonymous]" this="dc=obspm,dc=fr" We don't want to allow the anonymous bind here. I set ceph config set client.rgw rgw_ldap_binddn "cn=s3storage,ou=dsa,ou=services_centraux,dc=obspm,dc=fr" ceph config set client.rgw rgw_ldap_secret "/etc/ceph/ldappw.txt" ceph config set client.rgw rgw_ldap_searchdn "dc=obspm,dc=fr" I check the file /etc/ceph/ldappw.txt, give it to root:root, to ceph:ceph, change the right 600/644 but whatever I does it'still trying to bind anonymously. I can even find the string s3storage,ou=dsa inside the debug log of my ldap server. It's like rgw don't even try ton bind with the binddn I give it to him I try to run a shell inside the podman container but was unable to find where the rgw config is «instanciated» inside the container (to see what it get). Any clue ? Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: mar. 10 juin 2025 16:36:11 CEST

3 6

bucket notification for kafka with different sasl-mechanism
by farhad kh 11 Jun '25

11 Jun '25

hi , for bucket notification anybody have experience use kafka with encryption? i try config topic with this configuration { "topics": [ { "owner": "farhad", "name": "sre-s3", "dest": { "push_endpoint": "kafka://sre_fkhedrian:d2Q9BAYN9RVj@10.248.17.6:9092", "push_endpoint_args": "Version=2010-03-31&kafka-ack-level=broker&sasl_mechanism=SCRAM-SHA-256&security_protocol=SASL_PLAINTEXT&verify-ssl=False", "push_endpoint_topic": "sre-s3", "stored_secret": true, "persistent": true, "persistent_queue": ":sre-s3", "time_to_live": "None", "max_retries": "None", "retry_sleep_duration": "None" }, "arn": "arn:aws:sns:podspace::sre-s3", "opaqueData": "", "policy": "", "subscribed_buckets": [] } ] } but get this error in kafka [2025-06-09 12:58:12,087] INFO [SocketServer listenerType=BROKER, nodeId=1] Failed authentication with /10.248.35.142 (channelId=10.248.17.6:9092-10.248.35.142:40518-1-1055) (Unsupported SASL mechanism PLAIN) (org.apache.kafka.common.network.Selector), of course i have use sasl_mechanism=SCRAM-SHA-256 in kafka for other program and test with a python script i sure this working. where is problem ?how can i fix this ?

1 0

Change netmask of CIDR
by Mahdi Noorbala 10 Jun '25

10 Jun '25

Hello everybody We have a ceph cluster in production. It's used for openstack as rbd image. Now we want to change netmask from /22 to /21. What do you think? Is there any problem in this change? Is there any concern about this change? Thanks.

2 5

Announcing go-ceph v0.34.0
by Anoop C S 10 Jun '25

10 Jun '25

Hi, I'm happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.34.0 Highlights for this release includes the introduction of new smb admin API sub-package for handling SMB shares. More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Anoop C S.

1 0

Configure RGW without ceph.conf
by Albert Shih 10 Jun '25

10 Jun '25

Hi, I'm trying to configure my rgw with keycloak, according to the documentation I need to modify /etc/ceph/ceph.conf Is they are anyway to configure the RGW with keycloak WITHOUT modifying this file ? Like using ceph config set or event in /etc/ceph/rgw.conf (or whatever) I ask that because my ceph.conf are manage with some deployment tools (puppet in my case). Regards JAS -- Albert SHIH 🦫 🐸 France Heure locale/Local time: lun. 09 juin 2025 22:37:23 CEST

3 5

mds daemon damaged
by bbk＠nocloud.ch 10 Jun '25

10 Jun '25

Hi, after running the cluster for years, it is the first time i have a problem that seems to require some expert knowlege ;-) I had some network problems, which i suspect that lead to the damaged mds service. Just as a side note: i upgraded to 17.2.2 but it had been running for about a hour till the network outage, so i don't think this was related. However, now i wonder how to go from here. ``` # ceph -s cluster: health: HEALTH_ERR 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged services: mon: 5 daemons, quorum ceph3,ceph4,ceph1,ceph5,ceph2 (age 46m) mgr: ceph5.zmvagf(active, since 5h), standbys: ceph2.defhpj mds: 0/1 daemons up, 2 standby osd: 10 osds: 10 up (since 46m), 10 in (since 2h) data: volumes: 0/1 healthy, 1 recovering; 1 damaged pools: 4 pools, 193 pgs objects: 5.42M objects, 13 TiB usage: 26 TiB used, 11 TiB / 37 TiB avail pgs: 193 active+clean io: client: 200 KiB/s rd, 84 KiB/s wr, 199 op/s rd, 169 op/s wr ``` ``` # ceph health detail HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged [WRN] FS_DEGRADED: 1 filesystem is degraded fs cephfs is degraded [ERR] MDS_ALL_DOWN: 1 filesystem is offline fs cephfs is offline because no MDS is active for it. [ERR] MDS_DAMAGE: 1 mds daemon damaged fs cephfs mds.0 is damaged ``` ``` # ceph fs status cephfs - 0 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed POOL TYPE USED AVAIL cephfs.cephfs.meta metadata 1293M 1954G cephfs.cephfs.data data 2797G 1954G ecpool data 23.2T 2931G STANDBY MDS cephfs.ceph1.yzqmuo cephfs.ceph3.vmieie MDS version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable) ``` According to * https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ i came up with the following procedure. Deny all reconnect to clients. ```sh ceph config set mds mds_deny_all_reconnect true ``` Deny new clients. ```sh ceph fs set cephfs refuse_client_session true ``` Backup the current journal: ```sh mkdir /root/mds-damaged cephfs-journal-tool --rank=cephfs:0 journal export /root/mds-damaged/backup-rank0.bin ``` Recover file metadata and discard what is damaged: ```sh cephfs-journal-tool --rank=cephfs:0 event recover_dentries summary ``` Truncate any journal that is corrupt or that an MDS cannot replay: ```sh cephfs-journal-tool --rank=cephfs:0 journal reset --yes-i-really-really-mean-it ``` Reset the SessionMap: ```sh cephfs-table-tool all reset session ``` Does this make sense? Yours, bbk

2 3

Ceph Orchestrator error
by Iztok Gregori 10 Jun '25

10 Jun '25

Hi to all! I'm trying to setup a RGW service on my cluster and I want that the SSL certificates are create automatically, so I wrote the following spec file: # rgw.yaml service_type: rgw service_id: rgw.foo placement: label: rgw count_per_host: 1 networks: - 172.16.10.0/24 spec: rgw_realm: 'foo' rgw_zone: 'fool' rgw_zonegroup: 'fool_group' ssl: true generate_cert: true rgw_frontend_port: 4443 wildcard_enabled: true zonegroup_hostnames: - "fool.foo.com" extra_container_args: - "--stop-timeout=120" config: rgw_exit_timeout_secs: "120" rgw_graceful_stop: true When I apply it gives me the following error: # ceph orch apply -i rgw.yaml Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument 'generate_cert' Am I missing something obvious regarding the "generate_cert" part? The cluster is a fresh install of Squid (19.2.2). Any help is appreciated! Regards Iztok -- Iztok Gregori ICT Systems and Services Elettra - Sincrotrone Trieste S.C.p.A. Telephone: +39 040 3758948 http://www.elettra.eu

2 3

Confuse by rgw and certificate
by Albert Shih 10 Jun '25

10 Jun '25

Hi everyone. I'm trying to configure a RGW for S3. I'm currently running reef 18.2.7 I was able to make the rgw working (= the service is up and listen on the correct port) without ssl. I'm trying to configure the ssl port and I didn't find the correct syntaxe for that. I create a yaml file with something like spec: rgw_frontend_port: 8080 zone_endpoints: https://host1:8080, https://host2:8080,etc. ssl: true rgw_frontend_ssl_certificate: | -----BEGIN RSA PRIVATE KEY----- .... -----END RSA PRIVATE KEY----- -----BEGIN CERTIFICATE----- ..... -----END CERTIFICATE----- or with the all chain of CA spec: rgw_frontend_port: 8080 zone_endpoints: https://host1:8080, https://host2:8080,etc. ssl: true rgw_frontend_ssl_certificate: | -----BEGIN RSA PRIVATE KEY----- .... -----END RSA PRIVATE KEY----- -----BEGIN CERTIFICATE----- ..... -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- ..... -----END CERTIFICATE----- ..... -----BEGIN CERTIFICATE----- ..... -----END CERTIFICATE----- both not working the journalctl say ssl_private_key was not found: rgw/cert/obspm/meudon.key I find somewhere on the net this syntaxe spec: rgw_frontend_port: 8080 zone_endpoints: https://host1:8080, https://host2:8080,etc. ssl: true ssl_private_key: | -----BEGIN RSA PRIVATE KEY----- .... -----END RSA PRIVATE KEY----- ssl_certificate: | -----BEGIN CERTIFICATE----- ..... -----END CERTIFICATE----- and I got Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument 'ssl_private_key' Any clue ? Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: ven. 06 juin 2025 10:47:08 CEST

3 5

Discrepancy between users in Ceph dashboard Bucket users and radosgw-admin user list
by Al 05 Jun '25

05 Jun '25

I've been playing with installing rook-ceph in a Rancher managed k8s cluster for the past week or so and while it does work, some things aren't working as expected. In particular I'm trying to use the CLI to add users and buckets during cluster creation, to integrate with my IaC/IaaS scripts, and am running into a confusing issue. I can use the Ceph dashboard to create users and buckets, and then use the aws s3api commands to manipulate the buckets and all works well; however the documentation says that I can add bucket users with radosgw-admin, but when I do this, the users I create do not show up anywhere I can find in the Ceph dashboard, and they cannot manipulate buckets via the aws s3api commands. I'm sitting on a fresh install of rook-orchestrator and ceph-rook-cluster now via helm and the dashboard shows a single user in the Buckets->Users area named 'dashboard-admin' while 'radosgw-admin user list' returns an empty array. radosgw-admin zone list and zonegroups list show two zones and two zonegroups; one named 'ceph-objectstore' and one named 'default'. I'm not specifically trying to use zones or zonegroups but I thought I would mention this since it is relevant. Both show in the Object-Multi site page in the dash as well, though there's an extra "something" above the ceph-objectstore tree that has a balance scale icon; i don't know what these represent as there is no legend on the page. I haven't tried to add a user in this current config since I'd like to wait for some ideas from the group, but the commands I've been trying are: TOOLS_POD=$(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[*].metadata.name}') kubectl -n rook-ceph exec -it $TOOLS_POD -- radosgw-admin user create --uid=testuser --display-name="Test User" --access=full The documentation indicates that this is how you add bucket users to the cluster. TIA for any assistance.

1 0

Put Bucket Logging | Method Not Allowed
by Wissem MIMOUNA - Ceph Users 05 Jun '25

05 Jun '25

Hi Ceph Users, I'm trying to test ceph radosgw bucket logging feature , but I keep getting "MethodNotAllowed" Error, though I reinstalled the radosgw from scratch (I recreated a new realm-zonegroup and zone). Our ceph cluster is ceph squid v 19.2.2 on debian 12. I followed the procedure in the following url to create a bucket policy (https://docs.ceph.com/en/latest/radosgw/bucket_logging/) I tried both , with normal users and with account users. Each time the the user/account who created the both bucket ( source bucket and logging destination bucket) who try to enable logging but failed. The command which keep failing is (I tried with boto3 and with aws cli but the same error)==> 3cmd accesslog s3://myBucket --access-logging-target-prefix=logsBucket/prefix Any advices are welcome! Thanks Wissem M

3 2

MDS Repeatedly Crashing/Restarting - Unable to get CephFS Active
by Kasper Rasmussen 05 Jun '25

05 Jun '25

Ceph Version: 18.2.7 I've just migrated to cephadm, and upgrade from pacific to reef 18.2.7 last week. All successful except some minor issues with BlueFS Spillover Today the MDS of a specific fs refuse to start, and the ceph orch ps shows the daemons with status "error". I have three other cephfs that still works(though I haven't tested if they can fail over.) I've restartet the MDSs - No luck (the selected MDS just start/crash in a loop until it gives up) I've deployed 2 new MDSs - No luck same issue In all scenarios I see in ceph fs status, that a MDS is chosen. FS status goes to "replay" or "replay(laggy)" On the host with the MDS I see the MDS container just crashes after way less than 5 mins.. And status reported by ceph orch ps is error. (btw - mds_beacon_grace has been set to 360) I've managed to get a good 500 lines of log out with info like this: << ----------------- LOG EXAMPLE START ----------------- >> -7> 2025-05-19T16:05:02.840+0000 7f6739bb8640 10 monclient: _check_auth_tickets -6> 2025-05-19T16:05:02.840+0000 7f6739bb8640 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2025-05-19T16:04:32.845551+0000) -5> 2025-05-19T16:05:02.860+0000 7f673e3c1640 10 monclient: get_auth_request con 0x5616e9616c00 auth_method 0 -4> 2025-05-19T16:05:02.916+0000 7f673dbc0640 10 monclient: get_auth_request con 0x5616e7422800 auth_method 0 -3> 2025-05-19T16:05:02.968+0000 7f673d3bf640 10 monclient: get_auth_request con 0x5616f5eac800 auth_method 0 -2> 2025-05-19T16:05:02.972+0000 7f6736bb2640 2 mds.0.cache Memory usage: total 574800, rss 343772, heap 207124, baseline 182548, 0 / 7535 inodes have caps, 0 caps, 0 caps per inode -1> 2025-05-19T16:05:03.676+0000 7f67333ab640 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.7/rpm/el9/BUILD/ceph-18.2.7/src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]' thread 7f67333ab640 time 2025-05-19T16:05:03.680495+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.7/rpm/el9/BUILD/ceph-18.2.7/src/include/interval_set.h: 568: FAILED ceph_assert(p->first <= start) ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11e) [0x7f67406e6d2c] 2: /usr/lib64/ceph/libceph-common.so.2(+0x16beeb) [0x7f67406e6eeb] 3: /usr/bin/ceph-mds(+0x1f16fe) [0x5616e04d46fe] 4: /usr/bin/ceph-mds(+0x1f1745) [0x5616e04d4745] 5: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4bdc) [0x5616e0709a4c] 6: (EUpdate::replay(MDSRank*)+0x5d) [0x5616e0711afd] 7: (MDLog::_replay_thread()+0x75e) [0x5616e06bc02e] 8: /usr/bin/ceph-mds(+0x1404b1) [0x5616e04234b1] 9: /lib64/libc.so.6(+0x8a21a) [0x7f674009721a] 10: clone() 0> 2025-05-19T16:05:03.676+0000 7f67333ab640 -1 *** Caught signal (Aborted) ** in thread 7f67333ab640 thread_name:mds-log-replay ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) 1: /lib64/libc.so.6(+0x3ebf0) [0x7f674004bbf0] 2: /lib64/libc.so.6(+0x8bf5c) [0x7f6740098f5c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x178) [0x7f67406e6d86] 6: /usr/lib64/ceph/libceph-common.so.2(+0x16beeb) [0x7f67406e6eeb] 7: /usr/bin/ceph-mds(+0x1f16fe) [0x5616e04d46fe] 8: /usr/bin/ceph-mds(+0x1f1745) [0x5616e04d4745] 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x4bdc) [0x5616e0709a4c] 10: (EUpdate::replay(MDSRank*)+0x5d) [0x5616e0711afd] 11: (MDLog::_replay_thread()+0x75e) [0x5616e06bc02e] 12: /usr/bin/ceph-mds(+0x1404b1) [0x5616e04234b1] 13: /lib64/libc.so.6(+0x8a21a) [0x7f674009721a] 14: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. << ----------------- LOG EXAMPLE END ----------------- >> But to be honest, out of all those lines, I don't know what to provide (all +500 might be a bit to much) I really need this FS back online, so help will be very much appreciated

4 14

Help in upgrading CEPH
by Sergio Rabellino 05 Jun '25

05 Jun '25

Dear list, we are upgrading our ceph infrastructure from mimic to octopus (please be kind, we known that we are working with "old" tools, but these ceph releases are tied to our openstack installation needs) and _*all*_ the ceph actors (mon/mgr/osd/rgw - no mds as we do not serve filesystem) were upgraded fine as follow: > # ceph -v > ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) > octopus (stable) We're using a ubuntu/juju orchestrator for managing ceph (and openstack too). It seems all ok, but if I ask a mon to dump the sessions, we get that_all of them_ are on luminous: > "con_features": 4540138292840890367, > "con_features_hex": "3f01cfb8ffedffff", > "con_features_release": "luminous", We found this oddity when we tried to unset the "insecure global_id reclaim" flag and all things got broken, so we had to reactivate the flag. All ceph network layers are "closed", so we're not so urged to remove the flag, but we would like to understand if this is a known problem or an error done by us. Any hints ? thanks in advance. -- ing. Sergio Rabellino Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino <http://www.di.unito.it>

5 16

Write throughput increasing on CephFS metadata pool
by Adrien Georget 05 Jun '25

05 Jun '25

Hi, We have a CephFS Quincy cluster (17.2.7) used by Openshift for PVC provisioning (and lot of snapshots) with ceph-csi driver. On the metadata pool, we observed a continuous increase of the write throughput activity from 10MB/s just after a restart of the MDS to +200MB/s after 2 weeks. We first thought it was related to the snapshots deletion activity (lot of creations and deletions every day) because we saw the same kind of increase for the number of Strays. We tried to evaluate strays using the recursive scrub [1], Strays decrease from 450K to 350K but no impact on the metadata write throughput. Only a MDS restart makes the write throughput back to normal. I saw this very familiar issue https://tracker.ceph.com/issues/53542 and I'm wondering if this has been fixed in Quincy? I tried to increase "mds_log_events_per_segment" and "mds_log_max_segments" but nothing helped. As I answered in the tracker, I also observed large 4M objects in objecter_requests. Anything we can do to prevent this and avoid restarting the MDS every week? If we don't do that, PVC operations stay blocked on Openshift side. Cheers, Adrien [1] https://docs.ceph.com/en/quincy/cephfs/scrub/#evaluate-strays-using-recursi… The MDS perf dump if it can helps : { "AsyncMessenger::Worker-0": { "msgr_recv_messages": 452889868, "msgr_send_messages": 210932005, "msgr_recv_bytes": 381554341884, "msgr_send_bytes": 31897539519042, "msgr_created_connections": 5392, "msgr_active_connections": 213, "msgr_running_total_time": 121964.941943159, "msgr_running_send_time": 85193.342708858, "msgr_running_recv_time": 40274.249544605, "msgr_running_fast_dispatch_time": 15899.702855460, "msgr_send_messages_queue_lat": { "avgcount": 210932003, "sum": 140532.121408536, "avgtime": 0.000666243 }, "msgr_handle_ack_lat": { "avgcount": 110313532, "sum": 120.513184629, "avgtime": 0.000001092 } }, "AsyncMessenger::Worker-1": { "msgr_recv_messages": 393334601, "msgr_send_messages": 170841678, "msgr_recv_bytes": 307649876710, "msgr_send_bytes": 23017109865662, "msgr_created_connections": 12712, "msgr_active_connections": 218, "msgr_running_total_time": 93136.265793790, "msgr_running_send_time": 63604.143090523, "msgr_running_recv_time": 30881.136884349, "msgr_running_fast_dispatch_time": 12763.126679902, "msgr_send_messages_queue_lat": { "avgcount": 170841678, "sum": 153574.265735747, "avgtime": 0.000898927 }, "msgr_handle_ack_lat": { "avgcount": 81594240, "sum": 65.480936528, "avgtime": 0.000000802 } }, "AsyncMessenger::Worker-2": { "msgr_recv_messages": 481956104, "msgr_send_messages": 301348856, "msgr_recv_bytes": 349838769013, "msgr_send_bytes": 26783354654792, "msgr_created_connections": 34394, "msgr_active_connections": 215, "msgr_running_total_time": 108807.807447203, "msgr_running_send_time": 74577.666991790, "msgr_running_recv_time": 41392.667149426, "msgr_running_fast_dispatch_time": 13971.142479134, "msgr_send_messages_queue_lat": { "avgcount": 301348847, "sum": 138655.862052108, "avgtime": 0.000460117 }, "msgr_handle_ack_lat": { "avgcount": 156375610, "sum": 144.168097813, "avgtime": 0.000000921 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "queue_len": 0, "complete_latency": { "avgcount": 121564865, "sum": 160601.142778677, "avgtime": 0.001321114 } }, "finisher-PurgeQueue": { "queue_len": 0, "complete_latency": { "avgcount": 27338289, "sum": 8785.146872773, "avgtime": 0.000321349 } }, "mds": { "request": 210298249, "reply": 210297956, "reply_latency": { "avgcount": 210297956, "sum": 1148337.019419176, "avgtime": 0.005460523 }, "slow_reply": 15, "forward": 0, "dir_fetch": 49843551, "dir_commit": 65281430, "dir_split": 6, "dir_merge": 17, "inodes": 5113817, "inodes_top": 2080514, "inodes_bottom": 2263929, "inodes_pin_tail": 769374, "inodes_pinned": 2141653, "inodes_expired": 538454667, "inodes_with_caps": 413837, "caps": 457168, "subtrees": 2, "traverse": 262113637, "traverse_hit": 245551660, "traverse_forward": 0, "traverse_discover": 0, "traverse_dir_fetch": 1642726, "traverse_remote_ino": 0, "traverse_lock": 140893, "load_cent": 392174, "q": 0, "exported": 0, "exported_inodes": 0, "imported": 0, "imported_inodes": 0, "openino_dir_fetch": 407633, "openino_backtrace_fetch": 7011755, "openino_peer_discover": 0, "root_rfiles": 11618539, "root_rbytes": 11106041675168, "root_rsnaps": 7576, "scrub_backtrace_fetch": 7009788, "scrub_set_tag": 0, "scrub_backtrace_repaired": 0, "scrub_inotable_repaired": 0, "scrub_dir_inodes": 1526050, "scrub_dir_base_inodes": 3, "scrub_dirfrag_rstats": 1526047, "scrub_file_inodes": 5483733, "handle_inode_file_caps": 0, "ceph_cap_op_revoke": 5076186, "ceph_cap_op_grant": 60192925, "ceph_cap_op_trunc": 658164, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_flush_ack": 6, "handle_client_caps": 70349707, "handle_client_caps_dirty": 21547109, "handle_client_cap_release": 11468771, "process_request_cap_release": 83631146 }, "mds_cache": { "num_strays": 347989, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "strays_created": 19259955, "strays_enqueued": 20275628, "strays_reintegrated": 469, "strays_migrated": 0, "num_recovering_processing": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "recovery_started": 484, "recovery_completed": 484, "ireq_enqueue_scrub": 2, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 23, "ireq_fragstats": 0, "ireq_inodestats": 0 }, "mds_log": { "evadd": 148604638, "evex": 148687537, "evtrm": 148686957, "ev": 4640, "evexg": 0, "evexd": 580, "segadd": 5143165, "segex": 5143142, "segtrm": 5143128, "seg": 166, "segexg": 0, "segexd": 14, "expos": 2655552712972056, "wrpos": 2655556750517207, "rdpos": 2574717722176006, "jlat": { "avgcount": 23948614, "sum": 1026045.477642289, "avgtime": 0.042843626 }, "replayed": 86959 }, "mds_mem": { "ino": 5091233, "ino+": 537662754, "ino-": 532571521, "dir": 385511, "dir+": 48473070, "dir-": 48087559, "dn": 5114534, "dn+": 564239550, "dn-": 559125016, "cap": 457198, "cap+": 296065469, "cap-": 295608271, "rss": 21947020, "heap": 223516 }, "mds_server": { "dispatch_client_request": 265192774, "dispatch_server_request": 0, "handle_client_request": 210298249, "handle_client_session": 44456292, "handle_peer_request": 0, "req_create_latency": { "avgcount": 19045215, "sum": 243874.341572325, "avgtime": 0.012805019 }, "req_getattr_latency": { "avgcount": 13244393, "sum": 122372.637049784, "avgtime": 0.009239580 }, "req_getfilelock_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_link_latency": { "avgcount": 108, "sum": 0.089209875, "avgtime": 0.000826017 }, "req_lookup_latency": { "avgcount": 42591076, "sum": 61153.611919024, "avgtime": 0.001435831 }, "req_lookuphash_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_lookupino_latency": { "avgcount": 6119, "sum": 12.051986148, "avgtime": 0.001969600 }, "req_lookupname_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_lookupparent_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_lookupsnap_latency": { "avgcount": 11789, "sum": 1.916955778, "avgtime": 0.000162605 }, "req_lssnap_latency": { "avgcount": 174999, "sum": 59.628852495, "avgtime": 0.000340738 }, "req_mkdir_latency": { "avgcount": 386018, "sum": 5323.027974258, "avgtime": 0.013789584 }, "req_mknod_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_mksnap_latency": { "avgcount": 11784, "sum": 4342.774640033, "avgtime": 0.368531452 }, "req_open_latency": { "avgcount": 1719118, "sum": 2957.961208770, "avgtime": 0.001720627 }, "req_readdir_latency": { "avgcount": 49303877, "sum": 153658.488737674, "avgtime": 0.003116559 }, "req_rename_latency": { "avgcount": 1021332, "sum": 4173.200914205, "avgtime": 0.004086037 }, "req_renamesnap_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_rmdir_latency": { "avgcount": 355348, "sum": 6609.629921457, "avgtime": 0.018600442 }, "req_rmsnap_latency": { "avgcount": 11476, "sum": 4647.171801363, "avgtime": 0.404947002 }, "req_rmxattr_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_setattr_latency": { "avgcount": 951092, "sum": 60918.695526648, "avgtime": 0.064051317 }, "req_setdirlayout_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_setfilelock_latency": { "avgcount": 42236246, "sum": 87407.285106607, "avgtime": 0.002069485 }, "req_setlayout_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 }, "req_setxattr_latency": { "avgcount": 20450454, "sum": 351779.625311317, "avgtime": 0.017201555 }, "req_symlink_latency": { "avgcount": 2, "sum": 0.002286490, "avgtime": 0.001143245 }, "req_unlink_latency": { "avgcount": 18777510, "sum": 39044.878444925, "avgtime": 0.002079342 }, "cap_revoke_eviction": 0, "cap_acquisition_throttle": 0, "req_getvxattr_latency": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "mds_sessions": { "session_count": 545, "session_add": 26598, "session_remove": 26053, "sessions_open": 545, "sessions_stale": 0, "total_load": 11076, "average_load": 20, "avg_session_uptime": 37409721, "mdthresh_evicted": 0 }, "mempool": { "bloom_filter_bytes": 10309112, "bloom_filter_items": 10309112, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_Buffer_bytes": 0, "bluestore_Buffer_items": 0, "bluestore_Extent_bytes": 0, "bluestore_Extent_items": 0, "bluestore_Blob_bytes": 0, "bluestore_Blob_items": 0, "bluestore_SharedBlob_bytes": 0, "bluestore_SharedBlob_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_items": 0, "bluefs_bytes": 0, "bluefs_items": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "buffer_anon_bytes": 284731060, "buffer_anon_items": 5959770, "buffer_meta_bytes": 616, "buffer_meta_items": 7, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 81480, "osdmap_items": 2396, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "mds_co_bytes": 16277298996, "mds_co_items": 260156681, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "op_active": 11, "op_laggy": 0, "op_send": 253406063, "op_send_bytes": 81014361089067, "op_resend": 0, "op_reply": 253406052, "oplen_avg": { "avgcount": 253406063, "sum": 451959672 }, "op": 253406063, "op_r": 58269533, "op_w": 195136530, "op_rmw": 0, "op_pg": 0, "osdop_stat": 64974069, "osdop_create": 11505623, "osdop_read": 1275145, "osdop_write": 34882891, "osdop_writefull": 1169760, "osdop_writesame": 0, "osdop_append": 0, "osdop_zero": 2, "osdop_truncate": 4, "osdop_delete": 79836916, "osdop_mapext": 0, "osdop_sparse_read": 0, "osdop_clonerange": 0, "osdop_getxattr": 56700621, "osdop_setxattr": 17474636, "osdop_cmpxattr": 0, "osdop_rmxattr": 0, "osdop_resetxattrs": 0, "osdop_call": 0, "osdop_watch": 0, "osdop_notify": 0, "osdop_src_cmpxattr": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_other": 683839, "linger_active": 0, "linger_send": 0, "linger_resend": 0, "linger_ping": 0, "poolop_active": 0, "poolop_send": 0, "poolop_resend": 0, "poolstat_active": 0, "poolstat_send": 0, "poolstat_resend": 0, "statfs_active": 0, "statfs_send": 0, "statfs_resend": 0, "command_active": 0, "command_send": 0, "command_resend": 0, "map_epoch": 685274, "map_full": 0, "map_inc": 19880, "osd_sessions": 96, "osd_session_open": 96, "osd_session_close": 0, "osd_laggy": 0, "omap_wr": 78737471, "omap_rd": 99825478, "omap_del": 4893217 }, "oft": { "omap_total_objs": 4, "omap_total_kv_pairs": 105334, "omap_total_updates": 80286291, "omap_total_removes": 72648222 }, "purge_queue": { "pq_executing_ops": 0, "pq_executing_ops_high_water": 1559, "pq_executing": 0, "pq_executing_high_water": 64, "pq_executed": 20275628, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "val": 0, "max": 104857600, "get_started": 0, "get": 1328180571, "get_sum": 935681276002, "get_or_fail_fail": 0, "get_or_fail_success": 1328180571, "take": 0, "take_sum": 0, "put": 1328180571, "put_sum": 935681276002, "wait": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "throttle-objecter_bytes": { "val": 21734389, "max": 104857600, "get_started": 0, "get": 0, "get_sum": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "take": 253406063, "take_sum": 81151894362586, "put": 253406052, "put_sum": 81151872628197, "wait": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "throttle-objecter_ops": { "val": 11, "max": 1024, "get_started": 0, "get": 0, "get_sum": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "take": 253406063, "take_sum": 253406063, "put": 253406052, "put_sum": 253406052, "wait": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "throttle-write_buf_throttle": { "val": 0, "max": 3758096384, "get_started": 0, "get": 20275628, "get_sum": 2230259884, "get_or_fail_fail": 0, "get_or_fail_success": 20275628, "take": 0, "take_sum": 0, "put": 1242717, "put_sum": 2230259884, "wait": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } }, "throttle-write_buf_throttle-0x56401344c0a0": { "val": 0, "max": 3758096384, "get_started": 0, "get": 148604638, "get_sum": 80839025768070, "get_or_fail_fail": 0, "get_or_fail_success": 148604638, "take": 0, "take_sum": 0, "put": 23948617, "put_sum": 80839025768070, "wait": { "avgcount": 0, "sum": 0.000000000, "avgtime": 0.000000000 } } }

2 2

radosgw force usage of sse-kms?
by Boris 05 Jun '25

05 Jun '25

Hi, I am in a project where the requirement is: - All data MUST be encrypted at rest - At least one key per customer - The backend only supports the vault KV as secrect engine, so SSE-s3 is not an option The idea is, that we create a key for the customer and tell them in the panel that they need to use it and how it will work. But how to I prevent the user from uploading unecrypted objects? Do I check for a header in the proxy and return a <h1>uhuhuh, you didn't say the magic word!</h1> when there specific header is missing? And if this is the way, is there a shema I need to stick to? - Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 3

Ceph S3 Performance: HDDs for data + SSDs for metadata – Best practice?
by Ml Ml 04 Jun '25

04 Jun '25

Hi everyone, I'm currently planning a Ceph-based S3 storage backend, with a strong focus on balancing performance and cost (EUR per TB). Fully SSD-based setups seem too expensive for my use case, while HDD-only configurations might be too slow. My current idea is to use a hybrid approach: - HDDs for bluestore_block (bulk data) - Enterprise SSDs for bluestore_block.db and bluestore_block.wal (metadata and journal) The main client of this S3 backend will be Veeam, which, as far as I can tell, uses larger block sizes. Since this results in fewer random writes and mostly sequential I/O, I assume this kind of workload can work well with HDDs — as long as metadata is fast (hence SSDs for DB/WAL). Am I on the right track with this assumption? - Do you have any benchmark results (real-world or synthetic) comparing HDD-only vs Hybrid setups for object workloads like Veeam? That would help a lot to quantify the performance difference. - What SSD-to-HDD ratio would you recommend? (I’ve seen 1 SSD per 3–5 HDDs) Any advice or confirmation from those running similar setups would be much appreciated! So far i found: https://www.hyperscalers.com/image/catalog/00-Products/Storage%20Servers/Wh… => which mentions Ceph Verstion 0.94.5 and the performance boot does not seem too big here And here: https://www.ambedded.com.tw/en/use-case/use-case-08.html they dont mention such a Hybrid Setup Thanks a lot in advance! Best regards, Michael

4 3

radosgw-admin bucket radoslist returns 1.8 billion entries for a bucket with 17m objects
by Boris 04 Jun '25

04 Jun '25

Hi, I am in the process of checking orphan objects and the radoslist list grep over 300GB while the rados ls list was only 50GB. After some investigation I identified a saw that one bucket fell out with a lot of objects in the radoslist part. I did some sorting on the output file and just used the marker part to check if something is wrong: These are the last two lines after sorting: 16145185 ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297644274.205 1882589930 ff7a8b0c-07e6-463a-861b-78f0adeba8ad.1805769661.3772 The top bucket has roughtly the same amount ob objects in the index, but the other bucket only hast 17million object in the index. I stopped the search for orphan objects and pulled a fresh `radosgw-admin bucket radoslist --bucket BUCKET > file`. Then I sorted that list and ran a uniq -c against it and sorted that list again. The unique list ist now 34million objects long and there are 32026 entries that appear more than once and there are entries that 68849 times. 68849 ff7a8b0c-07e6-463a-861b-78f0adeba8ad.1805769661.3772__shadow_server/download/XXX/YYYY.2~k9UuEdtUkAHkikXz1ONc8UzgwKryYco.6_1 How can I clean that up? Or is this just a bug in the radosgw-admin bucket radoslist? The bucket has 401 shards and got "versioned: true" and "versioning_enabled: true" -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 2

Spurious CephFS backtrace objects in non-default pool
by Hector Martin 04 Jun '25

04 Jun '25

Hi, I just noticed an interesting behavior that I didn't find documented anywhere. Normally, CephFS stores backtraces in the both default data pool and the actual data pool a file's layout points to (if different): $ for pool in cephfs2_data cephfs2_data_hr3 cephfs2_data_hec5.4; do echo "# Pool: $pool" rados -p $pool listxattr 100032d5676.00000000 done # Pool: cephfs2_data parent # Pool: cephfs2_data_hr3 error getting xattr set cephfs2_data_hr3/100032d5676.00000000: (2) No such file or directory # Pool: cephfs2_data_hec5.4 layout parent However, I found that some files have backtrace objects in another pool: $ for pool in cephfs2_data cephfs2_data_hr3 cephfs2_data_hec5.4; do echo "# Pool: $pool" rados -p $pool listxattr 100032c464a.00000000 done # Pool: cephfs2_data parent # Pool: cephfs2_data_hr3 parent # Pool: cephfs2_data_hec5.4 layout parent Apparently, this happens when you create a file in a directory that is assigned to a non-default pool, and then change the pool before writing any data into the object. In other words, any pool that a file is *ever* assigned to after being created empty, ends up with a backtrace object. This is visible in the backtrace object itself: rados -p cephfs2_data getxattr 100032c464a.00000000 parent \ | head -c -1 \ | ceph-dencoder type inode_backtrace_t import - decode dump_json { "ino": 1099564861002, "ancestors": [ <snip> ], "pool": 14, "old_pools": [ 11, 12 ] } Here pool 11 is the default pool, pool 14 is the desired pool, and pool 12 is the pool of the directory layout for the directory the file was created in (before being reassigned prior to writing data). I found this rather surprising, as I'm migrating a large amount of data to a new pool by setting file layouts before writing out data to the new temporary file [1] and my temporary directory happened to be assigned to a non-default pool, so now I have a bunch of garbage objects in that pool (which happens to be an HDD pool, so this hurts performance). I would have expected the directory's assigned layout pool to be irrelevant if I override it after opening a new file for writing, but apparently this is not the case. I guess this is so existing clients can find the real pool if they have the old pool cached or something? I'm not entirely sure why this behavior exists. It would be nice if there were some way to clean this up without having to rewrite all the files again (it's terabytes of data). I don't mind if I have to take the FS down for a bit to run a cleanup tool, but doing manual surgery on the backtrace objects and inode omap values seems pretty difficult and error-prone :/ [1] https://gist.github.com/marcan/26cc3ac7241f866dca38916215dd10ff - Hector

2 5

PG inconsistent one extra object?
by Work Ceph 03 Jun '25

03 Jun '25

Hello guys, We are facing an unexpected error with a single PG after a node addition. We are seeing the following error message: ``` scrub : stat mismatch, got 299586/299585 objects, 0/0 clones, 299586/299585 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 366757864470/366755158108 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes. log_channel(cluster) log [ERR] scrub 1 error ``` It looks as if there is an extra object in the PG. Have you guys ever seen such a situation? The command "rados list-inconsistent-snapset" does not return anything. The command "rados list-inconsistent-pg" only returns the affected PG. The command "rados list-inconsistent-obj" does not return anything. We are suspecting that this might be caused by the number of PGs in the pool, which is too small. However, before trying to increase it, we are trying to understand what is going on. Is it safe to increase the number of PGs with a PG inconsistent with this situation with an extra object? We are running Ceph on Ubuntu 22. We are using the standard Ceph binaries from the distro.

2 5

Application for Official Ceph Mirror – Telepoint, Bulgaria
by mirror＠telepoint.bg 03 Jun '25

03 Jun '25

Hello, I am contacting you to apply to add our mirror as an official mirror of the Ceph project. Our mirror is hosted at Telepoint DataCenter in Sofia, Bulgaria. We have carefully reviewed and complied with all the official Ceph mirror requirements, including logging policies, synchronization frequency, directory structure, and protocol support. Here are the technical and administrative details: Domains: - Primary: mirror.telepoint.bg - Alternative: mirror.bg IP Addresses: - IPv4: 78.128.80.69, 78.128.80.70 - IPv6: 2a01:8740:ffff:fffc::31, 2a01:8740:ffff:fffc::32 Available Protocols: - HTTP: http://mirror.telepoint.bg/ceph/ - HTTPS: https://mirror.telepoint.bg/ceph/ - RSYNC: rsync://mirror.telepoint.bg/ceph/ Other Details: - Sync schedule: Every 3 hours - Bandwidth: 40 Gbps - Sponsor: Telepoint.bg - Location: Sofia, Bulgaria - Contact email: mirror(a)telepoint.bg We would like to request the CNAME bg.ceph.com to point to mirror.telepoint.bg. Thank you for your time and consideration. Please let us know if you need any additional information. -- Best regards, Valentin Nikolov Telepoint Mirror Service 122, Ovche Pole str. 1303 Sofia, Bulgaria Phone: +359 2 490 32 10 Mobile: +359 893 590 195 Mobile 2: +359 888 536 968 Fax: +359 2 490 3212 E-mail: noc(a)telepoint.bg web: https://www.telepoint.bg telepoint.bg

1 0

Make backup to another server.
by Nmz 02 Jun '25

02 Jun '25

Hello everyone, I came to Ceph from ZFS world. In ZFS I can use snapshots as backup way. Keep it localy or send to another server/pool. Is it possible to do it the same using Ceph? Right now in my cluster I use RBD and CephFS. RBD data structure: Ceph pools pool proxmox_metadata Rep SSD pool proxmox_data EC SSD storage.cfg rbd: ceph_rbd content images data-pool proxmox_data pool proxmox_metadata Ceph pools pool proxmox_data_1_ssd_other_hdd R 1 SSD 2 HDD storage.cfg rbd: ceph_rbd_1ssd_hdd content images pool proxmox_data_1_ssd_other_hdd CephFS data structure: Ceph pools pool cephfs_metadata Rep SSD pool cephfs_data Rep SSD pool web_data EC SSD pool db_data EC SSD pool data EC HDD # ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data web_data db_data data ] subvolumegroup | data_pool | subvolume home | web_data | nmz .... db | db_data | sql persoanl | data | Video .... As you can see subvolumes use specific ceph pools. For every subvolume I use snapshots. Ceph backup server will work in cold mode, single node, not 24/7. What is the best way to send data to backup server ? I want to use incremental backup sync. Can I achieve it ? Thanks

2 1

Mon Election Leader goes unresponsive with 100% cpu usage on fn_monstore
by Wesley Dillingham 01 Jun '25

01 Jun '25

Tracker issue made here with some additional details: https://tracker.ceph.com/issues/71501 Cluster version 18.2.4 I came to assist with a non-functional cluster which had OSDs erroneously --force purged and led to multiple (6) degraded + inactive PGs (4 remaining shards in a 4+2) and 1 remapped+incomplete PG (3 shards in a 4+2) In an effort to restore order to the cluster and get backfill working: the degraded PGs shared a common primary OSD and that OSD was restarted. Additionally min_size was dropped from 5 to 4 for this pool (a temporary measure while the cluster recovered). This caused the inactive degraded PGs to go active and start their backfill. The PGs steadily worked on backfill for a few hours. Near immediately after the final of the 6 degraded PGs finished its backfill the monitor quorum broke and the cluster became unresponsive. In this state 2 of the 5 mons showed 100% cpu usage. In an attempt to fix the mon quorum some combination of monitor service restart attempts occurred and ultimately all ceph services were brought down in order to isolate the MON issue. As the situation stands currently any combination of starting 3 MONs (quorum eligible at 3) causes the lowest MON (the to-be-leader) to hit a 100% cpu with fn_monstore thread and render unresponsive the admin socket of that mon only (the other 2 respond via admin socket and show either probing or electing). I have captured logs of the to-be-leader mon (its daemon started first) with debug_mon = 20 and debug_ms = 20 (I should probably recapture with debug_paxos = 20). The pegged cpu only occurs once the third MON is started (seems to be an election issue). The MON has been allowed to run for hours with no progress in that state. Eventually it seems the leader goes into the (leader) state (Claims "I win" in the logs) but the other 2 mons continue their election cycle. In states of probing or electing still. I have verified NTP clock is fine and sync'd to same source on all the mons and connectivity between the mons at both ports is functional. The situation the PGs were in during the subsequent MON fault leads me to believe the problem is more complex than typical monitor election issues. Also of note the backing disk of the to-be-leader MON is mostly idle. At this point I am interested in taking backups of all the mon stores and injecting a modified single-mon monmap in a mon and seeing if we can get back up but am also concerned that that single mon will be the de-facto leader and also be unresponsive. Interested in any suggestions from the wider community. Thanks! Respectfully, *Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes(a)wesdillingham.com

2 5