February 2026 - ceph-users

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

cephadm without containers
by Adrian Sevcenco 10 Apr '26

10 Apr '26

Hi! Is there any possibility to use cephadm without the container based tooling? (so to use bare-metal, with the packages directly installed on the machine) If not, is there something technical that prevents it to happen or just that there is a policy that container based deployment is the only best way to do it therefore cephadm will only use it this way? I ask this in the context of finding out that ceph-ansible is deprecated and that it should not be used, but i still need to use ansible for management and i think (so without factual experience) that bare metal services would be easier to manage after doing by hand deployment with cephadm Thank you! Adrian

11 20

CephFS: no MDS does join the filesystem
by Robert Sander 24 Mar '26

24 Mar '26

Hi, we are having an issue at a customer site where a 3PB CephFS is in failed state. The cluster itself is unhealthy and awaits replacements disks: # ceph -s cluster: id: 28ca2bfa-d87e-11ed-83a3-1070fddda30f health: HEALTH_ERR 4 failed cephadm daemon(s) There are daemons running an older version of ceph 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged 8 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 46 pgs backfill_toofull Possible data damage: 4 pgs inconsistent Degraded data redundancy: 6427646/15858772167 objects degraded (0.041%), 8 pgs degraded, 8 pgs undersized 6 pool(s) nearfull (muted: OSDMAP_FLAGS OSD_SCRUB_ERRORS(2d) PG_NOT_DEEP_SCRUBBED PG_NOT_SCRUBBED) services: mon: 3 daemons, quorum sn01,sn03,sn02 (age 3w) mgr: sn03.crlpzh(active, since 33h), standbys: sn01.tegfya, sn02.mzvgcr mds: 18/19 daemons up, 1 standby osd: 181 osds: 174 up (since 4d), 172 in (since 4d); 206 remapped pgs flags nodeep-scrub data: volumes: 2/3 healthy, 1 recovering; 1 damaged pools: 12 pools, 3585 pgs objects: 1.93G objects, 1.3 PiB usage: 2.5 PiB used, 501 TiB / 3.0 PiB avail pgs: 6427646/15858772167 objects degraded (0.041%) 293845758/15858772167 objects misplaced (1.853%) 2844 active+clean 532 active+clean+scrubbing 147 active+remapped+backfill_wait 28 active+remapped+backfill_toofull 11 active+remapped+backfill_wait+backfill_toofull 10 active+remapped+backfilling 6 active+undersized+degraded+remapped+backfill_toofull 2 active+clean+inconsistent 1 active+clean+scrubbing+deep+inconsistent+repair 1 active+undersized+remapped+backfilling 1 active+undersized+degraded+remapped+backfilling 1 active+recovering+degraded+remapped 1 active+remapped+inconsistent+backfill_toofull io: recovery: 183 MiB/s, 312 objects/s The CephFS metadata pool is not affected by the inconsistent PGs. The MDSs have this line in their logfile: "Monitors have assigned me to become a standby." The filesystem is joinable: # ceph fs lsflags storage_cluster joinable allow_snaps allow_multimds_snaps refuse_client_session But no MDS joins: # ceph fs status storage_cluster - 0 clients =============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed POOL TYPE USED AVAIL cephfs_metadata metadata 490G 12.9T cephfs_data data 970T 54.1T shared_data data 1351T 22.5T STANDBY MDS storage_cluster.sn04.cbvzzu MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) Why? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

6 13

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

Re: Broken dashboard after upgrade to 20.2.0
by Ankush Behl 11 Mar '26

11 Mar '26

I assume this is happening on a cluster that is running on debian. Tracker: https://tracker.ceph.com/issues/74199 PR for Fix: https://github.com/ceph/ceph/pull/67133 We have raised a backport PR in the Tentacle branch to fix it. https://github.com/ceph/ceph/pull/67510 On Wed, Feb 25, 2026 at 12:25 PM Nicola Mori via ceph-users < ceph-users(a)ceph.io> wrote: > Any help? > > On 18/02/26 8:17 AM, Nicola Mori wrote: > > I found that the error is reproducible from Ceph CLI: > > > > # ceph dashboard get-prometheus-api-host > > Error EIO: Module 'dashboard' has experienced an error and cannot handle > > commands: No module named 'smb' > > > > Any hint? > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 2

Re: reef 18.2.8 QE validation status
by Patrick Donnelly 10 Mar '26

10 Mar '26

The `fs` and `powercycle` suites will need rerun with [1]. I believe you're already doing that but I'm documenting that for the list. I'm going to look into the upgrade issues today [2]. [1] https://github.com/ceph/ceph/pull/67495 [2] https://tracker.ceph.com/issues/75034 On Mon, Feb 23, 2026 at 4:29 PM Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> wrote: > > Patrick, > > The only outstanding issues are the FS approval and upgrade issues, > reported by Laura. > > What do you suggest we do next here? > > On Mon, Feb 23, 2026 at 8:54 AM Adam King via Dev <dev(a)ceph.io> wrote: > > > orch approved. Run looks ugly due to us no having a proper ignorelist for > > the orch suite on reef, but the actual results are okay > > > > On Sat, Nov 29, 2025 at 12:09 PM Yuri Weinstein via ceph-users < > > ceph-users(a)ceph.io> wrote: > > > >> Rerun scheduled for krbd > >> > >> Still waiting for replies from dev leads > >> > >> On Mon, Nov 24, 2025 at 6:24 AM Ilya Dryomov <idryomov(a)gmail.com> wrote: > >> > >> > On Fri, Nov 21, 2025 at 12:04 AM Yuri Weinstein <yweinste(a)redhat.com> > >> > wrote: > >> > > > >> > > Note: this is an attempt to test and publish post EOL reef point > >> release > >> > > > >> > > Details of this release are summarized here: > >> > > > >> > > https://tracker.ceph.com/issues/73906#note-1 > >> <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F…> > >> > > Release Notes - TBD > >> > > LRC upgrade - N/A > >> > > > >> > > distro => jammy focal centos9 windows bookworm > >> > > > >> > > Seeking approvals/reviews for: > >> > > > >> > > smoke - Laura approved? > >> > > > >> > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King > >> approved? > >> > > (rados is also running on build 2 == build 1 + > >> > > https://github.com/ceph/ceph/pull/66056 > >> <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Fgithub.com%2Fceph%2…> > >> per Igor's request) > >> > > > >> > > rgw - Adam E approved? > >> > > > >> > > fs - Venky approved? > >> > > > >> > > orch - Adam King approved? > >> > > > >> > > rbd - Ilya approved? > >> > > krbd - Ilya approved? > >> > > >> > rbd approved, krbd needs a rerun as all jobs failed while > >> > going through cephlab.yml playbook (not related to RBD): > >> > > >> > ['Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong fs type, > >> > bad option, bad superblock on /dev/mapper/vg_nvme-lv_5, missing > >> > codepage or helper program, or other error.'] > >> > > >> > Thanks, > >> > > >> > Ilya > >> > > >> > > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > _______________________________________________ > > Dev mailing list -- dev(a)ceph.io > > To unsubscribe send an email to dev-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 1

Vote on the name for the Ceph V release
by Patrick Donnelly 02 Mar '26

02 Mar '26

The V release will follow Umbrella. It is targeting a release date of Spring 2027. Please share name ideas and vote on your favorite with a "+1" at https://pad.ceph.com/p/v -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 1

Re: Help recovering OSD: rocksdb: submit_common error: Corruption: block checksum mismatch
by Igor Fedotov 28 Feb '26

28 Feb '26

Hi Theo, Sorry I can't tell for sure what would marking OSD lost do with its encryption keys. Likely - yes, they'll be lost. But instead of going this way I'd rather suggest you to add another two OSDs and let Ceph recover more PGs replicas into them. Just in case - didn't you overwrite existing PG replicas at target OSDs when exporting PGs back to OSDs 1&3? The same PG can't have two replicas/shards at a single OSD while your OSD count is pretty limited.. Just curious for now - that still shouldn't be an issue given you have at least 2 replicas/shards for all the pools anyway. Thanks, Igor On 2/21/2026 11:00 PM, Theo Cabrerizo Diem via ceph-users wrote: > Hello Igor, > > First of all, sorry about the late reply. It took me a while to export all > shards that weren't available from the osd.2 (1 and 3 were fine, 2 didn't > start but i could use `ceph-objectstore-tool ... --op list-pgs` while osd.0 > I couldn't even list the pgs, it threw an error right away - more about it > later in the email) > > Two of the unavailable shards, when exporting, ceph-objectstore-tool core > dumped with the same issue in the rocksdb, but I should have enough chunks > to not need them - just mentioning in case is useful: > > sh-5.1# ceph-objectstore-tool --data-path /var/lib/ceph/osd --pgid 11.19s2 > --op export --file pg.11.19s2.dat > /ceph/rpmbuild/BUILD/ceph-20.2.0/src/kv/RocksDBStore.cc: In function > 'virtual int RocksDBStore::get(const std::string&, const std::string&, > ceph::bufferlist*)' thread 7ff3be4ca800 time 2026-02-04T09:42:00.743877+0000 > /ceph/rpmbuild/BUILD/ceph-20.2.0/src/kv/RocksDBStore.cc: 1961: > ceph_abort_msg("block checksum mismatch: stored = 246217859, computed = > 2155741315, type = 4 in db/170027.sst offset 28264757 size 1417") > ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle > (stable - RelWithDebInfo) > 1: (ceph::__ceph_abort(char const*, int, char const*, > std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&)+0xc9) [0x7ff3bf5391fd] > 2: (RocksDBStore::get(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x3bc) > [0x555667b340bc] > 3: > (BlueStore::omap_get_values(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, > ghobject_t const&, > std::set<std::__cxx11::basic_string<char,std::char_traits<char>, > std::allocator<char> >, std::less<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > >, > std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > > > const&, > std::map<std::__cxx11::basic_string<char,std::char_traits<char>, > std::allocator<char> >, ceph::buffer::v15_2_0::list, > std::less<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > >, > std::allocator<std::pair<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const, > ceph::buffer::v15_2_0::list> > >*)+0x401) [0x555667a25fe1] > 4: (PG::peek_map_epoch(ObjectStore*, spg_t, unsigned int*)+0x361) > [0x5556675e0101] > 5: main() > 6: /lib64/libc.so.6(+0x2a610) [0x7ff3be930610] > 7: __libc_start_main() > 8: _start() > *** Caught signal (Aborted) ** > in thread 7ff3be4ca800 thread_name:ceph-objectstor > ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle > (stable - RelWithDebInfo) > 1: /lib64/libc.so.6(+0x3fc30) [0x7ff3be945c30] > 2: /lib64/libc.so.6(+0x8d03c) [0x7ff3be99303c] > 3: raise() > 4: abort() > 5: (ceph::__ceph_abort(char const*, int, char const*, > std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&)+0x186) [0x7ff3bf5392ba] > 6: (RocksDBStore::get(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&, ceph::buffer::v15_2_0::list*)+0x3bc) > [0x555667b340bc] > 7: > (BlueStore::omap_get_values(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, > ghobject_t const&, > std::set<std::__cxx11::basic_string<char,std::char_traits<char>, > std::allocator<char> >, std::less<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > >, > std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > > > const&, > std::map<std::__cxx11::basic_string<char,std::char_traits<char>, > std::allocator<char> >, ceph::buffer::v15_2_0::list, > std::less<std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > >, > std::allocator<std::pair<std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const, > ceph::buffer::v15_2_0::list> > >*)+0x401) [0x555667a25fe1] > 8: (PG::peek_map_epoch(ObjectStore*, spg_t, unsigned int*)+0x361) > [0x5556675e0101] > 9: main() > 10: /lib64/libc.so.6(+0x2a610) [0x7ff3be930610] > 11: __libc_start_main() > 12: _start() > Aborted (core dumped) > > > > After importing all shards that I could recover that weren't available, I > don't have any "unknown" pgs anymore. I still have lots of PGs in "down" > state, which I assume I need to flag both "dead" OSDs as lost to unstuck > them. Since it is an operation I cannot go back, I would like to confirm > that is indeed the correct next step to take. > > I have a few questions to understand "what happens" in the next step > (marking osd as lost?): > > Shall I assume that once I flag an OSD as lost, I won't be able to > "activate" it since I use encryption when initializing the bluestore OSD, > or flagging them as lost won't destroy their unlocking keys? (which means > any hope of further extracting data to be gone, mostly on the osd.0 which I > couldn't use ceph-objectstore-tool at all since the power loss). > > I think I should have all the shards from the PGs but just in case, I've > managed to make a clone of the osd.0 on a different physical disk (the > other reason I took long to answer). But still ceph-objectstore-tool > refuses to run: > > # ceph-objectstore-tool --data-path /var/lib/ceph/osd --op list-pgs > Mount failed with '(5) Input/output error' > > # ls -l /var/lib/ceph/osd > total 28 > lrwxrwxrwx 1 ceph ceph 50 Feb 4 08:26 block -> > /dev/mapper/zNPZJR-i0TZ-6NtK-URto-tjfs-iJRb-GCAYEm > -rw------- 1 ceph ceph 37 Feb 4 08:26 ceph_fsid > -rw------- 1 ceph ceph 37 Feb 4 08:26 fsid > -rw------- 1 ceph ceph 55 Feb 4 08:26 keyring > -rw------- 1 ceph ceph 106 Jan 24 00:44 lockbox.keyring > -rw------- 1 ceph ceph 6 Feb 4 08:26 ready > -rw------- 1 ceph ceph 10 Feb 4 08:26 type > -rw------- 1 ceph ceph 2 Feb 4 08:26 whoami > > Just as information, all except 2 pools in my cluster are "replicated". > Pools id 11 and 16 are erasure coded (2+1). If I understood correctly, as > long as I have two acting shards (and at most one "NONE"), data should be > available (at least in read-only) once I mark the down OSDs as lost. Is > that understanding correct? > > Another information, pools 10 and 15 are the "replicated root pools" before > the erasure coded pools were created. > > Ignoring osd.0 for now, here are the current state of my cluster (mds is > intentionally not started while I try to fix the PGs): > ### ceph osd lspools > 3 .rgw.root > 4 default.rgw.log > 5 default.rgw.control > 6 default.rgw.meta > 10 ark.data > 11 ark.data_ec > 12 ark.metadata > 14 .mgr > 15 limbo > 16 limbo.data_ec > 18 default.rgw.buckets.index > 19 default.rgw.buckets.data > ### > > ### ceph health > # ceph -s > cluster: > id: 021f058f-dbf3-4a23-adb5-21d83f3f1bb6 > health: HEALTH_ERR > 1 filesystem is degraded > 1 filesystem has a failed mds daemon > 1 filesystem is offline > insufficient standby MDS daemons available > Reduced data availability: 143 pgs inactive, 143 pgs down > Degraded data redundancy: 1303896/7149898 objects degraded > (18.237%), 218 pgs degraded, 316 pgs undersized > 144 pgs not deep-scrubbed in time > 459 pgs not scrubbed in time > 256 slow ops, oldest one blocked for 1507794 sec, osd.1 has > slow ops > too many PGs per OSD (657 > max 500) > > services: > mon: 2 daemons, quorum ceph-ymir-mon2,ceph-ymir-mon1 (age 2w) > mgr: ceph-ymir-mgr1(active, since 2w) > mds: 0/1 daemons up (1 failed) > osd: 4 osds: 2 up (since 29m), 2 in (since 4w); 24 remapped pgs > > data: > volumes: 0/1 healthy, 1 failed > pools: 12 pools, 529 pgs > objects: 2.46M objects, 7.4 TiB > usage: 8.3 TiB used, 13 TiB / 22 TiB avail > pgs: 27.032% pgs not active > 1303896/7149898 objects degraded (18.237%) > 306628/7149898 objects misplaced (4.289%) > 218 active+undersized+degraded > 143 down > 98 active+undersized > 45 active+clean > 19 active+clean+remapped > 4 active+clean+remapped+scrubbing+deep > 1 active+clean+remapped+scrubbing > 1 active+clean+scrubbing+deep > ### ceph health > > ### ceph health detail > # ceph health detail > HEALTH_ERR 1 filesystem is degraded; 1 filesystem has a failed mds daemon; > 1 filesystem is offline; insufficient standby > MDS daemons available; Reduced data availability: 143 pgs inactive, 143 > pgs down; Degraded data redundancy: 1303896/714 > 9898 objects degraded (18.237%), 218 pgs degraded, 316 pgs undersized; 144 > pgs not deep-scrubbed in time; 459 pgs not sc > rubbed in time; 256 slow ops, oldest one blocked for 1508207 sec, osd.1 has > slow ops; too many PGs per OSD (657 > max 50 > 0) > [WRN] FS_DEGRADED: 1 filesystem is degraded > fs ark is degraded > [WRN] FS_WITH_FAILED_MDS: 1 filesystem has a failed mds daemon > fs ark has 1 failed mds > [ERR] MDS_ALL_DOWN: 1 filesystem is offline > fs ark is offline because no MDS is active for it. > [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available > have 0; want 1 more > [WRN] PG_AVAILABILITY: Reduced data availability: 143 pgs inactive, 143 pgs > down > pg 10.11 is down, acting [1,3] > pg 10.18 is down, acting [3,1] > pg 10.1d is down, acting [1,3] > pg 10.1f is down, acting [1,3] > pg 11.10 is down, acting [3,1,NONE] > pg 11.12 is down, acting [1,NONE,3] > pg 11.18 is stuck inactive for 4w, current state down, last acting > [1,3,NONE] > pg 11.19 is down, acting [3,1,NONE] > pg 11.1b is down, acting [1,NONE,3] > pg 11.62 is down, acting [NONE,3,1] > pg 11.63 is down, acting [3,NONE,1] > pg 11.64 is down, acting [NONE,1,3] > pg 11.66 is down, acting [NONE,3,1] > pg 11.67 is down, acting [1,NONE,3] > pg 11.68 is down, acting [3,NONE,1] > pg 11.69 is down, acting [NONE,1,3] > pg 11.6a is down, acting [1,NONE,3] > pg 11.6b is down, acting [NONE,1,3] > pg 11.6f is down, acting [NONE,3,1] > pg 11.71 is down, acting [1,3,NONE] > pg 11.72 is down, acting [1,3,NONE] > pg 11.74 is down, acting [NONE,3,1] > pg 11.76 is down, acting [1,NONE,3] > pg 11.78 is down, acting [3,1,NONE] > pg 11.7d is down, acting [NONE,3,1] > pg 11.7e is down, acting [NONE,1,3] > pg 15.15 is down, acting [1,3] > pg 15.16 is down, acting [3,1] > pg 15.17 is down, acting [1,3] > pg 15.1a is down, acting [3,1] > pg 16.1 is down, acting [1,3,NONE] > pg 16.4 is down, acting [1,3,NONE] > pg 16.b is down, acting [3,NONE,1] > pg 16.60 is down, acting [3,1,NONE] > pg 16.61 is down, acting [3,1,NONE] > pg 16.62 is down, acting [3,NONE,1] > pg 16.63 is down, acting [3,NONE,1] > pg 16.65 is down, acting [NONE,3,1] > pg 16.67 is down, acting [1,NONE,3] > pg 16.68 is down, acting [1,NONE,3] > pg 16.69 is down, acting [3,1,NONE] > pg 16.6a is down, acting [1,3,NONE] > pg 16.6c is down, acting [1,3,NONE] > pg 16.70 is down, acting [3,NONE,1] > pg 16.73 is down, acting [3,NONE,1] > pg 16.74 is down, acting [1,3,NONE] > pg 16.75 is down, acting [3,1,NONE] > pg 16.79 is down, acting [3,NONE,1] > pg 16.7a is down, acting [1,3,NONE] > pg 16.7e is down, acting [1,3,NONE] > pg 16.7f is down, acting [3,NONE,1] > [WRN] PG_DEGRADED: Degraded data redundancy: 1303896/7149898 objects > degraded (18.237%), 218 pgs degraded, 316 pgs under > sized > pg 3.18 is stuck undersized for 36m, current state active+undersized, > last acting [1,3] > ...<snipped for brevity> > ### > > Once again, I cannot thank you enough for looking into my issue. > I have the impression that being able to recover the data I need is just > around the corner. Although the croit.io blog did mention flagging the osd > as lost, I would like to double check it to avoid losing any possibility to > recover the data. > > If there's anything further I could check or if you need full output of the > commands, let me know. > > Thanks in advance. > > On Tue, 3 Feb 2026 at 10:26, Igor Fedotov <igor.fedotov(a)croit.io> wrote: > >> Hi Theo, >> >> you might want to try to use PG export/import using ceph-objectstore-tool. >> >> Please find more details here >> https://www.croit.io/blog/how-to-recover-inactive-pgs-using-ceph-objectstor… >> >> >> Thanks, >> >> Igor >> On 03/02/2026 02:38, Theo Cabrerizo Diem via ceph-users wrote: >> >> :12:18.895+0000 7f0c543eac00 -1 bluestore(/var/lib/ceph/osd) >> fsck error: free extent 0x1714c521000~978b26df000 intersects allocatedblocks >> fsck status: remaining 1 error(s) and warning(s) >> >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Re: Prometheus cannot connect to Ceph-Exporter
by Eugen Block 28 Feb '26

28 Feb '26

Hi, to me it seems like it's not available yet. Looking at [0], there is a section to update the exporter: security_enabled, _, _ = self.mgr._get_security_config() if security_enabled: exporter_config.update({'https_enabled': True}) tls_creds = self.get_certificates(daemon_spec) exporter_config['files'] = { 'ceph-exporter.crt': tls_creds.cert, 'ceph-exporter.key': tls_creds.key } But I don't see this section neither in current Tentacle [1] nor in current Squid [2]. Tentacle does already support setting a cert and key file, but apparently not enabling https: tentacle:~ # ceph config ls | grep exporter exporter_sock_dir exporter_addr exporter_http_port exporter_cert_file exporter_key_file exporter_prio_limit exporter_stats_period exporter_sort_metrics But I'm not a dev, so take that with a grain of salt. Regards, Eugen [0] https://github.com/ceph/ceph/blob/1ef2745047383fbb712b9494e125b2a312fa2f39/… [1] https://github.com/ceph/ceph/blob/v20.2.0/src/pybind/mgr/cephadm/services/c… [2] https://github.com/ceph/ceph/blob/v19.2.3/src/pybind/mgr/cephadm/services/c… Zitat von "Maintz, Thomas via ceph-users" <ceph-users(a)ceph.io>: > Hello everyone, > > We deployed the monitoring stack with Grafana and Prometheus via > cephadm and set “mgr/cephadm/secure_monitoring_stack true.” > To get the OSD latencies into the dashboard, we deployed the > ceph-exporter on the hosts. > However, due to the secure_monitoring_stack, Prometheus tries to > access the Ceph Exporter via https, as the schema has been set to > https by cephadm. > The Ceph exporter can be accessed locally via http with curl, but > not via https. > If I manually configure the schema for the Ceph exporter to http in > prometheus.yml and restart the service, I get the data correctly. > However, a redeploy overwrites my change. > Can someone tell me how to either configure the Ceph-exporter to > respond to https or tell Prometheus to only access the Ceph exporter > via http? Unfortunately, there is no documentation on this. > > Best regards and many thanks in advance. > Thomas Maintz > > > > > --------------------------------------------------------------------------------------------- > --------------------------------------------------------------------------------------------- > Forschungszentrum Jülich GmbH > 52425 Jülich > Sitz der Gesellschaft: Jülich > Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Stefan Müller > Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), > Dr. Stephanie Bauer (stellvertretende Vorsitzende), > Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers > --------------------------------------------------------------------------------------------- > ---------------------------------------------------------------------------------------------

1 0

should ceph mount propagate inotify?
by lejeczek 27 Feb '26

27 Feb '26

HI guys. Should ceph propagate inotify/logs "backwards" across all client-mounts? Say... clientA locks -> mount -> ceph fs <- mount <- clientB gets/sees the file has a lock/inotify does that make sense and if so & ceph should do that - what have I missed, possibly, for I do not get this behaviour. many thanks, L.

1 0

Re: 8443: connect: connection refused
by lejeczek 27 Feb '26

27 Feb '26

I did not think that ceph should be aware of any "external" proxy being used - it runs on one of ceph nodes - I assumed such a proxy would only be relevant to users/clients looking at ceph via web-browser. As far a ceph goes I did: -> $ ceph config set mgr mgr/dashboard/standby_behaviour "redirect" # set it back to default -> $ ceph config set mgr mgr/dashboard/standby_behaviour "error" -> $ ceph config set mgr mgr/dashboard/redirect_resolve_ip_addr true # resolve to hostnames -> $ ceph config set mgr mgr/dashboard/ssl false -> $ ceph mgr services { "dashboard": "http://10.1.1.63:8080/", "prometheus": "http://10.1.1.63:9283/" } Rest is Nginx's, what I'd call a "standard" proxy configuration. If needed - I should spam it here? thanks, L.

1 0

Prometheus cannot connect to Ceph-Exporter
by Maintz, Thomas 27 Feb '26

27 Feb '26

Hello everyone, We deployed the monitoring stack with Grafana and Prometheus via cephadm and set “mgr/cephadm/secure_monitoring_stack true.” To get the OSD latencies into the dashboard, we deployed the ceph-exporter on the hosts. However, due to the secure_monitoring_stack, Prometheus tries to access the Ceph Exporter via https, as the schema has been set to https by cephadm. The Ceph exporter can be accessed locally via http with curl, but not via https. If I manually configure the schema for the Ceph exporter to http in prometheus.yml and restart the service, I get the data correctly. However, a redeploy overwrites my change. Can someone tell me how to either configure the Ceph-exporter to respond to https or tell Prometheus to only access the Ceph exporter via http? Unfortunately, there is no documentation on this. Best regards and many thanks in advance. Thomas Maintz --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellvertretende Vorsitzende), Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers --------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------

1 0

RGW Admin Ops: question about /admin/account permissions
by Jeffrey van Doesburg 25 Feb '26

25 Feb '26

Hello, I am experiencing some issues with the RADOSGW Admin Ops API in Ceph Tentacle (20.2.0). According to the latest documentation[1], it should be possible (since Squid) to perform calls to /admin/account. However, in my setup this endpoint consistently returns 403 AccessDenied. What I find confusing is that these /admin/account options are only present in the latest documentation. They do not appear in the Squid or Tentacle version specific documentation. My user is created with the following caps: radosgw-admin caps add \ --uid="<user>" \ -- caps="accounts=*;buckets=*;info=read;metadata=read;usage=read;users=*" For example, a call to the metadata endpoint works without any issues: curl -sS \ --user "${ACCESS_KEY}:${SECRET_KEY}" \ --aws-sigv4 "aws:amz:us-east-1:s3" \ "${RGW_URL}/admin/metadata/account?format=json" Response: ["RGW67272693614667731","RGW60101944561690517","RGW31205003107553515"] However, a call to /admin/account returns AccessDenied: curl -sS \ --user "${ACCESS_KEY}:${SECRET_KEY}" \ --aws-sigv4 "aws:amz:us-east-1:s3" \ "${RGW_URL}/admin/account?id=RGW67272693614667731&format=json" Response: {"Code":"AccessDenied","Message":"","RequestId":"tx00000f330b10a36af581 f-00699ef8b2-24707-default","HostId":"24707-default-default"} From my testing, the /admin/account endpoint works when the user is marked as a system user. But returns AccessDenied when using a "regular" user that has accounts=* caps. Should accounts=* be sufficient to access /admin/account, as the documentation suggests? Or is a system user explicitly required for these endpoints? Additionally, can someone clarify whether /admin/account is fully supported in Tentacle? Given that it only appears in the latest documentation and not in the version specific Tentacle docs? Thanks in advance! Kind regards, Jeffrey [1] https://docs.ceph.com/en/latest/radosgw/adminops/#get-account-info

1 0

Re: metrics duplicate in prometheus after migrate and upgrade to cephadm
by Francesco Usseglio Gaudi 25 Feb '26

25 Feb '26

Hi Roberto, thank you very much, this is the right parameter to toggle! I have change it from "false" to "true" and now we have graphs working again. Have a good day Francesco Il 25/02/26 14:20, Roberto Valverde Cameselle ha scritto: > Hi Francesco, > Can you check if you have this feature set? > mgr advanced mgr/prometheus/exclude_perf_counters > Make sure that is set to true so then performance metrics are gathered > from the ceph-exporters and not from the manager. > If your cluster is small, you can also not add ceph-exporters and set > the above setting to false so the manager will be also exposing the > performance metrics. > Cheers > > > *-----------------------------------------------------------------* > *Roberto VALVERDE CAMESELLE* > IT Storage And Data Management > CERN, European Organization for Nuclear Research > Esplanade des Particules 1, Geneve (Switzerland) > ------------------------------------------------------------------------ > *From:* Francesco Usseglio Gaudi via ceph-users <ceph-users(a)ceph.io> > *Sent:* Tuesday, February 24, 2026 6:17 PM > *To:* ceph-users(a)ceph.io <ceph-users(a)ceph.io> > *Subject:* [ceph-users] metrics duplicate in prometheus after migrate > and upgrade to cephadm > Hi all, > > we have a 10 nodes cluster used for S3 with radosgw. > > Initially the cluster has been installed with ceph-ansible and was at > version 18. > > Some months ago an external consultant converted to cephadm and upgrade > to 19.2.3 > > Almost everything is ok but we have some issues with dashboard and > grafana graphs. > > In Dashboard → Cluster → OSDs → Overall Performance we can't see "OSD > Read Latencies" and "OSD Write Latencies" graphs, they have red triangle > icons with this error popup: > > "execution: found duplicate series for the match group > {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: > [{ceph_daemon=\"osd.0\", > cluster=\"6e06959e-3ef3-4017-a467-b1d482bc7269\", > instance=\"ceph_cluster\", job=\"ceph\"}, {ceph_daemon=\"osd.0\", > cluster=\"6e06959e-3ef3-4017-a467-b1d482bc7269\", instance=\"node1\", > job=\"ceph-exporter\"}];many-to-many matching not allowed: matching > labels must be unique on one side" > > After some my investigations I found out there are some duplicate > metrics in prometheus db, one coming from ceph (I think mgr module?) and > one from ceph-exporter, for example this metric: > > ceph_osd_op_r_latency_sum > > be there with two distinct values with different job labels (values are > exactly the same) > > Every node has a running ceph-exporter container and there are 3 nodes > with mgr services. > > I have tried to modify grafana dashboard queries to use only one of two > metrics specifying the job label but they are readonly (I can make a > copy and modifying it that works, but i don't know how to use this new > grafana dashboard in ceph dashboard). > > I have read from ceph documentation and I understand ceph-exporter is no > longer required but on another ceph cluster recently set up with cephadm > version 19.2.3 from the very start, ceph-exporter container are presents > on every nodes, so I'm not sure if I should keep ceph-exporter > containers running > > Which is the right way of extract these metrics from ceph? > > Thank you all > > Francesco Usseglio > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

RGW users with same email
by Gilles Mocellin 25 Feb '26

25 Feb '26

Hello Cephers, Does someone manage to have the same email on several RGW/S3 users ? There is an option to allow that, because it was not by default. But, at least in Ceph Reef (v18.2.7 here), it does not work. In a v20.2.0 LAB, it works in CLI, but the Dashboard still verify the uniqueness of email... I've created an issue here : https://tracker.ceph.com/issues/75166 --Gilles

1 0

Re: NVMe ARM cluster: throughput plateaus and initial OSD hotspot
by Anthony D'Atri 25 Feb '26

25 Feb '26

> What internal Ceph mechanism could explain these three performance phases? > Not necessarily internal. Please tell us exactly which model these SSDs are, and if they have the latest firmware. I suspect that they are client-class and not intended for sustained workload. > Why does one OSD initially receive more load before balancing occurs? > CRUSH is effectively a hash function, and the output is typically a bell curve. Balancing corrects for that, like a field flattener on a telescope or a collimator on a laser. As your cluster fills from empty, with defaults on the balancer and autoscaler will split and move PGs, so this isn’t an ideal scenario for benching. I suggest adding the `bulk` flag to the data pools, filling the cluster at least halfway, then disabling the balancer and autoscaler so neither kicks in during your benching. Also, if you’re using the PG autoscaler with default settings, you likely have way too few PGs. ceph config set global mon_max_pg_per_osd 600 ceph config set global mon_target_pg_per_osd 300 That will help the utilization even out.

1 0

Re: NVMe ARM cluster: throughput plateaus and initial OSD hotspot
by Maged Mokhtar 25 Feb '26

25 Feb '26

i would try simple things first: 1) disable scrub/deepscrub and re-run the test. if this was the cause, you can re-enable scrub again but lower the scrub load 2) disable pg auto scaling, manually set the pg_num yourself on new pool and re-run test. /maged On 25/02/2026 10:47, Lina SADI via ceph-users wrote: > > Hello Ceph users, > > I am currently running performance investigations on a small ARM-based > Ceph cluster (Raspberry Pi + NVMe), and I would really appreciate > feedback from experienced users or developers. We are trying to > understand a non-trivial behavior that appears systematic rather than > random. > > Below is a technical summary of our setup and methodology. I am > attaching the figures directly so that the detailed interpretation can > be derived visually from the data. > > _CLUSTER ARCHITECTURE > _Hardware: > > * > > 3× Raspberry Pi storage nodes > > * > > Each node equipped with an identical 512 GB NVMe device > > * > > Nodes interconnected through a 1 Gb/s Ethernet switch > > * > > 1 additional Raspberry Pi acting as a client on the same network > > Software topology (Ceph Pacific 16.2.15 – default configuration): > > * > > Node 1 (umbriel): MON + MGR + OSD > > * > > Node 2 (elara): MON + MDS + OSD > > * > > Node 3 (kerberos): MON + OSD > > * > > Client node: continuous camera ingestion pipeline writing frames > into a default CephFS pool . > > _EXPERIMENT OBJECTIVE > _Our goal is to establish a stable performance baseline before > injecting failure scenarios (e.g., verifying whether images are lost > during Ceph faults). However, even with identical workloads on an > empty cluster, the system consistently evolves through several > distinct performance phases. > > _WORKLOAD DESCRIPTION > _The workload consists of a simple 2-hour execution of the camera > program (~42 FPS) with a frame size of 512 KB: > > Collected information: > > * > > Ceph metrics: cluster health, PG state, and OSD performance > (commit/apply latency) > > * > > System metrics on all nodes: CPU, memory, disk and network usage, > as well as temperature and Raspberry Pi throttling flags > > * > > Client metrics: latency and throughput > > Metrics were collected in parallel with offline parsing to minimize > perturbation. > > _OBSERVATIONS (see attached figures) > _Across multiple executions, performance does not drift gradually but > instead evolves through three clear plateaus affecting client latency, > client throughput, OSD commit/apply latency, and network traffic, > while the cluster remains HEALTH_OK. CPU usage stays stable, and > memory shows periodic cache-related fluctuations. *The most notable > behavior appears at the disk level*: one NVMe starts with much higher > utilization (~95%) while the others remain around ~40%, then activity > progressively converges until all OSDs stabilize at similar levels. > DVFS and thermal throttling do not explain the phenomenon, as running > experiments with throttling already active still produces the same > three phases. > > Our main questions are therefore: > > * > > *What internal Ceph mechanism could explain these three > performance phases?* > > * > > *Why does one OSD initially receive more load before balancing > occurs?* > > Any pointers on relevant metrics, configuration aspects, or known > behaviors on small ARM-based clusters would be extremely valuable. > > Thank you very much for your time and insights. > > Screenshot 2026-02-23 164425.png > Screenshot 2026-02-23 164440.pngScreenshot 2026-02-24 112112.png > Screenshot 2026-02-23 164517.png > Screenshot 2026-02-23 164536.png > Screenshot 2026-02-23 164552.png > Screenshot 2026-02-23 164602.png > Screenshot 2026-02-23 164624.png > Screenshot 2026-02-23 164658.png > > > _______________________________________________ > ceph-users mailing list --ceph-users(a)ceph.io > To unsubscribe send an email toceph-users-leave(a)ceph.io

1 0

NVMe ARM cluster: throughput plateaus and initial OSD hotspot
by Lina SADI 25 Feb '26

25 Feb '26

Hello Ceph users, I am currently running performance investigations on a small ARM-based Ceph cluster (Raspberry Pi + NVMe), and I would really appreciate feedback from experienced users or developers. We are trying to understand a non-trivial behavior that appears systematic rather than random. Below is a technical summary of our setup and methodology. I am attaching the figures directly so that the detailed interpretation can be derived visually from the data. *CLUSTER ARCHITECTURE*Hardware: - 3× Raspberry Pi storage nodes - Each node equipped with an identical 512 GB NVMe device - Nodes interconnected through a 1 Gb/s Ethernet switch - 1 additional Raspberry Pi acting as a client on the same network Software topology (Ceph Pacific 16.2.15 – default configuration): - Node 1 (umbriel): MON + MGR + OSD - Node 2 (elara): MON + MDS + OSD - Node 3 (kerberos): MON + OSD - Client node: continuous camera ingestion pipeline writing frames into a default CephFS pool . *EXPERIMENT OBJECTIVE*Our goal is to establish a stable performance baseline before injecting failure scenarios (e.g., verifying whether images are lost during Ceph faults). However, even with identical workloads on an empty cluster, the system consistently evolves through several distinct performance phases. *WORKLOAD DESCRIPTION*The workload consists of a simple 2-hour execution of the camera program (~42 FPS) with a frame size of 512 KB: Collected information: - Ceph metrics: cluster health, PG state, and OSD performance (commit/apply latency) - System metrics on all nodes: CPU, memory, disk and network usage, as well as temperature and Raspberry Pi throttling flags - Client metrics: latency and throughput Metrics were collected in parallel with offline parsing to minimize perturbation. *OBSERVATIONS (see attached figures)*Across multiple executions, performance does not drift gradually but instead evolves through three clear plateaus affecting client latency, client throughput, OSD commit/apply latency, and network traffic, while the cluster remains HEALTH_OK. CPU usage stays stable, and memory shows periodic cache-related fluctuations. *The most notable behavior appears at the disk level*: one NVMe starts with much higher utilization (~95%) while the others remain around ~40%, then activity progressively converges until all OSDs stabilize at similar levels. DVFS and thermal throttling do not explain the phenomenon, as running experiments with throttling already active still produces the same three phases. Our main questions are therefore: - *What internal Ceph mechanism could explain these three performance phases?* - *Why does one OSD initially receive more load before balancing occurs?* Any pointers on relevant metrics, configuration aspects, or known behaviors on small ARM-based clusters would be extremely valuable. Thank you very much for your time and insights. [image: Screenshot 2026-02-23 164425.png] [image: Screenshot 2026-02-23 164440.png][image: Screenshot 2026-02-24 112112.png] [image: Screenshot 2026-02-23 164517.png] [image: Screenshot 2026-02-23 164536.png] [image: Screenshot 2026-02-23 164552.png] [image: Screenshot 2026-02-23 164602.png] [image: Screenshot 2026-02-23 164624.png] [image: Screenshot 2026-02-23 164658.png]

1 0

Broken dashboard after upgrade to 20.2.0
by Nicola Mori 25 Feb '26

25 Feb '26

Dear Ceph users, I just successfully upgraded my cehpadm-managed cluster to 20.2.0, but the status reports the following error: Module 'dashboard' has failed: No module named 'smb' I upgraded using my self-built Ceph image based on Ubuntu 22.04 and making use of the official Ceph .deb packages, and I noticed that the pysmb package was actually not present. So I rebuilt another image in which I install it with sudo pip3 install pysmb (since I found no .deb), and this should have fixed the problem since from a shell in a container I can launch python3 and load the smb module. But still I get the dashboard error, even after restarting the managers, and the dashboard is unreachable. Any hint about how can I fix this? Thank you, Nicola

1 2

metrics duplicate in prometheus after migrate and upgrade to cephadm
by Francesco Usseglio Gaudi 24 Feb '26

24 Feb '26

Hi all, we have a 10 nodes cluster used for S3 with radosgw. Initially the cluster has been installed with ceph-ansible and was at version 18. Some months ago an external consultant converted to cephadm and upgrade to 19.2.3 Almost everything is ok but we have some issues with dashboard and grafana graphs. In Dashboard → Cluster → OSDs → Overall Performance we can't see "OSD Read Latencies" and "OSD Write Latencies" graphs, they have red triangle icons with this error popup: "execution: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{ceph_daemon=\"osd.0\", cluster=\"6e06959e-3ef3-4017-a467-b1d482bc7269\", instance=\"ceph_cluster\", job=\"ceph\"}, {ceph_daemon=\"osd.0\", cluster=\"6e06959e-3ef3-4017-a467-b1d482bc7269\", instance=\"node1\", job=\"ceph-exporter\"}];many-to-many matching not allowed: matching labels must be unique on one side" After some my investigations I found out there are some duplicate metrics in prometheus db, one coming from ceph (I think mgr module?) and one from ceph-exporter, for example this metric: ceph_osd_op_r_latency_sum be there with two distinct values with different job labels (values are exactly the same) Every node has a running ceph-exporter container and there are 3 nodes with mgr services. I have tried to modify grafana dashboard queries to use only one of two metrics specifying the job label but they are readonly (I can make a copy and modifying it that works, but i don't know how to use this new grafana dashboard in ceph dashboard). I have read from ceph documentation and I understand ceph-exporter is no longer required but on another ceph cluster recently set up with cephadm version 19.2.3 from the very start, ceph-exporter container are presents on every nodes, so I'm not sure if I should keep ceph-exporter containers running Which is the right way of extract these metrics from ceph? Thank you all Francesco Usseglio

1 0

Re: since squid (19.2.3) orch puts OSD block.db and data on same device
by Adrien Georget 24 Feb '26

24 Feb '26

Hi Boris, The workaround we used to replace disks affected by this bug is to force devices in the osd creation using the orchestrator without yaml specs. VG and LV are first created manually and then we used something like this : |ceph orch daemon add osd <host>:data_devices=/dev/data01,db_devices=/dev/fast01/db01| See the advanced OSD creation : https://docs.ceph.com/en/reef/cephadm/services/osd/#creating-new-osds Adrien Le 24/02/2026 à 14:57, Boris via ceph-users a écrit : > Thanks Marek, but this did not work here, because the SSDs still have other > block.db LVs on it. > > > Am Di., 24. Feb. 2026 um 14:46 Uhr schrieb Marek Szuba via ceph-users < > ceph-users(a)ceph.io>: > >> On 2026-02-24 11:15, Robert Sander via ceph-users wrote: >> >>> The orchestrator in Ceph 19 and 20 has a bug with hybrid OSDs: >>> >>> https://tracker.ceph.com/issues/72696 >>> >>> There is a patch available but it has not been merged yet. >> Meanwhile, a workaround that has worked for us is: >> * tell the orchestrator to delete unwanted non-hybrid OSDs _without >> zapping the drives_; >> * manually wipe both underlying devices, with as little time in >> between the two as possible; >> * explicitly tell the orchestrator to refresh its device list, so that >> it sees both available drives at the same time. >> >> -- >> MS >> _______________________________________________ >> ceph-users mailing list --ceph-users(a)ceph.io >> To unsubscribe send an email toceph-users-leave(a)ceph.io >> >

2 1

Save the date(s) - 2026 Ceph Meetup in Berlin, Germany and Frankfurt/Main, Germany - interested people are welcome !
by Matthias Muench 24 Feb '26

24 Feb '26

Hi people in Germany, we run two meetup groups in Berlin and Frankfurt with meetings nearly quarterly. For each location, we provide also a quick introduction to Ceph before the main event. Currently, we have a new set of appointments. You can join us by RSVP to the events - check out the meetup groups: - Ceph Berlin meetup: https://www.meetup.com/ceph-berlin/ - Ceph Frankfurt meetup: https://www.meetup.com/ceph-frankfurt/ What are the next dates ? ( March set, the other meetups are in planning for now - but the dates are likely those) *Ceph Berlin: * - 02.03.2026 - 02.06.2026 - 14.09.2026 - 30.11.2026 * Ceph Frankfurt: * - 12.03.2026 - 24.06.2026 - 09.09.2026 - 01.12.2026 Any suggestions ? Meet us in person or on the meetup group pages - you can open a discussion and see what people think about your idea. Happy to see you soon, -matt -- —————————————————— Matthias Muench Principal Specialist Solution Architect, OpenShift EMEA Storage Specialist matthias.muench(a)redhat.com Phone: +49-160-92654111 Red Hat GmbH Technopark II Werner-von-Siemens-Ring 12 85630 Grasbrunn Germany _______________________________________________________________________ Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243, Managing Directors: Ryan Barnhart, Charles Cachera, Michael O'Neill, Amy Ross

1 0

Re: since squid (19.2.3) orch puts OSD block.db and data on same device
by Boris 24 Feb '26

24 Feb '26

Thanks Marek, but this did not work here, because the SSDs still have other block.db LVs on it. Am Di., 24. Feb. 2026 um 14:46 Uhr schrieb Marek Szuba via ceph-users < ceph-users(a)ceph.io>: > On 2026-02-24 11:15, Robert Sander via ceph-users wrote: > > > The orchestrator in Ceph 19 and 20 has a bug with hybrid OSDs: > > > > https://tracker.ceph.com/issues/72696 > > > > There is a patch available but it has not been merged yet. > > Meanwhile, a workaround that has worked for us is: > * tell the orchestrator to delete unwanted non-hybrid OSDs _without > zapping the drives_; > * manually wipe both underlying devices, with as little time in > between the two as possible; > * explicitly tell the orchestrator to refresh its device list, so that > it sees both available drives at the same time. > > -- > MS > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

Re: since squid (19.2.3) orch puts OSD block.db and data on same device
by Marek Szuba 24 Feb '26

24 Feb '26

On 2026-02-24 11:15, Robert Sander via ceph-users wrote: > The orchestrator in Ceph 19 and 20 has a bug with hybrid OSDs: > > https://tracker.ceph.com/issues/72696 > > There is a patch available but it has not been merged yet. Meanwhile, a workaround that has worked for us is: * tell the orchestrator to delete unwanted non-hybrid OSDs _without zapping the drives_; * manually wipe both underlying devices, with as little time in between the two as possible; * explicitly tell the orchestrator to refresh its device list, so that it sees both available drives at the same time. -- MS

1 0

Re: since squid (19.2.3) orch puts OSD block.db and data on same device
by Boris 24 Feb '26

24 Feb '26

Ah nice. Thanks Robert! Am Di., 24. Feb. 2026 um 12:16 Uhr schrieb Robert Sander via ceph-users < ceph-users(a)ceph.io>: > Am 24.02.26 um 12:04 PM schrieb Boris via ceph-users: > > Hi, > > we've upgraded from reef to squid lately and are encountering the problem > > that the orchestrator now puts the block.db together with the data on the > > same HDD. > > > > We've set the orch config to use non rotational devices for the block.db > > and the rotational devices for the data. We use one SSD/NVMe for multiple > > HDDs. > > The orchestrator in Ceph 19 and 20 has a bug with hybrid OSDs: > > https://tracker.ceph.com/issues/72696 > > There is a patch available but it has not been merged yet. > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

network ring recommendations wanted
by Kai Zimmer 24 Feb '26

24 Feb '26

Dear listers, for cluster network i have currently 3x Mikrotik switches CRS518-16XS-2XQ (connected via 100 GBit) and for the "public" connection to clients via 2x10 GBit i have 5x Mikrotik switches CRS326-24S+2Q+RM (connected via 40 GBit) available. Looking at the hardware to me it seems there is only oen option to build network rings. MLAG seems to be supported only for up to two devices (not enough for me), RSTP seems unsupported. I read about people using MSTP (which is supported) but i'm unsure if it is suitable for ceph needs. I'm not talking about a high performance setup here, rather than file server and virtualization needs. Any thoughts/comments? Best regards, Kai

1 0

Re: since squid (19.2.3) orch puts OSD block.db and data on same device
by Robert Sander 24 Feb '26

24 Feb '26

Am 24.02.26 um 12:04 PM schrieb Boris via ceph-users: > Hi, > we've upgraded from reef to squid lately and are encountering the problem > that the orchestrator now puts the block.db together with the data on the > same HDD. > > We've set the orch config to use non rotational devices for the block.db > and the rotational devices for the data. We use one SSD/NVMe for multiple > HDDs. The orchestrator in Ceph 19 and 20 has a bug with hybrid OSDs: https://tracker.ceph.com/issues/72696 There is a patch available but it has not been merged yet. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

since squid (19.2.3) orch puts OSD block.db and data on same device
by Boris 24 Feb '26

24 Feb '26

Hi, we've upgraded from reef to squid lately and are encountering the problem that the orchestrator now puts the block.db together with the data on the same HDD. We've set the orch config to use non rotational devices for the block.db and the rotational devices for the data. We use one SSD/NVMe for multiple HDDs. We've now encountered that in two different clusters where we replaced HDDs. The existing block.db VGs still have capacity left. Below the output of lsblk with the mentioned disk, ceph orch ls export (only the osd service), vgs sds 65:32 0 14.6T 0 disk ├─ceph--92b5842c--8df3--43ef--9814--6707eb970cc6-osd--block--f0fe19ce--7b79--466f--a889--1935aa3e8b85 252:8 0 14.1T 0 lvm │ └─YbVkPH-f3ZG-IZfE-jdGc-CCf6-597l-uo8J3W 252:59 0 14.1T 0 crypt └─ceph--92b5842c--8df3--43ef--9814--6707eb970cc6-osd--db--1a1a654e--145f--45b5--a5b0--9cb322db6093 252:26 0 430G 0 lvm └─Xpt9TE-3Z40-1lvl-j8iS-OWYj-mQXx-waw2cR 252:77 0 430G 0 crypt --- service_type: osd service_id: osd_1_ssd service_name: osd.osd_1_ssd placement: host_pattern: '*' spec: data_devices: limit: 2 rotational: 0 size: :2TB vendor: SAMSUNG encrypted: true filter_logic: AND objectstore: bluestore --- service_type: osd service_id: osd_2_hdd_8tb service_name: osd.osd_2_hdd_8tb placement: host_pattern: '*' spec: block_db_size: 343597383680 data_devices: rotational: 1 size: :10TB db_devices: rotational: 0 size: :2TB db_slots: 5 encrypted: true filter_logic: AND objectstore: bluestore --- service_type: osd service_id: osd_4_hdd_16tb service_name: osd.osd_4_hdd_16tb placement: hosts: - s3db23 spec: block_db_size: 461708984320 data_devices: rotational: 1 size: '10TB:' db_devices: rotational: 0 size: 3TB:4TB db_slots: 8 encrypted: true filter_logic: AND objectstore: bluestore VG #PV #LV #SN Attr VSize VFree ceph-0161de84-5042-4925-a959-b9d214852511 1 4 0 wz--n- 3.49t 1.81t ceph-02da2287-597a-4619-a3d3-9358cbb60622 1 1 0 wz--n- 14.55t 0 ceph-05813ea9-d14a-4e6c-81b0-9605f0001766 1 1 0 wz--n- 14.55t 0 ceph-06d405e7-abfa-4f5d-b4e3-7229bb96f2fd 1 1 0 wz--n- 14.55t 0 ceph-18ac75a3-d187-4fc2-a8aa-3b6a26b70246 1 1 0 wz--n- 14.55t 0 ceph-19a0db63-89ed-4f2e-b5ab-961dc0f8d413 1 5 0 wz--n- 3.49t 1.39t ceph-2ecb6728-0fc3-4622-acd7-298ec3a02c02 1 1 0 wz--n- 14.55t 0 ceph-306a63bc-5a8a-44d9-af80-d4cf825ceeca 1 5 0 wz--n- 3.49t 1.39t ceph-47a79684-9cba-4e51-b143-c05f3307a876 1 1 0 wz--n- <1.75t 0 ceph-4d80d916-db9b-46f6-9e34-29ae35879e8e 1 1 0 wz--n- 14.55t 0 ceph-5fe43992-3918-4a81-bbee-fc08e1329c0b 1 1 0 wz--n- <1.75t 0 ceph-84fb035b-0c57-4249-98d5-92fdb010867d 1 1 0 wz--n- 14.55t 0 ceph-92b5842c-8df3-43ef-9814-6707eb970cc6 1 2 0 wz--n- 14.55t 0 <------ ceph-96d2979a-ac09-492a-a4c5-466fb8ca44c3 1 1 0 wz--n- 14.55t 0 ceph-a49a0de2-a9e9-47d4-aff2-e96660c4267a 1 1 0 wz--n- 14.55t 0 ceph-ac752e7a-db71-4c3b-9158-dcc3b6aebcb7 1 1 0 wz--n- 14.55t 0 ceph-c6e94f1e-0b48-4966-9e31-a67232733a63 1 1 0 wz--n- 14.55t 0 ceph-c791562d-69b2-4463-ae53-77b0997c1cd7 1 1 0 wz--n- 14.55t 0 ceph-c7bbf14b-ac87-47cc-86f9-173a113a9b80 1 1 0 wz--n- 14.55t 0 ceph-cd38189c-b8c9-40e1-a528-d645838144e4 1 1 0 wz--n- 14.55t 0 ceph-cf93829f-086b-451a-b5cb-b4ca9462f660 1 1 0 wz--n- 14.55t 0 ceph-d6dfd5aa-635d-4ba3-8efd-15764f4cdd18 1 1 0 wz--n- 14.55t 0 ceph-e247cda8-e206-42bc-ad0b-88639944ddf2 1 1 0 wz--n- 14.55t 0 ceph-e3f5355e-8653-46c4-8ea2-ae632196a062 1 5 0 wz--n- 3.49t 1.39t ceph-eee43b8f-6653-4b9d-8547-f40d94927840 1 1 0 wz--n- 14.55t 0 ceph-f208c77f-9821-4580-ab7b-63093f12a671 1 1 0 wz--n- 14.55t 0

1 0

NVMe ARM cluster: throughput plateaus and initial OSD hotspot
by Lina SADI 24 Feb '26

24 Feb '26

1 0

OSD memory usage exceeding configured limits during reads (Tentacle v20.2.0, ec_optimizations on)
by Chanu Yu 24 Feb '26

24 Feb '26

Hi all, I have a question regarding OSD memory behavior during read workloads. During sequential read tests, I observe that a small number of OSD daemons sometimes grow their memory usage far beyond configured limits and end up consuming most of the host memory. This memory is not reclaimed even after the workload stops, unless the OSD is restarted. I have already set explicit limits such as osd_memory_target and bluestore cache size (with autotune off), but some OSDs still exceed them significantly. My main questions are: 1. Is it considered normal for OSDs to exceed configured memory targets by a large margin during read workloads? 2. Is there any supported way to strictly enforce OSD memory limits (beyond current tuning options), without relying on container OOM or suffering major performance degradation? If this behavior is expected or already known, I’d appreciate any pointers. I can provide more details if needed. Thanks in advance. Best regards, Chanu Yu

1 0

Re: Ceph MDS super slow
by Martin Gerhard Loschwitz 24 Feb '26

24 Feb '26

Hey Venky, thank you for your reply! > Am 16.02.2026 um 08:34 schrieb Venky Shankar via ceph-users <ceph-users(a)ceph.io>: > > Hi Martin, > > Could you additionally do a minute worth of MDS perf dump captures at > 5 second intervals during the observed slowness? This can be done > using > >> ceph tell mds.<id> perf dump > > and pipe the result to separate (time-stamped) files and share those. > I am attaching a tarball with the perf dumps alongside the wall clock profiles generated at the same time. Please note that when running gdbpmp on the MDS, it appears to stall / hang so that after a while, the MONs declare the rank failed and initiate a fail-over. I am not sure that is supposed to be this way … I am seeing SIGINT messages in those moments in the logs. At such events, the „perf dump“ Generated would be empty. Best regards Martin -- Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH Phone: +49 2433 5253130 Mobile: +49 176 61832178 Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany Legal: HRB 21985, Amtsgericht Mönchengladbach VAT: DE363893844 True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.

2 1

Re: reef 18.2.8 QE validation status
by Yuri Weinstein 23 Feb '26

23 Feb '26

Patrick, The only outstanding issues are the FS approval and upgrade issues, reported by Laura. What do you suggest we do next here? On Mon, Feb 23, 2026 at 8:54 AM Adam King via Dev <dev(a)ceph.io> wrote: > orch approved. Run looks ugly due to us no having a proper ignorelist for > the orch suite on reef, but the actual results are okay > > On Sat, Nov 29, 2025 at 12:09 PM Yuri Weinstein via ceph-users < > ceph-users(a)ceph.io> wrote: > >> Rerun scheduled for krbd >> >> Still waiting for replies from dev leads >> >> On Mon, Nov 24, 2025 at 6:24 AM Ilya Dryomov <idryomov(a)gmail.com> wrote: >> >> > On Fri, Nov 21, 2025 at 12:04 AM Yuri Weinstein <yweinste(a)redhat.com> >> > wrote: >> > > >> > > Note: this is an attempt to test and publish post EOL reef point >> release >> > > >> > > Details of this release are summarized here: >> > > >> > > https://tracker.ceph.com/issues/73906#note-1 >> <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F…> >> > > Release Notes - TBD >> > > LRC upgrade - N/A >> > > >> > > distro => jammy focal centos9 windows bookworm >> > > >> > > Seeking approvals/reviews for: >> > > >> > > smoke - Laura approved? >> > > >> > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King >> approved? >> > > (rados is also running on build 2 == build 1 + >> > > https://github.com/ceph/ceph/pull/66056 >> <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Fgithub.com%2Fceph%2…> >> per Igor's request) >> > > >> > > rgw - Adam E approved? >> > > >> > > fs - Venky approved? >> > > >> > > orch - Adam King approved? >> > > >> > > rbd - Ilya approved? >> > > krbd - Ilya approved? >> > >> > rbd approved, krbd needs a rerun as all jobs failed while >> > going through cephlab.yml playbook (not related to RBD): >> > >> > ['Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong fs type, >> > bad option, bad superblock on /dev/mapper/vg_nvme-lv_5, missing >> > codepage or helper program, or other error.'] >> > >> > Thanks, >> > >> > Ilya >> > >> > >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io >

1 0

Re: reef 18.2.8 QE validation status
by Adam King 23 Feb '26

23 Feb '26

orch approved. Run looks ugly due to us no having a proper ignorelist for the orch suite on reef, but the actual results are okay On Sat, Nov 29, 2025 at 12:09 PM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > Rerun scheduled for krbd > > Still waiting for replies from dev leads > > On Mon, Nov 24, 2025 at 6:24 AM Ilya Dryomov <idryomov(a)gmail.com> wrote: > > > On Fri, Nov 21, 2025 at 12:04 AM Yuri Weinstein <yweinste(a)redhat.com> > > wrote: > > > > > > Note: this is an attempt to test and publish post EOL reef point > release > > > > > > Details of this release are summarized here: > > > > > > https://tracker.ceph.com/issues/73906#note-1 > > > Release Notes - TBD > > > LRC upgrade - N/A > > > > > > distro => jammy focal centos9 windows bookworm > > > > > > Seeking approvals/reviews for: > > > > > > smoke - Laura approved? > > > > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King approved? > > > (rados is also running on build 2 == build 1 + > > > https://github.com/ceph/ceph/pull/66056 per Igor's request) > > > > > > rgw - Adam E approved? > > > > > > fs - Venky approved? > > > > > > orch - Adam King approved? > > > > > > rbd - Ilya approved? > > > krbd - Ilya approved? > > > > rbd approved, krbd needs a rerun as all jobs failed while > > going through cephlab.yml playbook (not related to RBD): > > > > ['Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong fs type, > > bad option, bad superblock on /dev/mapper/vg_nvme-lv_5, missing > > codepage or helper program, or other error.'] > > > > Thanks, > > > > Ilya > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: reef 18.2.8 QE validation status
by Venky Shankar 23 Feb '26

23 Feb '26

Hi Yuri, On Thu, Feb 12, 2026 at 9:30 PM Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> wrote: > > Still seeking approvals for: > > (smoke - FAILED, was reviewed and approved by Laura and Patrick, see Slack > thread for details) > rados - FAILED, Afreen, Travis? ( Laura approved for core) > > rgw - FAILED, Adam E approved? > > (rbd, krbd - FAILED, Ilya approved) > fs - FAILED, Venky approved? I wasn't able to finish reviewing the run and teuthology results are unavailable right now due to the lab outage. > orch - FAILED, Adam K approved? > > upgrade/pacific-x (reef) - FAILED Laura, approved? > upgrade/quincy-x (reef) - FAILED Laura, approved? > upgrade/reef-p2p - FAILED Laura, approved? > > (powercycle - FAILED Brad, approved) > ceph-volume - Approved Guillaume? > > On Wed, Feb 11, 2026 at 12:10 PM Laura Flores via Dev <dev(a)ceph.io> wrote: > > > The rados suite is approved: > > https://tracker.ceph.com/projects/rados/wiki/REEF#reef-v1828-httpstrackerce… > > > > On Tue, Feb 10, 2026 at 6:33 PM Brad Hubbard via Dev <dev(a)ceph.io> wrote: > > > >> Powercycle approved. > >> > >> On Tue, Feb 10, 2026 at 10:35 AM Brad Hubbard via Dev <dev(a)ceph.io> > >> wrote: > >> > >>> > >>> > >>> On Fri, Feb 6, 2026 at 12:36 PM Brad Hubbard <bhubbard(a)redhat.com> > >>> wrote: > >>> > >>>> > >>>> > >>>> On Fri, Feb 6, 2026 at 8:33 AM Brad Hubbard via ceph-users < > >>>> ceph-users(a)ceph.io> wrote: > >>>> > >>>>> On Fri, Feb 6, 2026 at 7:59 AM Yuri Weinstein via ceph-users < > >>>>> ceph-users(a)ceph.io> wrote: > >>>>> > >>>>> > We are continuing on this release. > >>>>> > > >>>>> > > >>>>> > powercycle - FAILED Brad, approved? > >>>>> > > >>>>> > >>>>> Unfortunately, I don't see any logs anywhere where I have access to > >>>>> them so > >>>>> I have no way of checking this? > >>>>> > >>>> > >>>> OK, I have the latest logs. The three failing jobs appear to have all > >>>> timed out after running for about six hours. At this time I don't know why > >>>> but I continue to investigate. > >>>> > >>> > >>> One point of commonality of the three tests that failed is that they are > >>> all running tests using ceph fuse. I'll reach out to the FS guys to see if > >>> they can help me get to the bottom of these failures. > >>> > >>> > >>>> > >>>> > >>>>> > >>>>> > >>>>> > ceph-volume - Approved Guillaume? > >>>>> > > >>>>> > Patrick was reviewing some tests, please see his comments in the same > >>>>> > tracker. > >>>>> > TIA > >>>>> > > >>>>> > > >>>>> > On Sat, Dec 13, 2025 at 1:19 AM Ilya Dryomov via ceph-users < > >>>>> > ceph-users(a)ceph.io> wrote: > >>>>> > > >>>>> > > On Mon, Dec 8, 2025 at 11:31 PM Yuri Weinstein via Dev < > >>>>> dev(a)ceph.io> > >>>>> > > wrote: > >>>>> > > > > >>>>> > > > This test branch was built loner then 2.5 weeks ago > >>>>> > > > I have to rebuild it first and then will schedule krbd > >>>>> > > > >>>>> > > krbd approved based on > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > >>>>> https://pulpito.ceph.com/yuriw-2025-12-10_21:26:56-krbd-reef-release-testin… > >>>>> > > > >>>>> > > > >>>>> > > >>>>> https://pulpito.ceph.com/yuriw-2025-12-12_18:20:32-krbd-reef-release-testin… > >>>>> > > > >>>>> > > Thanks, > >>>>> > > > >>>>> > > Ilya > >>>>> > > > >>>>> > > > > >>>>> > > > On Mon, Dec 8, 2025 at 1:39 PM Ilya Dryomov via ceph-users < > >>>>> > > ceph-users(a)ceph.io> wrote: > >>>>> > > >> > >>>>> > > >> On Mon, Dec 8, 2025 at 8:50 PM Ilya Dryomov <idryomov(a)gmail.com > >>>>> > > >>>>> > wrote: > >>>>> > > >> > > >>>>> > > >> > On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users > >>>>> > > >> > <ceph-users(a)ceph.io> wrote: > >>>>> > > >> > > > >>>>> > > >> > > Hi Yuri, > >>>>> > > >> > > > >>>>> > > >> > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein < > >>>>> > yweinste(a)redhat.com> > >>>>> > > wrote: > >>>>> > > >> > > > > >>>>> > > >> > > > Note: this is an attempt to test and publish post EOL > >>>>> reef > >>>>> > point > >>>>> > > release > >>>>> > > >> > > > > >>>>> > > >> > > > Details of this release are summarized here: > >>>>> > > >> > > > > >>>>> > > >> > > > https://tracker.ceph.com/issues/73906#note-1 > >>>>> > > >> > > > Release Notes - TBD > >>>>> > > >> > > > LRC upgrade - N/A > >>>>> > > >> > > > > >>>>> > > >> > > > distro => jammy focal centos9 windows bookworm > >>>>> > > >> > > > > >>>>> > > >> > > > Seeking approvals/reviews for: > >>>>> > > >> > > > > >>>>> > > >> > > > smoke - Laura approved? > >>>>> > > >> > > > > >>>>> > > >> > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam > >>>>> King > >>>>> > > approved? > >>>>> > > >> > > > (rados is also running on build 2 == build 1 + > >>>>> > > >> > > > https://github.com/ceph/ceph/pull/66056 per Igor's > >>>>> request) > >>>>> > > >> > > > > >>>>> > > >> > > > rgw - Adam E approved? > >>>>> > > >> > > > > >>>>> > > >> > > > fs - Venky approved? > >>>>> > > >> > > > >>>>> > > >> > > There are lot many failures in the fs suite resembling > >>>>> > > >> > > > >>>>> > > >> > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: > >>>>> wrong > >>>>> > fs > >>>>> > > >> > > type, bad option ... > >>>>> > > >> > > > >>>>> > > >> > > which is an issue with the testing kernel. Adding Alex who > >>>>> was > >>>>> > > looking > >>>>> > > >> > > into this issue. > >>>>> > > >> > > > >>>>> > > >> > > As far as the run validation is concerned, it's risky to > >>>>> ignore > >>>>> > the > >>>>> > > >> > > large number of failures. We have two options: > >>>>> > > >> > > > >>>>> > > >> > > - fix the testing kernel issue and rerun > >>>>> > > >> > > - rerun with `for-linus` branch (with `-k for-linus` in the > >>>>> > > >> > > teuthology-suite schedule command) > >>>>> > > >> > > > >>>>> > > >> > > The latter would be using an older kernel branch, but I > >>>>> think it > >>>>> > > >> > > should suffice for this case. > >>>>> > > >> > > >>>>> > > >> > Hi Venky, > >>>>> > > >> > > >>>>> > > >> > With your report of the fs suite being affected as well, I > >>>>> carved > >>>>> > out > >>>>> > > >> > some time to take a look. > >>>>> > > >> > > >>>>> > > >> > I think it's happening because XFS maintainers disabled > >>>>> support for > >>>>> > V4 > >>>>> > > >> > filesystems that was deprecated 5 years ago -- as of 6.18 > >>>>> kernel > >>>>> > it's > >>>>> > > >> > no longer compiled in by default. The proper fix would be to > >>>>> create > >>>>> > > an > >>>>> > > >> > image with/move to V5 on-disk format; I have opened [1] to > >>>>> try to > >>>>> > > >> > achieve that. We can also re-enable V4 on-disk format in our > >>>>> kernel > >>>>> > > >> > builds as a workaround. > >>>>> > > >> > > >>>>> > > >> > [1] https://github.com/ceph/ceph-cm-ansible/pull/799 > >>>>> > > >> > >>>>> > > >> This should be fixed now. Yuri, please schedule a rerun for > >>>>> krbd. > >>>>> > > >> > >>>>> > > >> Thanks, > >>>>> > > >> > >>>>> > > >> Ilya > >>>>> > > >> _______________________________________________ > >>>>> > > >> ceph-users mailing list -- ceph-users(a)ceph.io > >>>>> > > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >>>>> > > > > >>>>> > > > _______________________________________________ > >>>>> > > > Dev mailing list -- dev(a)ceph.io > >>>>> > > > To unsubscribe send an email to dev-leave(a)ceph.io > >>>>> > > _______________________________________________ > >>>>> > > ceph-users mailing list -- ceph-users(a)ceph.io > >>>>> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >>>>> > > > >>>>> > _______________________________________________ > >>>>> > ceph-users mailing list -- ceph-users(a)ceph.io > >>>>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >>>>> > > >>>>> > >>>>> > >>>>> -- > >>>>> Cheers, > >>>>> Brad > >>>>> _______________________________________________ > >>>>> ceph-users mailing list -- ceph-users(a)ceph.io > >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >>>>> > >>>> > >>>> > >>>> -- > >>>> Cheers, > >>>> Brad > >>>> > >>> > >>> > >>> -- > >>> Cheers, > >>> Brad > >>> _______________________________________________ > >>> Dev mailing list -- dev(a)ceph.io > >>> To unsubscribe send an email to dev-leave(a)ceph.io > >>> > >> > >> > >> -- > >> Cheers, > >> Brad > >> _______________________________________________ > >> Dev mailing list -- dev(a)ceph.io > >> To unsubscribe send an email to dev-leave(a)ceph.io > >> > > > > > > -- > > > > Laura Flores > > > > She/Her/Hers > > > > Software Engineer, Ceph Storage <https://ceph.io> > > > > Chicago, IL > > > > lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> > > M: +17087388804 > > > > > > _______________________________________________ > > Dev mailing list -- dev(a)ceph.io > > To unsubscribe send an email to dev-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Cheers, Venky

1 1

Re: 8443: connect: connection refused
by Eugen Block 23 Feb '26

23 Feb '26

It could be, yes. But maybe you could add more details to the story so we could get a full picture. In the first post you didn't mention nginx, I assume you're trying to use the mgmt-gateway? Zitat von listy via ceph-users <ceph-users(a)ceph.io>: > -> $ ceph mgr services > { > "dashboard": "http://10.1.1.63:8080/", > "prometheus": "http://10.1.1.63:9283/" > } > > Could the fact that I use - or rather try to for cannot figure out > configs - nginx-ha for accesing web-dashboard be the culprit here? > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: 8443: connect: connection refused
by listy 23 Feb '26

23 Feb '26

-> $ ceph mgr services { "dashboard": "http://10.1.1.63:8080/", "prometheus": "http://10.1.1.63:9283/" } Could the fact that I use - or rather try to for cannot figure out configs - nginx-ha for accesing web-dashboard be the culprit here?

1 0

Re: Help recovering OSD: rocksdb: submit_common error: Corruption: block checksum mismatch
by Igor Fedotov 21 Feb '26

21 Feb '26

Hi Theo, you might want to try to use PG export/import using ceph-objectstore-tool. Please find more details here https://www.croit.io/blog/how-to-recover-inactive-pgs-using-ceph-objectstor… <https://www.croit.io/blog/how-to-recover-inactive-pgs-using-ceph-objectstor…> Thanks, Igor On 03/02/2026 02:38, Theo Cabrerizo Diem via ceph-users wrote: > :12:18.895+0000 7f0c543eac00 -1 bluestore(/var/lib/ceph/osd) > fsck error: free extent 0x1714c521000~978b26df000 intersects allocatedblocks > fsck status: remaining 1 error(s) and warning(s)

2 2

Re: reef 18.2.8 QE validation status
by Ilya Dryomov 20 Feb '26

20 Feb '26

On Thu, Feb 19, 2026 at 9:00 PM Laura Flores via Dev <dev(a)ceph.io> wrote: > I reviewed the upgrade suites. There were some infrastructure problems > that affected a significant number of the results, which I think we should > get fixed before we approve the upgrades. > > @David Galloway <david.galloway(a)ibm.com>, the main issue is that we are > missing packages/containers for Quincy and Pacific. See details in > https://tracker.ceph.com/issues/75034 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F…> > . > > The reef point-to-point tests did not have any infrastructure failures, > but there were two new bugs that I raised for RBD and the Dashboard. @Ilya > Dryomov <idryomov(a)ibm.com> and @afreen@ibm.com <afreen(a)ibm.com>, can you > take a look? > > *upgrade:quincy-x* > 1. https://tracker.ceph.com/issues/75034 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F…> - > Missing packages and container images in upgrade tests - (Infrastructure) > > *upgrade:pacific-x* > 1. https://tracker.ceph.com/issues/75034 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F…> - > Missing packages and container images in upgrade tests - (Infrastructure) > > *upgrade:reef-p2p* > 1. https://tracker.ceph.com/issues/75037 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F…> - > src/test/pybind/test_rbd.py::test_remove_canceled fails from coredump - > (rbd) > Hi Laura, I think this is the same thing that I have noted in an earlier email in this thread: > [...] but the crashes in test_librbd_python.sh are concerning. > Patrick and I discussed them briefly -- they seem to be specific to > 20.04, but I don't remember seeing them before. There were multiple > passes on 20.04 in the run that I approved last year [2], so it looks > like something around Python, Cython or the build process has changed > in the interim. > > [1] https://pulpito.ceph.com/dis-2026-02-02_19:37:16-rbd-reef-release-distro-de… > [2] https://pulpito.ceph.com/yuriw-2025-11-18_23:00:18-rbd-reef-release-distro-… -- Ilya

1 1

Re: ceph squid 19.2.2 - modifyng HDD OSD to use NVME DB
by Steven Vacaroaia 20 Feb '26

20 Feb '26

Thanks for the link and the details I have not installed cephadm and respectively ceph-volume on other hosts but the controller I am running Ubuntu 24.04.2 and noticed the versions available for both cephadm and ceph-volume are either 19.2.3 or 19.2.0 - no 19.2.2.?? On the controller, I have 19.2.2 Should I install 19.2.3 or 19.2.0 ? Steven On Fri, 20 Feb 2026 at 08:29, Dario Graña via ceph-users <ceph-users(a)ceph.io> wrote: > I used the approach recommended by Eugen a few months ago and it worked > well. > > I made a small change to the lv names, instead of osd.0 in > # Create Logical Volume > ceph:~ # lvcreate -L 5G -n ceph-osd0-db ceph-db > > I used > ceph:~ # lvcreate -L 5G -n ceph-db-${OSD_FSID} ceph-db > > where ${OSD_FSID} is the following > # Get the OSD's FSID > [ceph: root@ceph /]# OSD_FSID=$(ceph-volume lvm list 0 | awk '/osd fsid/ > {print $3}') > fb69ba54-4d56-4c90-a855-6b350d186df5 > > > > On Thu, Feb 19, 2026 at 8:35 PM Eugen Block via ceph-users < > ceph-users(a)ceph.io> wrote: > > > Maybe this helps: > > > > > > > https://heiterbiswolkig.blogs.nde.ag/2025/02/05/cephadm-migrate-db-wal-to-n… > > > > Note that cephadm shell is always executed locally on the host, not > > remotely, there’s no ‚--host‘ parameter. > > > > Zitat von Steven Vacaroaia via ceph-users <ceph-users(a)ceph.io>: > > > > > Hi, > > > > > > I have a 7 nodes cluster deployed using cephadm with a mixt of HDD, SSD > > and > > > NVME > > > > > > I would like to reconfigure some of the HDD OSDs and put their DB on > NVME > > > There are only 6 x18 TB OSDs and there is a dedicate , "empty" 1.6 TB > > NVME > > > installed for this purpose > > > > > > The cluster is configured as EC 4+2 and is not under heavy use > > > > > > I would appreciate if you could advise what is the best way of > achieving > > > that > > > > > > As far as I was able to find, there are these 2 options > > > > > > 1. remove and redeploy OSDs > > > 2. use ceph-volume > > > > > > Option 1 is pretty straight forward but more time consuming > > > > > > For Option2 though I was not able to find any clear instructions > > > > > > The first thing I stumbled one was how to use ceph-volume in a > > > cephadm managed cluster on a host that does not have it > > > > > > ( "cephadm shell --host HOSTNAME -- ceph-volume ls" > > > is complaining about "unrecognized arguments --host") > > > > > > Any help,links,suggestions will be greatly appreciated > > > > > > Steven > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: ceph squid 19.2.2 - modifyng HDD OSD to use NVME DB
by Dario Graña 20 Feb '26

20 Feb '26

I used the approach recommended by Eugen a few months ago and it worked well. I made a small change to the lv names, instead of osd.0 in # Create Logical Volume ceph:~ # lvcreate -L 5G -n ceph-osd0-db ceph-db I used ceph:~ # lvcreate -L 5G -n ceph-db-${OSD_FSID} ceph-db where ${OSD_FSID} is the following # Get the OSD's FSID [ceph: root@ceph /]# OSD_FSID=$(ceph-volume lvm list 0 | awk '/osd fsid/ {print $3}') fb69ba54-4d56-4c90-a855-6b350d186df5 On Thu, Feb 19, 2026 at 8:35 PM Eugen Block via ceph-users < ceph-users(a)ceph.io> wrote: > Maybe this helps: > > > https://heiterbiswolkig.blogs.nde.ag/2025/02/05/cephadm-migrate-db-wal-to-n… > > Note that cephadm shell is always executed locally on the host, not > remotely, there’s no ‚--host‘ parameter. > > Zitat von Steven Vacaroaia via ceph-users <ceph-users(a)ceph.io>: > > > Hi, > > > > I have a 7 nodes cluster deployed using cephadm with a mixt of HDD, SSD > and > > NVME > > > > I would like to reconfigure some of the HDD OSDs and put their DB on NVME > > There are only 6 x18 TB OSDs and there is a dedicate , "empty" 1.6 TB > NVME > > installed for this purpose > > > > The cluster is configured as EC 4+2 and is not under heavy use > > > > I would appreciate if you could advise what is the best way of achieving > > that > > > > As far as I was able to find, there are these 2 options > > > > 1. remove and redeploy OSDs > > 2. use ceph-volume > > > > Option 1 is pretty straight forward but more time consuming > > > > For Option2 though I was not able to find any clear instructions > > > > The first thing I stumbled one was how to use ceph-volume in a > > cephadm managed cluster on a host that does not have it > > > > ( "cephadm shell --host HOSTNAME -- ceph-volume ls" > > is complaining about "unrecognized arguments --host") > > > > Any help,links,suggestions will be greatly appreciated > > > > Steven > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

rgw squid - reshard stale-instances in multisite environment
by Boris 20 Feb '26

20 Feb '26

Hi, we have a multisite environment, which only syncs metadata. Today I checked radosgw-admin reshard stale-instances list which brought up 29k entries. I've checked some randomly and those bucket_ids don't exist anymore. Is there a way to clean it up, or do I just leave it there and ignore it? I always read that this should not be touched in multisite environments. Cheers Boris

1 0

rgw reshard bucket to 20k shards?
by Boris 20 Feb '26

20 Feb '26

Hi, we have a bucket with 300m versioned objects and the autosharding stopped at 1999 (configured limit). We now have shards with 200k - 900k objects :/ What happens if we change the amount of shards to 19997? Does this even do anything? -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

Re: reef 18.2.8 QE validation status
by Laura Flores 19 Feb '26

19 Feb '26

I reviewed the upgrade suites. There were some infrastructure problems that affected a significant number of the results, which I think we should get fixed before we approve the upgrades. @David Galloway <david.galloway(a)ibm.com>, the main issue is that we are missing packages/containers for Quincy and Pacific. See details in https://tracker.ceph.com/issues/75034. The reef point-to-point tests did not have any infrastructure failures, but there were two new bugs that I raised for RBD and the Dashboard. @Ilya Dryomov <idryomov(a)ibm.com> and @afreen@ibm.com <afreen(a)ibm.com>, can you take a look? *upgrade:quincy-x* 1. https://tracker.ceph.com/issues/75034 - Missing packages and container images in upgrade tests - (Infrastructure) *upgrade:pacific-x* 1. https://tracker.ceph.com/issues/75034 - Missing packages and container images in upgrade tests - (Infrastructure) *upgrade:reef-p2p* 1. https://tracker.ceph.com/issues/75037 - src/test/pybind/test_rbd.py::test_remove_canceled fails from coredump - (rbd) 2. https://tracker.ceph.com/issues/75038 - Dashboard module and Prometheus module fail to load during upgrade test - (Dashboard) On Mon, Feb 16, 2026 at 4:40 AM Venky Shankar via ceph-users < ceph-users(a)ceph.io> wrote: > Hi Yuri, > > On Thu, Feb 12, 2026 at 9:30 PM Yuri Weinstein via ceph-users > <ceph-users(a)ceph.io> wrote: > > > > Still seeking approvals for: > > > > (smoke - FAILED, was reviewed and approved by Laura and Patrick, see > Slack > > thread for details) > > rados - FAILED, Afreen, Travis? ( Laura approved for core) > > > > rgw - FAILED, Adam E approved? > > > > (rbd, krbd - FAILED, Ilya approved) > > fs - FAILED, Venky approved? > > I wasn't able to finish reviewing the run and teuthology results are > unavailable right now due to the lab outage. > > > orch - FAILED, Adam K approved? > > > > upgrade/pacific-x (reef) - FAILED Laura, approved? > > upgrade/quincy-x (reef) - FAILED Laura, approved? > > upgrade/reef-p2p - FAILED Laura, approved? > > > > (powercycle - FAILED Brad, approved) > > ceph-volume - Approved Guillaume? > > > > On Wed, Feb 11, 2026 at 12:10 PM Laura Flores via Dev <dev(a)ceph.io> > wrote: > > > > > The rados suite is approved: > > > > https://tracker.ceph.com/projects/rados/wiki/REEF#reef-v1828-httpstrackerce… > > > > > > On Tue, Feb 10, 2026 at 6:33 PM Brad Hubbard via Dev <dev(a)ceph.io> > wrote: > > > > > >> Powercycle approved. > > >> > > >> On Tue, Feb 10, 2026 at 10:35 AM Brad Hubbard via Dev <dev(a)ceph.io> > > >> wrote: > > >> > > >>> > > >>> > > >>> On Fri, Feb 6, 2026 at 12:36 PM Brad Hubbard <bhubbard(a)redhat.com> > > >>> wrote: > > >>> > > >>>> > > >>>> > > >>>> On Fri, Feb 6, 2026 at 8:33 AM Brad Hubbard via ceph-users < > > >>>> ceph-users(a)ceph.io> wrote: > > >>>> > > >>>>> On Fri, Feb 6, 2026 at 7:59 AM Yuri Weinstein via ceph-users < > > >>>>> ceph-users(a)ceph.io> wrote: > > >>>>> > > >>>>> > We are continuing on this release. > > >>>>> > > > >>>>> > > > >>>>> > powercycle - FAILED Brad, approved? > > >>>>> > > > >>>>> > > >>>>> Unfortunately, I don't see any logs anywhere where I have access to > > >>>>> them so > > >>>>> I have no way of checking this? > > >>>>> > > >>>> > > >>>> OK, I have the latest logs. The three failing jobs appear to have > all > > >>>> timed out after running for about six hours. At this time I don't > know why > > >>>> but I continue to investigate. > > >>>> > > >>> > > >>> One point of commonality of the three tests that failed is that they > are > > >>> all running tests using ceph fuse. I'll reach out to the FS guys to > see if > > >>> they can help me get to the bottom of these failures. > > >>> > > >>> > > >>>> > > >>>> > > >>>>> > > >>>>> > > >>>>> > ceph-volume - Approved Guillaume? > > >>>>> > > > >>>>> > Patrick was reviewing some tests, please see his comments in the > same > > >>>>> > tracker. > > >>>>> > TIA > > >>>>> > > > >>>>> > > > >>>>> > On Sat, Dec 13, 2025 at 1:19 AM Ilya Dryomov via ceph-users < > > >>>>> > ceph-users(a)ceph.io> wrote: > > >>>>> > > > >>>>> > > On Mon, Dec 8, 2025 at 11:31 PM Yuri Weinstein via Dev < > > >>>>> dev(a)ceph.io> > > >>>>> > > wrote: > > >>>>> > > > > > >>>>> > > > This test branch was built loner then 2.5 weeks ago > > >>>>> > > > I have to rebuild it first and then will schedule krbd > > >>>>> > > > > >>>>> > > krbd approved based on > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > >>>>> > https://pulpito.ceph.com/yuriw-2025-12-10_21:26:56-krbd-reef-release-testin… > > >>>>> > > > > >>>>> > > > > >>>>> > > > >>>>> > https://pulpito.ceph.com/yuriw-2025-12-12_18:20:32-krbd-reef-release-testin… > > >>>>> > > > > >>>>> > > Thanks, > > >>>>> > > > > >>>>> > > Ilya > > >>>>> > > > > >>>>> > > > > > >>>>> > > > On Mon, Dec 8, 2025 at 1:39 PM Ilya Dryomov via ceph-users < > > >>>>> > > ceph-users(a)ceph.io> wrote: > > >>>>> > > >> > > >>>>> > > >> On Mon, Dec 8, 2025 at 8:50 PM Ilya Dryomov < > idryomov(a)gmail.com > > >>>>> > > > >>>>> > wrote: > > >>>>> > > >> > > > >>>>> > > >> > On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via > ceph-users > > >>>>> > > >> > <ceph-users(a)ceph.io> wrote: > > >>>>> > > >> > > > > >>>>> > > >> > > Hi Yuri, > > >>>>> > > >> > > > > >>>>> > > >> > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein < > > >>>>> > yweinste(a)redhat.com> > > >>>>> > > wrote: > > >>>>> > > >> > > > > > >>>>> > > >> > > > Note: this is an attempt to test and publish post EOL > > >>>>> reef > > >>>>> > point > > >>>>> > > release > > >>>>> > > >> > > > > > >>>>> > > >> > > > Details of this release are summarized here: > > >>>>> > > >> > > > > > >>>>> > > >> > > > https://tracker.ceph.com/issues/73906#note-1 > > >>>>> > > >> > > > Release Notes - TBD > > >>>>> > > >> > > > LRC upgrade - N/A > > >>>>> > > >> > > > > > >>>>> > > >> > > > distro => jammy focal centos9 windows bookworm > > >>>>> > > >> > > > > > >>>>> > > >> > > > Seeking approvals/reviews for: > > >>>>> > > >> > > > > > >>>>> > > >> > > > smoke - Laura approved? > > >>>>> > > >> > > > > > >>>>> > > >> > > > rados - Radek, Laura approved? Travis? Nizamudeen? > Adam > > >>>>> King > > >>>>> > > approved? > > >>>>> > > >> > > > (rados is also running on build 2 == build 1 + > > >>>>> > > >> > > > https://github.com/ceph/ceph/pull/66056 per Igor's > > >>>>> request) > > >>>>> > > >> > > > > > >>>>> > > >> > > > rgw - Adam E approved? > > >>>>> > > >> > > > > > >>>>> > > >> > > > fs - Venky approved? > > >>>>> > > >> > > > > >>>>> > > >> > > There are lot many failures in the fs suite resembling > > >>>>> > > >> > > > > >>>>> > > >> > > Error mounting /var/lib/ceph: mount: > /var/lib/ceph: > > >>>>> wrong > > >>>>> > fs > > >>>>> > > >> > > type, bad option ... > > >>>>> > > >> > > > > >>>>> > > >> > > which is an issue with the testing kernel. Adding Alex > who > > >>>>> was > > >>>>> > > looking > > >>>>> > > >> > > into this issue. > > >>>>> > > >> > > > > >>>>> > > >> > > As far as the run validation is concerned, it's risky to > > >>>>> ignore > > >>>>> > the > > >>>>> > > >> > > large number of failures. We have two options: > > >>>>> > > >> > > > > >>>>> > > >> > > - fix the testing kernel issue and rerun > > >>>>> > > >> > > - rerun with `for-linus` branch (with `-k for-linus` in > the > > >>>>> > > >> > > teuthology-suite schedule command) > > >>>>> > > >> > > > > >>>>> > > >> > > The latter would be using an older kernel branch, but I > > >>>>> think it > > >>>>> > > >> > > should suffice for this case. > > >>>>> > > >> > > > >>>>> > > >> > Hi Venky, > > >>>>> > > >> > > > >>>>> > > >> > With your report of the fs suite being affected as well, I > > >>>>> carved > > >>>>> > out > > >>>>> > > >> > some time to take a look. > > >>>>> > > >> > > > >>>>> > > >> > I think it's happening because XFS maintainers disabled > > >>>>> support for > > >>>>> > V4 > > >>>>> > > >> > filesystems that was deprecated 5 years ago -- as of 6.18 > > >>>>> kernel > > >>>>> > it's > > >>>>> > > >> > no longer compiled in by default. The proper fix would > be to > > >>>>> create > > >>>>> > > an > > >>>>> > > >> > image with/move to V5 on-disk format; I have opened [1] to > > >>>>> try to > > >>>>> > > >> > achieve that. We can also re-enable V4 on-disk format in > our > > >>>>> kernel > > >>>>> > > >> > builds as a workaround. > > >>>>> > > >> > > > >>>>> > > >> > [1] https://github.com/ceph/ceph-cm-ansible/pull/799 > > >>>>> > > >> > > >>>>> > > >> This should be fixed now. Yuri, please schedule a rerun for > > >>>>> krbd. > > >>>>> > > >> > > >>>>> > > >> Thanks, > > >>>>> > > >> > > >>>>> > > >> Ilya > > >>>>> > > >> _______________________________________________ > > >>>>> > > >> ceph-users mailing list -- ceph-users(a)ceph.io > > >>>>> > > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >>>>> > > > > > >>>>> > > > _______________________________________________ > > >>>>> > > > Dev mailing list -- dev(a)ceph.io > > >>>>> > > > To unsubscribe send an email to dev-leave(a)ceph.io > > >>>>> > > _______________________________________________ > > >>>>> > > ceph-users mailing list -- ceph-users(a)ceph.io > > >>>>> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >>>>> > > > > >>>>> > _______________________________________________ > > >>>>> > ceph-users mailing list -- ceph-users(a)ceph.io > > >>>>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >>>>> > > > >>>>> > > >>>>> > > >>>>> -- > > >>>>> Cheers, > > >>>>> Brad > > >>>>> _______________________________________________ > > >>>>> ceph-users mailing list -- ceph-users(a)ceph.io > > >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >>>>> > > >>>> > > >>>> > > >>>> -- > > >>>> Cheers, > > >>>> Brad > > >>>> > > >>> > > >>> > > >>> -- > > >>> Cheers, > > >>> Brad > > >>> _______________________________________________ > > >>> Dev mailing list -- dev(a)ceph.io > > >>> To unsubscribe send an email to dev-leave(a)ceph.io > > >>> > > >> > > >> > > >> -- > > >> Cheers, > > >> Brad > > >> _______________________________________________ > > >> Dev mailing list -- dev(a)ceph.io > > >> To unsubscribe send an email to dev-leave(a)ceph.io > > >> > > > > > > > > > -- > > > > > > Laura Flores > > > > > > She/Her/Hers > > > > > > Software Engineer, Ceph Storage <https://ceph.io> > > > > > > Chicago, IL > > > > > > lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> > > > M: +17087388804 > > > > > > > > > _______________________________________________ > > > Dev mailing list -- dev(a)ceph.io > > > To unsubscribe send an email to dev-leave(a)ceph.io > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > -- > Cheers, > Venky > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

Re: ceph squid 19.2.2 - modifyng HDD OSD to use NVME DB
by Eugen Block 19 Feb '26

19 Feb '26

Maybe this helps: https://heiterbiswolkig.blogs.nde.ag/2025/02/05/cephadm-migrate-db-wal-to-n… Note that cephadm shell is always executed locally on the host, not remotely, there’s no ‚--host‘ parameter. Zitat von Steven Vacaroaia via ceph-users <ceph-users(a)ceph.io>: > Hi, > > I have a 7 nodes cluster deployed using cephadm with a mixt of HDD, SSD and > NVME > > I would like to reconfigure some of the HDD OSDs and put their DB on NVME > There are only 6 x18 TB OSDs and there is a dedicate , "empty" 1.6 TB NVME > installed for this purpose > > The cluster is configured as EC 4+2 and is not under heavy use > > I would appreciate if you could advise what is the best way of achieving > that > > As far as I was able to find, there are these 2 options > > 1. remove and redeploy OSDs > 2. use ceph-volume > > Option 1 is pretty straight forward but more time consuming > > For Option2 though I was not able to find any clear instructions > > The first thing I stumbled one was how to use ceph-volume in a > cephadm managed cluster on a host that does not have it > > ( "cephadm shell --host HOSTNAME -- ceph-volume ls" > is complaining about "unrecognized arguments --host") > > Any help,links,suggestions will be greatly appreciated > > Steven > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

ceph squid 19.2.2 - modifyng HDD OSD to use NVME DB
by Steven Vacaroaia 19 Feb '26

19 Feb '26

Hi, I have a 7 nodes cluster deployed using cephadm with a mixt of HDD, SSD and NVME I would like to reconfigure some of the HDD OSDs and put their DB on NVME There are only 6 x18 TB OSDs and there is a dedicate , "empty" 1.6 TB NVME installed for this purpose The cluster is configured as EC 4+2 and is not under heavy use I would appreciate if you could advise what is the best way of achieving that As far as I was able to find, there are these 2 options 1. remove and redeploy OSDs 2. use ceph-volume Option 1 is pretty straight forward but more time consuming For Option2 though I was not able to find any clear instructions The first thing I stumbled one was how to use ceph-volume in a cephadm managed cluster on a host that does not have it ( "cephadm shell --host HOSTNAME -- ceph-volume ls" is complaining about "unrecognized arguments --host") Any help,links,suggestions will be greatly appreciated Steven

1 0

Re: 8443: connect: connection refused
by Eugen Block 18 Feb '26

18 Feb '26

Hi, port 8443 is the default dashboard port on the active MGR. If none of your hosts listens on that port, then your dashboard most likely isn't working. Is the cluster healthy? What does 'ceph mgr services' report (mask sensitive data)? The message you quote is the alertmanager, trying to connect to the dashboard so you would see the alerts in the dashboard (I assume). Zitat von listy via ceph-users <ceph-users(a)ceph.io>: > Hi guys. > > One node logs: > ... > 5.744Z caller=notify.go:848 level=warn component=dispatcher > receiver=ceph-dashboard integration=webhook[0] > aggrGroup="{}/{}:{alertname=\"CephPGBackfillAtRisk\"}" msg="Notify > attempt failed, will retry later" attempts=1 err="Post > \"<redacted>\": dial tcp 10.1.1.63:8443: connect: connection refused" > ... > but no node listen on this port and I have: > > -> $ ceph config get mgr mgr/dashboard/ssl > false > > What am I missing or having mis-configured? > > all thoughts are much appreciated, thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Debian version and ceph.
by Eugen Block 18 Feb '26

18 Feb '26

Right, if you want to bootstrap a cephadm cluster, you don't need to worry too much about the package versions on the host. Although we do recommend to keep them somewhat aligned. But there's also an option for the cephadm bootstrap command: --allow-mismatched-release allow bootstrap of ceph that doesn't match this version of cephadm I used that many times on openSUSE hosts where the latest available version was Pacific (there are packages available again with Reef and Squid from openSUSE Leap 16, Tumbleweed probably too, haven't checked). Last week I also bootstrapped a Squid cluster like that on a Debian Trixie where the host package versions are 18.2.7. Zitat von Robert Sander via ceph-users <ceph-users(a)ceph.io>: > Am 18.02.26 um 5:53 PM schrieb Albert Shih via ceph-users: > >> 2/ Install a Debian 13 and use the Debian ceph pkg. > > This is the way. > > The client code does not need to match the cluster version. > I usually install ceph-common from the distribution. > > Just make sure you have a cephadm binary matching the desired Ceph > version before bootstrapping the cluster. > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

8443: connect: connection refused
by listy 18 Feb '26

18 Feb '26

Hi guys. One node logs: ... 5.744Z caller=notify.go:848 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] aggrGroup="{}/{}:{alertname=\"CephPGBackfillAtRisk\"}" msg="Notify attempt failed, will retry later" attempts=1 err="Post \"<redacted>\": dial tcp 10.1.1.63:8443: connect: connection refused" ... but no node listen on this port and I have: -> $ ceph config get mgr mgr/dashboard/ssl false What am I missing or having mis-configured? all thoughts are much appreciated, thanks, L.

1 0

Re: Debian version and ceph.
by Robert Sander 18 Feb '26

18 Feb '26

Am 18.02.26 um 5:53 PM schrieb Albert Shih via ceph-users: > 2/ Install a Debian 13 and use the Debian ceph pkg. This is the way. The client code does not need to match the cluster version. I usually install ceph-common from the distribution. Just make sure you have a cephadm binary matching the desired Ceph version before bootstrapping the cluster. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Debian version and ceph.
by Albert Shih 18 Feb '26

18 Feb '26

Hi everyone. I need to install a new cluster ceph. New hardware mean of couse new fresh install. Currently I still don't see any package for Debian trixie. In any case the ceph would be install with cephadm with podman, so no huge dependencies with the OS. But still ceph-base ceph-common libcephfs2 python3-ceph python3-ceph-argparse python3-ceph-common python3-cephfs In that case what would be advise to do ? (I want to stay with Debian). I see 1/ Install a Debian 12 and use the official ceph.com pkg. 2/ Install a Debian 13 and use the Debian ceph pkg. What would you do ? Regards. -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mer. 18 févr. 2026 17:44:51 CET

1 0

Re: Issue with QEMU clones and excessive PG counts on Ceph Squid (19.2.3)
by Konold, Martin 18 Feb '26

18 Feb '26

Am 2026-02-17 20:24, schrieb Robert Lukan via ceph-users: Hi Robert, Are there known risks to "stepping down" PGs in increments of, say, 10-20% at a time while the cluster is under load? 2. Reducing the number is best done via halveing not some aribtrary percentage in order to keep the number a power of 2. Given the 16k PG count, is there a specific "recovery/backfill" throttle we should prioritize to ensure VM performance isn't impacted during the merge process? The performance impact of reducing the number of PGs should be fine. Regards -- --martin konold ppa. Martin Konold -- Martin Konold - Prokurist, CTO KONSEC GmbH -⁠ make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany

1 0

Re: Issue with QEMU clones and excessive PG counts on Ceph Squid (19.2.3)
by Anthony D'Atri 18 Feb '26

18 Feb '26

> >> *Hardware:* Intel NVMe drives, Intel, or Solidigm? I highly recommend using SST (or if you got them from Dell, DSU) to update firmware. >> *Fails:* Cloning via QEMU (librbd). Check that your qemu / ProxMox defaults to enabling the layering, object-map, and fast-diff feature flags on RBD volumes. >> , the autoscaler increased the total PG >> count to 16k. We suspect this excessive PG count might be contributing to >> the cloning failures, though we aren't certain. I’m skeptical. >> We would like to safely decrease the number of PGs back to a standard range What do you consider a standard range? If you have one OSD deployed on each SSD, the PGS column from `ceph osd df ` should be in the 200-300 range. The autoscaler pgs_per_osd_target defaults to 100 which is rather low for most clusters, plus it’s actually a ceiling not a target. I suggest setting to 300 and mon_max_pgs_per_osd to 600. 16k PGs, assuming no other large pools and replica size=3, would thus be just smurfy across ….. ~~ 240 OSDs, plus or minus. How many do you have? What PGS values do you have? >> Decrease the PG count in small increments. The Manager does this for you since Nautilus I think it was. >> Monitor for I/O latency on the Intel NVMes. Remember that iostat %util isn’t great for that. And update your firmware! >> >> *Questions for the list:* >> >> 1. >> >> Are there known risks to "stepping down" PGs in increments of, say, >> 10-20% at a time while the cluster is under load? No, the Manager steps pg_num and pgp_num for you. Remember to only set the pg_num target to a power of two. >> 2. >> >> Given the 16k PG count, is there a specific "recovery/backfill" throttle >> we should prioritize to ensure VM performance isn't impacted during the >> merge process? osd_max_backfills <= 4 >> >> We are trying to decide if this approach is feasible or if we should create >> a new cluster and migrate VMs I doubt that would be necessary. >> . Any insights would be greatly appreciated. >> >> Thank you, >> >> Robert >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Issue with QEMU clones and excessive PG counts on Ceph Squid (19.2.3)
by Eugen Block 17 Feb '26

17 Feb '26

IIRC, pg merge is only done one PG at a time, so you have kind of a „natural“ throttling, but it will take quite some time to complete. What’s the output of ‚ceph osd df‘? Do you see ceph warnings? Do you have any more info on the qemu errors, either on client side or in ceph logs? I can’t comment on qemu clones, we only do that with rbd (manually) or let nova/glance/cinder handle it. Zitat von Robert Lukan via ceph-users <ceph-users(a)ceph.io>: > Hello Ceph Community, > > We are seeking advice regarding a strange behavior with cloning > functionality and a strategy for PG reduction on our current cluster. > > *Environment:* > > - > > *Ceph Version:* Squid 19.2.3 (via Proxmox) > - > > *Hardware:* Intel NVMe drives, with very good margin of resources (CPU, > RAM and bandwidth) > - > > *Workload:* Active VM storage - RBD only (a couple 100 VMs, but no > extreme load) > > *The Issue:* We are experiencing a failure with cloning functionality, but > it appears to be specific to which way we do it: > > - > > *Fails:* Cloning via QEMU (librbd). > - > > *Succeeds:* Cloning via rbd clone or deep-copy. > - > > *Succeeds:* Cloning when using krbd enabled on a pool > > This cluster was created about a year ago for testing(PoC) and has evolved > into production. Initially cloning worked fine. Due to disk replacements, > expansions and misconfiguration, the autoscaler increased the total PG > count to 16k. We suspect this excessive PG count might be contributing to > the cloning failures, though we aren't certain. > > We would like to safely decrease the number of PGs back to a standard range > without destabilizing the running VMs. Our plan is to: > > 1. > > Decrease the PG count in small increments. > 2. > > Monitor for I/O latency on the Intel NVMes. > > *Questions for the list:* > > 1. > > Are there known risks to "stepping down" PGs in increments of, say, > 10-20% at a time while the cluster is under load? > 2. > > Given the 16k PG count, is there a specific "recovery/backfill" throttle > we should prioritize to ensure VM performance isn't impacted during the > merge process? > > We are trying to decide if this approach is feasible or if we should create > a new cluster and migrate VMs. Any insights would be greatly appreciated. > > Thank you, > > Robert > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Issue with QEMU clones and excessive PG counts on Ceph Squid (19.2.3)
by Robert Lukan 17 Feb '26

17 Feb '26

Hello Ceph Community, We are seeking advice regarding a strange behavior with cloning functionality and a strategy for PG reduction on our current cluster. *Environment:* - *Ceph Version:* Squid 19.2.3 (via Proxmox) - *Hardware:* Intel NVMe drives, with very good margin of resources (CPU, RAM and bandwidth) - *Workload:* Active VM storage - RBD only (a couple 100 VMs, but no extreme load) *The Issue:* We are experiencing a failure with cloning functionality, but it appears to be specific to which way we do it: - *Fails:* Cloning via QEMU (librbd). - *Succeeds:* Cloning via rbd clone or deep-copy. - *Succeeds:* Cloning when using krbd enabled on a pool This cluster was created about a year ago for testing(PoC) and has evolved into production. Initially cloning worked fine. Due to disk replacements, expansions and misconfiguration, the autoscaler increased the total PG count to 16k. We suspect this excessive PG count might be contributing to the cloning failures, though we aren't certain. We would like to safely decrease the number of PGs back to a standard range without destabilizing the running VMs. Our plan is to: 1. Decrease the PG count in small increments. 2. Monitor for I/O latency on the Intel NVMes. *Questions for the list:* 1. Are there known risks to "stepping down" PGs in increments of, say, 10-20% at a time while the cluster is under load? 2. Given the 16k PG count, is there a specific "recovery/backfill" throttle we should prioritize to ensure VM performance isn't impacted during the merge process? We are trying to decide if this approach is feasible or if we should create a new cluster and migrate VMs. Any insights would be greatly appreciated. Thank you, Robert

1 0

Re: discard options for bdevs (OSDs)
by Igor Fedotov 17 Feb '26

17 Feb '26

Hi Robert, here is a brief overview on these parameters: bdev_enable_discard - flag to enable/disable trimming. Once enabled and depending on bdev_async_discard_threads parameter discard requests could be issued either synchronously or asynchronously. Each requests originates when OSD releases specific disk extent, e.g. when overwriting or removing an object (or part of it). In async mode requests are put into a queue and are processed by one of relevant threads. bdev_async_discard_threads - amount of threads to process discard requests asynchronously, if set to 0 and bdev_enable_discart is set to true - all the discard request are issued synchronously bdev_async_discard - this is a deprecated parameter, please don't use it I have no idea how this supposed to work with qcow2 files as the primary targets are SSD drives but in recent Ceph releases (e.g. v19.2.3 and upcoming v18.2.8) one can learn the amount of issued discard requests from OSD perf counters: ceph tell osd.0 perf dump | grep discard -B 1 "blk-kernel-device-bluestore": { "discard_op": 0, "discard_threads": 0 -- "blk-kernel-device-db": { "discard_op": 0, "discard_threads": 0 -- "blk-kernel-device-slow": { "discard_op": 0, "discard_threads": 0 -- "blk-kernel-device-wal": { "discard_op": 0, "discard_threads": 0 Hope this is helpful. Best regards, Igor On 17/02/2026 12:15, Robert Sander via ceph-users wrote: > Hi, > > what is the purpose and effect of these configuration options? > > bdev_async_discard > bdev_async_discard_threads > bdev_enable_discard > > They are not mentioned on docs.ceph.com. > > They read as they would enable a regular trim job on OSDs where the > hardware supports trimming/discarding. How often would this operation > run? > > On a virtual test cluster where the OSDs are backed with qcow2 files > and are presented to the VM as SSDs with discard they seem to have no > effect on the size of the qcow2 file. > > With a normal filesystem an fstrim operation will reduce the size of > the qcow2 file. > > Regards

1 0

Re: CephFS Metadata loss, data present, inodes in pool, scan_links finishes instantly - Squid
by Eugen Block 17 Feb '26

17 Feb '26

Hi, it's not really clear to me if you had the NVMe drives for the CephFS metadata pool only or if you had DB/WAL for spinners as well on those drives. I assume it was only the pool on them, regarding your statement that there are active+clean PGs. More comments inline... Zitat von Enzo Van Thorre via ceph-users <ceph-users(a)ceph.io>: > Hello Ceph users and readers, > > I am facing a really unknown situation for me (being kinda new) with > a CephFS cluster (Squid 19.2.2, Cephadm) where the data pool is > intact (7.2 TB used, 4.9TB stored), inode objects exist in the > recovered metadata pool, but when i try to mount the filesystem, its > empty. > > What initially happened is that I had two groups of drives: nvme’s > and spinning. I had separate pools for both, and I eventually > deleted the nvme’s pool and then removed the nvmes from the machine. > I never thought that they had included metadata for spinning rust’s > pool, but here we are. For the future: 'ceph osd ok-to-stop <ID>' can be helpful if you're not sure if it's okay to stop an OSD. That could have prevented it from happening. > > Version: 19.2.2 Squid > Metadata pool ID 1 (cephfs.cephfs.meta), Size=1 (No Replicas :(, yes > I know I messed up). yeah... > Data Pool ID 6 (ec_data), erasure coded with k=2 m=1. that's not great either (k2, m1) > All OSDs are up. 256 PGs are active+clean. there’s then quite a bit > that are inconsistent, degraded, undersized, remapped, and other > stuff. what is "other stuff"? A 'ceph health detail' could be helpful. > Here’s what I’ve done since losing the metadata pool due to OSD > purges. I tried to rebuild metadata from the data pool by doing: > 1. cephfs-data-scan init --force-init > 2. cephfs-data-scan scan_extents > 3. cephfs-data-scan scan_inodes > > > > ceph df shows 7.2 TiB stored in pool ec_data (which is the spinning > rust pool) > Inodes do exist. running rados -p cephfs.cephfs.meta ls | grep > "100." returns tons of objects (ex. 1000005f1f3.00000000). > If i try to mount via ceph-fuse, it shows an empty root. there is > no lost+found folder either. > > A big problem is that when I try to link the orphaned inodes to > lost+found, the command finishes instantly with no output, as if it > believes the cluster is clean. > > I run: cephfs-data-scan scan_links --filesystem cephfs > # and it returns code 0, no output, and runs in <1 second > > So my final question is since scan_inodes has populated the metadata > pool with objects (100.xxxx), but scan_links ignores them, how do I > force a re-scan or just do SOMETHING so scan_links will actually > link these files into lost+found? Unfortunately, I don't have a comment on that right now. But maybe increasing debug level for MDS can help here? > > All The Best, > Enzo > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 1

discard options for bdevs (OSDs)
by Robert Sander 17 Feb '26

17 Feb '26

Hi, what is the purpose and effect of these configuration options? bdev_async_discard bdev_async_discard_threads bdev_enable_discard They are not mentioned on docs.ceph.com. They read as they would enable a regular trim job on OSDs where the hardware supports trimming/discarding. How often would this operation run? On a virtual test cluster where the OSDs are backed with qcow2 files and are presented to the VM as SSDs with discard they seem to have no effect on the size of the qcow2 file. With a normal filesystem an fstrim operation will reduce the size of the qcow2 file. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Ceph on Windows
by Anthony Middleton 16 Feb '26

16 Feb '26

Hello, Ceph friends! Throughout 2025, Ceph technical leaders surveyed our community about their usage of Ceph on Windows. We received 55 responses to our inquiry. When asked whether their organization currently uses Ceph on Windows, 93% responded no, and 7% responded yes. When asked if their organization plans to use Ceph on Windows or CephFS in the future, 60% responded no, and 40% responded yes. Given the continued need for this project within our community, the Ceph Steering Committee (CSC) would like to ask for help in maintaining Ceph for Windows. If you're interested in contributing, please introduce yourself on the Ceph Development email list <https://lists.ceph.io/postorius/lists/dev.ceph.io/> and volunteer to help. This request is not a one-time effort. We aim to establish a dedicated committee for this project. Please let me or any CSC member know if you are interested in participating and what resources you might need to support your involvement. Best, -- Anthony Middleton Community Manager, Ceph Foundation community-manager(a)ceph.io LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: Bug #71083 (s3 DeleteBucketLifecycle does not delete lifecycle config), Squid backport and 19.2.4 release
by Sascha Lucas 16 Feb '26

16 Feb '26

Hi Florian, On Mon, 16 Feb 2026, Florian Haas via ceph-users wrote: > If I understand correctly, this is a bug that can potentially cause data > loss: if > > - an S3 bucket at some point had a lifecycle policy automating deletion of > expired objects, and > - that lifecycle policy was then deleted (which due to the bug in question > would be ignored), It seems the situation is not that bad. The delete-bucket-lifecycle API is ignored. But a careful user might verify the deletion with get-bucket-lifecycle-configuration and notice that it's still present ;-). As a workaround use "Status": "Disabled" in the lifecycle rules. +1 waiting for 19.2.4. Thanks, Sascha.

1 0

Re: mgmt-gateway and oauth2-proxy for radosgw?
by Daniel Parkes 16 Feb '26

16 Feb '26

Hi Florian, Both integrations will coexist. As noted, RGW provides a native implementation for authenticating against external OIDC providers that leverages a subset of the AWS IAM/STS API. Following this model offers significant advantages: - Granular Control: Enables robust RBAC (Role-Based Access Control) and ABAC (Attribute-Based Access Control). - Enhanced Security: Inherits the security benefits of the STS (Security Token Service) model, such as temporary credentials. - Ecosystem Compatibility: By adhering to AWS standards, we ensure seamless integration with the broader ecosystem, including Backup & Restore tools and Iceberg REST API catalogs, without additional configuration. Agreed, the initial setup for an RGW OIDC provider is a bit complex. We are looking to simplify this by integrating the setup workflow directly into the Ceph Dashboard soon. Regarding the *mgmt-gateway + oauth2-proxy*: In theory(not something I have tested), oauth2-proxy can act as the OIDC provider for RGW. By using AssumeRoleWithWebIdentity, users can authenticate to RGW using a JWT provided by oauth2-proxy. If your proxy is already linked to an IDP, it effectively bridges that identity to RGW. Not sure if this is a good idea, as you create dependencies, and the users for the dashboard (admins) and RGW (end users) are completely different. Also, how the attributes are minted in the JWT by oauth2-proxy for RGW to use would need to be checked. Regards.

1 0

Re: [rgw multiple zonegroups] ERROR: S3 error: 404 (NoSuchKey) while copy
by Boris 16 Feb '26

16 Feb '26

Hi Alexander, 1. I think you can skip the "--region", because that RGW daemon is usually attached to a zone. 2. I just tested it with our multisite setup (latest squid) and I always get a "301 (Moved Permanently)", when I use a bucket from another zonegroup. I did not manage to get "404 (NoSuchKey)" with any constellation I tested. Can you post a "radosgw-admin period get" from both hosts? Am Do., 12. Feb. 2026 um 15:29 Uhr schrieb Alexander NA via ceph-users < ceph-users(a)ceph.io>: > Hello everyone. > I'm trying to set up one realm ("r1"), two zone groups ("zg1" - master, > "zg2"), and one zone in each zone group ("z1" - master and "z2", > respectively). The first zone group and its zone are masters. > Endpoint "zg1" - 10.1.1.1 > Endpoint "zg2" - 10.1.1.2 > "zg1" has bucket "buc1" > "zg2" has bucket "buc2" > > If I copy the file like this: > s3cmd put test_file s3://buc1 --region=zg2 --host=10.1.1.2 > I get the correct error code (as per the documentation): > "ERROR: S3 error: 301 (Moved Permanently)" > > BUT, if I copy it like this: > s3cmd put test_file s3://buc2 --region=zg1 --host=10.1.1.1 > I get this code: > ERROR: S3 error: 404 (NoSuchKey) > > Can you point me to something that might be wrong? > Thanks > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 1

Bug #71083 (s3 DeleteBucketLifecycle does not delete lifecycle config), Squid backport and 19.2.4 release
by Florian Haas 16 Feb '26

16 Feb '26

Hello everyone, The fix for rgw bug #71083[1] was backported to the Squid branch back in July[2], almost immediately after the 19.2.3 release, and the backport was merged in September[3]. However, Squid hasn't had a point release since.[4] If I understand correctly, this is a bug that can potentially cause data loss: if - an S3 bucket at some point had a lifecycle policy automating deletion of expired objects, and - that lifecycle policy was then deleted (which due to the bug in question would be ignored), … then rgw would keep deleting data that's past its previously defined expiry date, and users would probably not find out until it's too late. Is that a correct reading of the problem? If so, can I ask someone in the know to please provide a rough indication as to when 19.2.4 might land, with the bug being resolved? Thanks! Cheers, Florian [1] https://tracker.ceph.com/issues/71083 [2] https://tracker.ceph.com/issues/71544 [3] https://github.com/ceph/ceph/pull/64741 [4] https://docs.ceph.com/en/latest/releases/

1 0

Re: Ceph MDS super slow
by Venky Shankar 16 Feb '26

16 Feb '26

Hi Martin, Could you additionally do a minute worth of MDS perf dump captures at 5 second intervals during the observed slowness? This can be done using > ceph tell mds.<id> perf dump and pipe the result to separate (time-stamped) files and share those. On Fri, Feb 13, 2026 at 4:59 PM Martin Gerhard Loschwitz via ceph-users <ceph-users(a)ceph.io> wrote: > > Folks, > > we are seeing very slow MDSes on a CephFS that has nearly nothing to do. Slow as in „200 metadata requests per second“ slow when putting a bit of load onto it. We managed to get a wallclock of the MDS: > > [root@node01 gdbpmp]# ./gdbpmp.py -i gdbpmp.data > > Thread: 7 (mds-rank-fin) - 17 samples > > + 100.00% clone > + 100.00% start_thread > + 100.00% Finisher::finisher_thread_entry() > + 94.12% MDSIOContextBase::complete > |+ 88.24% MDSRank::respawn > ||+ 88.24% Context::complete > || + 88.24% MDSDaemon::respawn > || + 88.24% ceph::logging::Log::dump_recent() > || + 52.94% ceph::logging::Log::_flush(std::vector<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >&, bool) > || |+ 23.53% __snprintf_chk > || ||+ 23.53% __vsnprintf_internal > || || + 17.65% __vfprintf_internal > || || + 5.88% __strchrnul_evex > || || + 5.88% _IO_default_xsputn > || |+ 17.65% ceph::logging::Log::_flush_logbuf() > || ||+ 17.65% ceph::logging::Log::_log_safe_write(std::basic_string_view<char, std::char_traits<char> >) > || || + 17.65% safe_write > || || + 17.65% write > || |+ 5.88% __tz_convert > || | + 5.88% __tzfile_compute > || | + 5.88% __strcmp_evex > || + 5.88% __memmove_avx512_unaligned_erms > |+ 5.88% std::lock_guard<ceph::fair_mutex>::lock_guard > | + 5.88% ceph::fair_mutex::lock > | + 5.88% std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&, ceph::fair_mutex::lock()::{lambda()#1}) > | + 5.88% std::condition_variable::wait(std::unique_lock<std::mutex>&) > | + 5.88% pthread_cond_wait@@GLIBC_2.3.2 > | + 5.88% __futex_abstimed_wait_common > + 5.88% std::condition_variable::wait(std::unique_lock<std::mutex>&) > + 5.88% pthread_cond_wait@@GLIBC_2.3.2 > + 5.88% __futex_abstimed_wait_common > > Thread: 10 (mds-rank-progr) - 17 samples > > + 100.00% clone > + 100.00% start_thread > + 100.00% MDSRank::ProgressThread::entry > + 100.00% std::_V2::condition_variable_any::wait<std::unique_lock<ceph::fair_mutex>, MDSRank::ProgressThread::entry()::<lambda()> > > + 100.00% std::_V2::condition_variable_any::wait<std::unique_lock<ceph::fair_mutex> > > + 88.24% std::condition_variable::wait(std::unique_lock<std::mutex>&) > |+ 88.24% pthread_cond_wait@@GLIBC_2.3.2 > | + 88.24% __futex_abstimed_wait_common > + 11.76% std::_V2::condition_variable_any::_Unlock<std::unique_lock<ceph::fair_mutex> >::~_Unlock > + 11.76% std::unique_lock<ceph::fair_mutex>::lock > + 11.76% ceph::fair_mutex::lock > + 11.76% std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&, ceph::fair_mutex::lock()::{lambda()#1}) > + 11.76% std::condition_variable::wait(std::unique_lock<std::mutex>&) > + 11.76% pthread_cond_wait@@GLIBC_2.3.2 > + 11.76% __futex_abstimed_wait_common > > Thread: 11 (mds-log-trim) - 17 samples > > + 100.00% clone > + 100.00% start_thread > + 100.00% execute_native_thread_routine > + 100.00% MDLog::log_trim_upkeep > + 100.00% std::_V2::condition_variable_any::wait_for<std::unique_lock<ceph::fair_mutex>, long, std::ratio<1l, 1000l> > > + 100.00% std::_V2::condition_variable_any::wait_until<std::unique_lock<ceph::fair_mutex>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > > + 82.35% std::_V2::condition_variable_any::_Unlock<std::unique_lock<ceph::fair_mutex> >::~_Unlock > |+ 82.35% std::unique_lock<ceph::fair_mutex>::lock > | + 82.35% ceph::fair_mutex::lock > | + 82.35% std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&, ceph::fair_mutex::lock()::{lambda()#1}) > | + 82.35% std::condition_variable::wait(std::unique_lock<std::mutex>&) > | + 82.35% pthread_cond_wait@@GLIBC_2.3.2 > | + 82.35% __futex_abstimed_wait_common > + 17.65% std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > > + 17.65% std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > > + 17.65% std::__condvar::wait_until > + 17.65% pthread_cond_clockwait(a)GLIBC_2.30 > + 17.65% __futex_abstimed_wait_common > > Thread: 12 (mds-cache-trim) - 17 samples > > + 100.00% clone > + 100.00% start_thread > + 100.00% execute_native_thread_routine > + 100.00% MDCache::upkeep_main > + 70.59% std::scoped_lock<ceph::fair_mutex>::scoped_lock > |+ 70.59% ceph::fair_mutex::lock > | + 70.59% std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&, ceph::fair_mutex::lock()::{lambda()#1}) > | + 70.59% std::condition_variable::wait(std::unique_lock<std::mutex>&) > | + 70.59% pthread_cond_wait@@GLIBC_2.3.2 > | + 70.59% __futex_abstimed_wait_common > + 23.53% std::condition_variable::wait_for<long, std::ratio<1l, 1000000000l> > > |+ 23.53% std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > > | + 23.53% std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > > | + 23.53% std::__condvar::wait_until > | + 23.53% pthread_cond_clockwait(a)GLIBC_2.30 > | + 23.53% __futex_abstimed_wait_common > + 5.88% MDCache::check_memory_usage > + 5.88% MemoryModel::sample > + 5.88% MemoryModel::_sample > + 5.88% std::getline<char, std::char_traits<char>, std::allocator<char> > > + 5.88% std::basic_istream<char, std::char_traits<char> >& std::getline<char, std::char_traits<char>, std::allocator<char> >(std::basic_istream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char > , std::char_traits<char>, std::allocator<char> >&, char) > + 5.88% std::basic_filebuf<char, std::char_traits<char> >::underflow() > + 5.88% std::__basic_file<char>::xsgetn(char*, long) > + 5.88% read > > > Thread: 15 (safe_timer) - 17 samples > > + 100.00% clone > + 100.00% start_thread > + 100.00% CommonSafeTimerThread<std::mutex>::entry() > + 100.00% CommonSafeTimer<std::mutex>::timer_thread() > + 88.24% pthread_cond_clockwait(a)GLIBC_2.30 > |+ 88.24% __futex_abstimed_wait_common > + 11.76% std::condition_variable::wait(std::unique_lock<std::mutex>&) > + 11.76% pthread_cond_wait@@GLIBC_2.3.2 > + 11.76% __futex_abstimed_wait_common > > Thread: 17 (safe_timer) - 17 samples > > + 100.00% clone > + 100.00% start_thread > + 100.00% CommonSafeTimerThread<ceph::fair_mutex>::entry() > + 100.00% CommonSafeTimer<ceph::fair_mutex>::timer_thread() > + 70.59% std::_V2::condition_variable_any::_Unlock<std::unique_lock<ceph::fair_mutex> >::~_Unlock() > |+ 70.59% std::condition_variable::wait(std::unique_lock<std::mutex>&) > | + 70.59% pthread_cond_wait@@GLIBC_2.3.2 > | + 70.59% __futex_abstimed_wait_common > + 29.41% pthread_cond_clockwait(a)GLIBC_2.30 > + 29.41% __futex_abstimed_wait_common > > > Thread: 20 (ms_dispatch) - 17 samples > > + 100.00% clone > + 100.00% start_thread > + 100.00% DispatchQueue::DispatchThread::entry() > + 100.00% DispatchQueue::entry() > + 100.00% MDSDaemon::ms_dispatch2 > + 94.12% std::lock_guard<ceph::fair_mutex>::lock_guard > |+ 94.12% ceph::fair_mutex::lock > | + 94.12% std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&, ceph::fair_mutex::lock()::{lambda()#1}) > | + 94.12% std::condition_variable::wait(std::unique_lock<std::mutex>&) > | + 94.12% pthread_cond_wait@@GLIBC_2.3.2 > | + 94.12% __futex_abstimed_wait_common > + 5.88% MDSRankDispatcher::ms_dispatch > + 5.88% MDSRank::_dispatch > + 5.88% MDSRank::_advance_queues > + 5.88% MDSContext::complete > + 5.88% Context::complete > + 5.88% Server::handle_client_readdir > + 5.88% CInode::encode_inodestat > + 5.88% encode<optmetadata_singleton<optmetadata_client_t<std::allocator>, std::allocator>, std::allocator> > + 5.88% optmetadata_multiton<optmetadata_singleton<optmetadata_client_t<std::allocator>, std::allocator>, std::allocator>::encode > + 5.88% ceph::buffer::v15_2_0::list::append_hole(unsigned int) > + 5.88% ceph::buffer::v15_2_0::list::refill_append_space(unsigned int) > + 5.88% mempool::pool_t::adjust_count(long, long) > > Thread: 21 (io_context_pool) - 17 samples > > + 100.00% clone > + 100.00% start_thread > + 100.00% execute_native_thread_routine > + 100.00% std::thread::_State_impl<std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::i > o_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> > >::_M_run() > + 100.00% std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(sh > ort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> >::operator()() > + 100.00% std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(s > hort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) > + 100.00% std::__invoke<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1 > }&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&, ceph::async::io_context_pool::start(sho > rt)::{lambda()#1}&&) > + 100.00% std::__invoke_impl<void, make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short): > :{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, make_named_thread<ceph::async::io_context_pool::start(short)::{la > mbda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}&&, ceph::async::io_context_pool::sta > rt(short)::{lambda()#1}&&) > + 100.00% make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambd > a(auto:1&&, (auto:2&&)...)#1}::operator()<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) const > + 100.00% std::invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) > + 100.00% std::__invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) > + 100.00% std::__invoke_impl<void, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, ceph::async::io_context_pool::start(short)::{lambda()#1}&&) > + 100.00% ceph::async::io_context_pool::start(short)::{lambda()#1}::operator()() const > + 100.00% boost::asio::io_context::run > + 100.00% boost::asio::detail::scheduler::run(boost::system::error_code&) [clone .constprop.0] [clone .isra.0] > + 100.00% boost::asio::detail::scheduler::do_run_one > + 94.12% boost::asio::detail::conditionally_enabled_event::wait > |+ 94.12% boost::asio::detail::posix_event::wait<boost::asio::detail::conditionally_enabled_mutex::scoped_lock> > | + 94.12% pthread_cond_wait@@GLIBC_2.3.2 > | + 94.12% __futex_abstimed_wait_common > + 5.88% boost::asio::detail::scheduler_operation::complete > + 5.88% boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>, std::allocator<voi > d>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) > + 5.88% boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>::operator()() > + 5.88% boost::asio::detail::executor_op<boost::asio::detail::binder0<MonClient::handle_config(MConfig*)::{lambda()#2}>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete > (void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) [clone .lto_priv.0] > + 5.88% ceph::common::ConfigProxy::set_mon_vals(ceph::common::CephContext*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<cha > r, std::char_traits<char>, std::allocator<char> >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__c > xx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > co > nst&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>) [clone .isra.0] > + 5.88% md_config_t::set_mon_vals(ceph::common::CephContext*, ConfigValues&, ConfigTracker const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_ > _cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocato > r<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>) > + 5.88% md_config_t::update_legacy_vals(ConfigValues&) > + 5.88% md_config_t::update_legacy_val(ConfigValues&, Option const&, std::variant<long ConfigValues::*, unsigned long ConfigValues::*, std::__cxx11::basic_string<char, std::char_traits<char>, s > td::allocator<char> > ConfigValues::*, double ConfigValues::*, bool ConfigValues::*, entity_addr_t ConfigValues::*, entity_addrvec_t ConfigValues::*, uuid_d ConfigValues::*>) > + 5.88% md_config_t::_get_val(ConfigValues const&, Option const&, boost::container::small_vector<std::pair<Option const*, std::variant<std::monostate, std::__cxx11::basic_string<char, std::cha > r_traits<char>, std::allocator<char> >, unsigned long, long, double, bool, entity_addr_t, entity_addrvec_t, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long > , std::ratio<1l, 1000l> >, Option::size_t, uuid_d> const*>, 4ul, void, void>*, std::basic_ostream<char, std::char_traits<char> >*) const > + 5.88% md_config_t::_get_val_nometa[abi:cxx11](ConfigValues const&, Option const&) const > > Thread: 22 (io_context_pool) - 17 samples > > + 100.00% clone > + 100.00% start_thread > + 100.00% execute_native_thread_routine > + 100.00% std::thread::_State_impl<std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::i > o_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> > >::_M_run() > + 100.00% std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(sh > ort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> >::operator()() > + 100.00% std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(s > hort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) > + 100.00% std::__invoke<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1 > }&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&, ceph::async::io_context_pool::start(sho > rt)::{lambda()#1}&&) > + 100.00% std::__invoke_impl<void, make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short): > :{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, make_named_thread<ceph::async::io_context_pool::start(short)::{la > mbda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}&&, ceph::async::io_context_pool::sta > rt(short)::{lambda()#1}&&) > + 100.00% make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambd > a(auto:1&&, (auto:2&&)...)#1}::operator()<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) const > + 100.00% std::invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) > + 100.00% std::__invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) > + 100.00% std::__invoke_impl<void, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, ceph::async::io_context_pool::start(short)::{lambda()#1}&&) > + 100.00% ceph::async::io_context_pool::start(short)::{lambda()#1}::operator()() const > + 100.00% boost::asio::io_context::run > + 100.00% boost::asio::detail::scheduler::run(boost::system::error_code&) [clone .constprop.0] [clone .isra.0] > + 100.00% boost::asio::detail::scheduler::do_run_one > + 100.00% boost::asio::detail::conditionally_enabled_event::wait > + 100.00% boost::asio::detail::posix_event::wait<boost::asio::detail::conditionally_enabled_mutex::scoped_lock> > + 100.00% pthread_cond_wait@@GLIBC_2.3.2 > + 100.00% __futex_abstimed_wait_common > > Thread: 26 (msgr-worker-2) - 17 samples > > + 100.00% clone > + 100.00% start_thread > + 100.00% execute_native_thread_routine > + 100.00% std::_Function_handler<void (), NetworkStack::add_thread(Worker*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) > + 100.00% EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*) > + 94.12% EpollDriver::event_wait(std::vector<FiredFileEvent, std::allocator<FiredFileEvent> >&, timeval*) > |+ 94.12% epoll_wait > + 5.88% AsyncConnection::process() > + 5.88% ProtocolV2::run_continuation(Ct<ProtocolV2>&) > + 5.88% ProtocolV2::handle_message() > + 5.88% decode_message(ceph::common::CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&, boost::intrusive_ptr<Conne > ction>) > + 5.88% MMgrMap::decode_payload() > + 5.88% MgrMap::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&) > + 5.88% MgrMap::ModuleInfo::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&) > + 5.88% std::enable_if<denc_traits<std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocat > or<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, void>::supported&&(!denc_traits<std::set<std::__cxx11::basic_string<char, std::cha > r_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_trait > s<char>, std::allocator<char> > > >, void>::need_contiguous), void>::type ceph::decode<std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cx > x11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, denc_traits<std::set<s > td::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__c > xx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, void> >(std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx1 > 1::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, ceph::buffer::v15_2_0:: > list::iterator_impl<true>&) > + 5.88% ceph::buffer::v15_2_0::ptr::release() > > Sidenote: while recording this, the MDS became so slow that the MDS cluster declared it dead … As can be seen, the MDS is waiting in futex to no end, but how can we find out what it is waiting for? FS has many small files but nearly no activity …. > > Thanks in advance > Martin > > -- > Martin Gerhard Loschwitz > Geschäftsführer / CEO, True West IT Services GmbH > Phone: +49 2433 5253130 > Mobile: +49 176 61832178 > Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany > Legal: HRB 21985, Amtsgericht Mönchengladbach > VAT: DE363893844 > > True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender. > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Cheers, Venky

1 0

Re: [MDS crash / ceph_assert failure] CephFS MDS crash on Reef 18.2.2 (Filer::_probed assertion)
by Dhairya Parmar 16 Feb '26

16 Feb '26

Hi, thanks for sharing the logs. I see lines: Jan 21 19:52:21 s11 bash[3097096]: debug 2026-01-21T19:52:21.946+0000 7f6cd7e74700 -1 log_channel(cluster) log [ERR] : bad backtrace on directory inode 0x100013897dd Jan 21 19:52:21 s11 bash[3097096]: debug 2026-01-21T19:52:21.948+0000 7f6cd7e74700 -1 log_channel(cluster) log [ERR] : bad backtrace on directory inode 0x1000139903b Jan 21 19:52:21 s11 bash[3097096]: debug 2026-01-21T19:52:21.948+0000 7f6cd7e74700 -1 log_channel(cluster) log [ERR] : bad backtrace on directory inode 0x10001399315 this seems to be hinting that the metadata recovery was inconsistent, this doesn't directly translate into the root cause but this _may_ have lead to MDS sending wrong input (possibly the file layout) while calling probe() in RecoveryQueue::_start making use of the "projected inode" that passes on the inode number, the file layout and the max file size to filer.probe() which is calling the Stripper::file_to_extents to calculate the extents and their compuated sizes should've been more than what the object size is returned by objecter->stat(). If you can set the debug mds to 20, I think it should reveal what the values of the object extent was calculated by Stripper code and the file size returned by the OSD. *Dhairya Parmar* Software Engineer, CephFS On Sat, Jan 31, 2026 at 11:40 PM Андрей Муханов <mukhanovster(a)gmail.com> wrote: > Hi Dhairya, > The debug logs were included in my original email, but they seem to have > disappeared from the mailing list. Please find them here. > > To my understanding, yes, this appears to be the same issue described in > the link you provided, although the root cause may be different. > > Jan 21 19:52:13 s11 systemd[1]: Started Ceph > mds.production_cephfs.s11.srybkc for 486d3212-1558-11ec-913a-ac1f6b95d690. > Jan 21 19:52:17 s11 systemd-journald[1217]: Suppressed 705 messages from > ceph-486d3212-1558-11ec-913a-ac1f6b95d690(a)mds.production_cephfs.s11.srybkc.service > Jan 21 19:52:17 s11 bash[3097096]: debug 2026-01-21T19:52:17.798+0000 > 7f6ce964ab00 0 set uid:gid to 167:167 (ceph:ceph) > Jan 21 19:52:17 s11 bash[3097096]: debug 2026-01-21T19:52:17.798+0000 > 7f6ce964ab00 0 ceph version 18.2.2 > (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable), process ceph-mds, > pid 6 > Jan 21 19:52:17 s11 bash[3097096]: debug 2026-01-21T19:52:17.798+0000 > 7f6ce964ab00 1 main not setting numa affinity > Jan 21 19:52:17 s11 bash[3097096]: debug 2026-01-21T19:52:17.798+0000 > 7f6ce964ab00 0 pidfile_write: ignore empty --pid-file > Jan 21 19:52:17 s11 bash[3097096]: starting > mds.production_cephfs.s11.srybkc at > Jan 21 19:52:17 s11 bash[3097096]: debug 2026-01-21T19:52:17.930+0000 > 7f6cdde80700 1 mds.production_cephfs.s11.srybkc Updating MDS map to > version 1111785 from mon.2 > Jan 21 19:52:17 s11 bash[3097096]: debug 2026-01-21T19:52:17.930+0000 > 7f6cdbe7c700 5 mds.beacon.production_cephfs.s11.srybkc Sending beacon > up:boot seq 1 > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.327+0000 > 7f6cdde80700 1 mds.production_cephfs.s11.srybkc Updating MDS map to > version 1111786 from mon.2 > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.327+0000 > 7f6cdde80700 1 mds.production_cephfs.s11.srybkc Monitors have assigned me > to become a standby. > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.327+0000 > 7f6cdde80700 5 mds.beacon.production_cephfs.s11.srybkc set_want_state: > up:boot -> up:standby > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.356+0000 > 7f6ce0e86700 5 mds.beacon.production_cephfs.s11.srybkc received beacon > reply up:boot seq 1 rtt 0.425989 > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.510+0000 > 7f6cdde80700 1 mds.production_cephfs.s11.srybkc Updating MDS map to > version 1111787 from mon.2 > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.510+0000 > 7f6cdde80700 4 mds.0.purge_queue operator(): data pool 20 not found in > OSDMap > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.511+0000 > 7f6cdde80700 4 mds.0.purge_queue operator(): data pool 20 not found in > OSDMap > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.511+0000 > 7f6cdde80700 4 mds.0.0 apply_blocklist: killed 0, blocklisted sessions (0 > blocklist entries, 0) > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.511+0000 > 7f6cdde80700 1 mds.0.1111787 handle_mds_map i am now mds.0.1111787 > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.511+0000 > 7f6cdde80700 1 mds.0.1111787 handle_mds_map state change up:standby --> > up:replay > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.511+0000 > 7f6cdde80700 5 mds.beacon.production_cephfs.s11.srybkc set_want_state: > up:standby -> up:replay > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.511+0000 > 7f6cdde80700 1 mds.0.1111787 replay_start > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.511+0000 > 7f6cdde80700 1 mds.0.1111787 waiting for osdmap 5831472 (which blocklists > prior instance) > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.511+0000 > 7f6cd9e78700 2 mds.0.cache Memory usage: total 326984, rss 32376, heap > 182580, baseline 182580, 0 / 0 inodes have caps, 0 caps, 0 caps per inode > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.569+0000 > 7f6cd7e74700 2 mds.0.1111787 Booting: 0: opening inotable > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.569+0000 > 7f6cd7e74700 2 mds.0.1111787 Booting: 0: opening sessionmap > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.569+0000 > 7f6cd7e74700 2 mds.0.1111787 Booting: 0: opening mds log > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.569+0000 > 7f6cd7e74700 5 mds.0.log open discovering log bounds > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.569+0000 > 7f6cd7e74700 2 mds.0.1111787 Booting: 0: opening purge queue (async) > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.569+0000 > 7f6cd7e74700 4 mds.0.purge_queue open: opening > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.569+0000 > 7f6cd7e74700 2 mds.0.1111787 Booting: 0: loading open file table (async) > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.570+0000 > 7f6cd7e74700 2 mds.0.1111787 Booting: 0: opening snap table > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.576+0000 > 7f6cd7673700 4 mds.0.log Waiting for journal 0x200 to recover... > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.598+0000 > 7f6cd7673700 4 mds.0.log Journal 0x200 recovered. > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.598+0000 > 7f6cd7673700 4 mds.0.log Recovered journal 0x200 in format 1 > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.599+0000 > 7f6cd7e74700 2 mds.0.1111787 Booting: 1: loading/discovering base inodes > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.599+0000 > 7f6cd7e74700 0 mds.0.cache creating system inode with ino:0x100 > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.599+0000 > 7f6cd7e74700 0 mds.0.cache creating system inode with ino:0x1 > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.607+0000 > 7f6cd7e74700 2 mds.0.1111787 Booting: 2: replaying mds log > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.607+0000 > 7f6cd7e74700 2 mds.0.1111787 Booting: 2: waiting for purge queue recovered > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.607+0000 > 7f6cd6671700 1 mds.0.journal EResetJournal > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.607+0000 > 7f6cd6671700 1 mds.0.sessionmap wipe start > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.607+0000 > 7f6cd6671700 1 mds.0.sessionmap wipe result > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.607+0000 > 7f6cd6671700 1 mds.0.sessionmap wipe done > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.636+0000 > 7f6cd8e76700 4 mds.0.purge_queue operator(): open complete > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.636+0000 > 7f6cd7e74700 1 mds.0.1111787 Finished replaying journal > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.636+0000 > 7f6cd7e74700 1 mds.0.1111787 making mds journal writeable > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.636+0000 > 7f6cd7e74700 1 mds.0.1111787 wiping out client sessions > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.636+0000 > 7f6cd7e74700 1 mds.0.sessionmap wipe start > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.636+0000 > 7f6cd7e74700 1 mds.0.sessionmap wipe result > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.636+0000 > 7f6cd7e74700 1 mds.0.sessionmap wipe done > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.636+0000 > 7f6cd7e74700 2 mds.0.1111787 i am alone, moving to state reconnect > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.636+0000 > 7f6cd7e74700 3 mds.0.1111787 request_state up:reconnect > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.636+0000 > 7f6cd7e74700 5 mds.beacon.production_cephfs.s11.srybkc set_want_state: > up:replay -> up:reconnect > Jan 21 19:52:18 s11 bash[3097096]: debug 2026-01-21T19:52:18.636+0000 > 7f6cd7e74700 5 mds.beacon.production_cephfs.s11.srybkc Sending beacon > up:reconnect seq 2 > Jan 21 19:52:19 s11 bash[3097096]: debug 2026-01-21T19:52:19.512+0000 > 7f6cd9e78700 2 mds.0.cache Memory usage: total 353748, rss 37372, heap > 207156, baseline 182580, 0 / 3 inodes have caps, 0 caps, 0 caps per inode > Jan 21 19:52:20 s11 bash[3097096]: debug 2026-01-21T19:52:20.512+0000 > 7f6cd9e78700 2 mds.0.cache Memory usage: total 353748, rss 37372, heap > 207156, baseline 182580, 0 / 3 inodes have caps, 0 caps, 0 caps per inode > Jan 21 19:52:20 s11 bash[3097096]: debug 2026-01-21T19:52:20.833+0000 > 7f6cdde80700 1 mds.production_cephfs.s11.srybkc Updating MDS map to > version 1111788 from mon.2 > Jan 21 19:52:20 s11 bash[3097096]: debug 2026-01-21T19:52:20.833+0000 > 7f6cdde80700 1 mds.0.1111787 handle_mds_map i am now mds.0.1111787 > Jan 21 19:52:20 s11 bash[3097096]: debug 2026-01-21T19:52:20.833+0000 > 7f6cdde80700 1 mds.0.1111787 handle_mds_map state change up:replay --> > up:reconnect > Jan 21 19:52:20 s11 bash[3097096]: debug 2026-01-21T19:52:20.833+0000 > 7f6cdde80700 1 mds.0.1111787 reconnect_start > Jan 21 19:52:20 s11 bash[3097096]: debug 2026-01-21T19:52:20.833+0000 > 7f6cdde80700 1 mds.0.1111787 reopen_log > Jan 21 19:52:20 s11 bash[3097096]: debug 2026-01-21T19:52:20.833+0000 > 7f6cdde80700 4 mds.0.1111787 apply_blocklist: killed 0, blocklisted > sessions (42 blocklist entries, 0) > Jan 21 19:52:20 s11 bash[3097096]: debug 2026-01-21T19:52:20.833+0000 > 7f6cdde80700 1 mds.0.1111787 reconnect_done > Jan 21 19:52:20 s11 bash[3097096]: debug 2026-01-21T19:52:20.833+0000 > 7f6cdde80700 3 mds.0.1111787 request_state up:rejoin > Jan 21 19:52:20 s11 bash[3097096]: debug 2026-01-21T19:52:20.833+0000 > 7f6cdde80700 5 mds.beacon.production_cephfs.s11.srybkc set_want_state: > up:reconnect -> up:rejoin > Jan 21 19:52:20 s11 bash[3097096]: debug 2026-01-21T19:52:20.833+0000 > 7f6cdde80700 5 mds.beacon.production_cephfs.s11.srybkc Sending beacon > up:rejoin seq 3 > Jan 21 19:52:20 s11 bash[3097096]: debug 2026-01-21T19:52:20.837+0000 > 7f6ce0e86700 5 mds.beacon.production_cephfs.s11.srybkc received beacon > reply up:reconnect seq 2 rtt 2.20094 > Jan 21 19:52:20 s11 bash[3097096]: debug 2026-01-21T19:52:20.852+0000 > 7f6cdde80700 0 mds.0.server ignoring msg from not-open > sessionclient_reconnect(0 caps 0 realms ) v3 > Jan 21 19:52:20 s11 bash[3097096]: debug 2026-01-21T19:52:20.852+0000 > 7f6cdde80700 0 mds.0.server ignoring msg from not-open > sessionclient_reconnect(0 caps 0 realms ) v3 > Jan 21 19:52:20 s11 bash[3097096]: debug 2026-01-21T19:52:20.852+0000 > 7f6ce1687700 0 --1- [v2: > 20.2.0.104:6800/4062624433,v1:20.2.0.104:6801/4062624433] >> v1: > 20.2.0.66:0/1664934024 conn(0x564cc1219c00 0x564cc0148000 :6801 s=OPENED > pgs=4344 cs=1 l=0).fault server, going to standby > Jan 21 19:52:20 s11 bash[3097096]: debug 2026-01-21T19:52:20.852+0000 > 7f6ce0e86700 0 --1- [v2: > 20.2.0.104:6800/4062624433,v1:20.2.0.104:6801/4062624433] >> v1: > 20.2.0.65:0/3311648163 conn(0x564cc1200800 0x564cc0148800 :6801 s=OPENED > pgs=380 cs=1 l=0).fault server, going to standby > Jan 21 19:52:20 s11 bash[3097096]: debug 2026-01-21T19:52:20.875+0000 > 7f6cdde80700 5 mds.0.server session is closed, dropping > client.192669065:14698917 > Jan 21 19:52:20 s11 bash[3097096]: debug 2026-01-21T19:52:20.875+0000 > 7f6cdde80700 0 mds.0.server ignoring msg from not-open > sessionclient_reconnect(0 caps 0 realms ) v3 > Jan 21 19:52:21 s11 bash[3097096]: debug 2026-01-21T19:52:21.512+0000 > 7f6cd9e78700 2 mds.0.cache Memory usage: total 353748, rss 37372, heap > 207156, baseline 182580, 0 / 3 inodes have caps, 0 caps, 0 caps per inode > Jan 21 19:52:21 s11 bash[3097096]: debug 2026-01-21T19:52:21.936+0000 > 7f6cdde80700 1 mds.production_cephfs.s11.srybkc Updating MDS map to > version 1111789 from mon.2 > Jan 21 19:52:21 s11 bash[3097096]: debug 2026-01-21T19:52:21.936+0000 > 7f6cdde80700 1 mds.0.1111787 handle_mds_map i am now mds.0.1111787 > Jan 21 19:52:21 s11 bash[3097096]: debug 2026-01-21T19:52:21.936+0000 > 7f6cdde80700 1 mds.0.1111787 handle_mds_map state change up:reconnect --> > up:rejoin > Jan 21 19:52:21 s11 bash[3097096]: debug 2026-01-21T19:52:21.936+0000 > 7f6cdde80700 1 mds.0.1111787 rejoin_start > Jan 21 19:52:21 s11 bash[3097096]: debug 2026-01-21T19:52:21.936+0000 > 7f6cdde80700 1 mds.0.1111787 rejoin_joint_start > Jan 21 19:52:21 s11 bash[3097096]: debug 2026-01-21T19:52:21.940+0000 > 7f6ce0e86700 5 mds.beacon.production_cephfs.s11.srybkc received beacon > reply up:rejoin seq 3 rtt 1.10697 > Jan 21 19:52:21 s11 bash[3097096]: debug 2026-01-21T19:52:21.946+0000 > 7f6cd7e74700 -1 log_channel(cluster) log [ERR] : bad backtrace on directory > inode 0x100013897dd > Jan 21 19:52:21 s11 bash[3097096]: debug 2026-01-21T19:52:21.948+0000 > 7f6cd7e74700 -1 log_channel(cluster) log [ERR] : bad backtrace on directory > inode 0x1000139903b > Jan 21 19:52:21 s11 bash[3097096]: debug 2026-01-21T19:52:21.948+0000 > 7f6cd7e74700 -1 log_channel(cluster) log [ERR] : bad backtrace on directory > inode 0x10001399315 > Jan 21 19:52:21 s11 bash[3097096]: debug 2026-01-21T19:52:21.953+0000 > 7f6cd9677700 1 mds.0.1111787 rejoin_done > Jan 21 19:52:21 s11 bash[3097096]: debug 2026-01-21T19:52:21.956+0000 > 7f6cd9677700 3 mds.0.1111787 request_state up:active > Jan 21 19:52:21 s11 bash[3097096]: debug 2026-01-21T19:52:21.956+0000 > 7f6cd9677700 5 mds.beacon.production_cephfs.s11.srybkc set_want_state: > up:rejoin -> up:active > Jan 21 19:52:21 s11 bash[3097096]: debug 2026-01-21T19:52:21.956+0000 > 7f6cd9677700 5 mds.beacon.production_cephfs.s11.srybkc Sending beacon > up:active seq 4 > Jan 21 19:52:22 s11 bash[3097096]: debug 2026-01-21T19:52:22.513+0000 > 7f6cd9e78700 2 mds.0.cache Memory usage: total 355796, rss 42628, heap > 207156, baseline 182580, 0 / 445 inodes have caps, 0 caps, 0 caps per inode > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.130+0000 > 7f6cdde80700 1 mds.production_cephfs.s11.srybkc Updating MDS map to > version 1111790 from mon.2 > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.130+0000 > 7f6cdde80700 1 mds.0.1111787 handle_mds_map i am now mds.0.1111787 > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.130+0000 > 7f6cdde80700 1 mds.0.1111787 handle_mds_map state change up:rejoin --> > up:active > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.130+0000 > 7f6cdde80700 1 mds.0.1111787 recovery_done -- successful recovery! > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.132+0000 > 7f6ce1687700 0 --1- [v2: > 20.2.0.104:6800/4062624433,v1:20.2.0.104:6801/4062624433] >> v1: > 20.2.0.65:0/3311648163 conn(0x564cc1275800 0x564cc0149000 :6801 > s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 > accept peer reset, then tried to connect to us, replacing > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.132+0000 > 7f6ce0685700 0 --1- [v2: > 20.2.0.104:6800/4062624433,v1:20.2.0.104:6801/4062624433] >> v1: > 20.2.0.66:0/1664934024 conn(0x564cc139b000 0x564cc0149800 :6801 > s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 > accept peer reset, then tried to connect to us, replacing > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.133+0000 > 7f6cdde80700 1 mds.0.1111787 active_start > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.133+0000 > 7f6cdde80700 1 mds.0.1111787 cluster recovered. > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.133+0000 > 7f6cdde80700 4 mds.0.1111787 set_osd_epoch_barrier: epoch=5831472 > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.133+0000 > 7f6cdde80700 5 mds.production_cephfs.s11.srybkc ms_handle_remote_reset on > v1:20.2.0.65:0/3311648163 > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.133+0000 > 7f6cdde80700 3 mds.production_cephfs.s11.srybkc ms_handle_remote_reset > closing connection for session client.186227172 v1:20.2.0.65:0/3311648163 > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.133+0000 > 7f6cdde80700 5 mds.production_cephfs.s11.srybkc ms_handle_reset on v1: > 20.2.0.65:0/3311648163 > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.133+0000 > 7f6cdde80700 5 mds.production_cephfs.s11.srybkc ms_handle_remote_reset on > v1:20.2.0.66:0/1664934024 > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.133+0000 > 7f6cdde80700 3 mds.production_cephfs.s11.srybkc ms_handle_remote_reset > closing connection for session client.186208578 v1:20.2.0.66:0/1664934024 > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.133+0000 > 7f6cdde80700 5 mds.production_cephfs.s11.srybkc ms_handle_reset on v1: > 20.2.0.66:0/1664934024 > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.150+0000 > 7f6ce0e86700 5 mds.beacon.production_cephfs.s11.srybkc received beacon > reply up:active seq 4 rtt 1.19397 > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.150+0000 > 7f6cdde80700 5 mds.production_cephfs.s11.srybkc ms_handle_remote_reset on > 20.2.0.106:0/834757530 > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.150+0000 > 7f6cdde80700 3 mds.production_cephfs.s11.srybkc ms_handle_remote_reset > closing connection for session client.192669065 20.2.0.106:0/834757530 > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.151+0000 > 7f6cdde80700 5 mds.production_cephfs.s11.srybkc ms_handle_reset on > 20.2.0.106:0/834757530 > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.151+0000 > 7f6cd6e72700 5 mds.0.log _submit_thread 4195668~8352 : EUpdate > check_inode_max_size [metablob 0x1, 9 dirs] > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.151+0000 > 7f6cd6e72700 5 mds.0.log _submit_thread 4204040~1582 : EUpdate > check_inode_max_size [metablob 0x10001399315, 1 dirs] > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.151+0000 > 7f6cd6e72700 5 mds.0.log _submit_thread 4205642~1554 : EUpdate > check_inode_max_size [metablob 0x10001399315, 1 dirs] > Jan 21 19:52:23 s11 bash[3097096]: > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/osdc/Filer.cc: > In function 'bool Filer::_probed(Filer::Probe*, const object_t&, uint64_t, > ceph::real_time, Filer::Probe::unique_lock&)' thread 7f6cd7e74700 time > 2026-01-21T19:52:23.152720+0000 > Jan 21 19:52:23 s11 bash[3097096]: > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/osdc/Filer.cc: > 224: FAILED ceph_assert(probe->known_size[p->oid] <= shouldbe) > Jan 21 19:52:23 s11 bash[3097096]: ceph version 18.2.2 > (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) > Jan 21 19:52:23 s11 bash[3097096]: 1: (ceph::__ceph_assert_fail(char > const*, char const*, int, char const*)+0x135) [0x7f6ce8482e15] > Jan 21 19:52:23 s11 bash[3097096]: 2: > /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f6ce8482fdb] > Jan 21 19:52:23 s11 bash[3097096]: 3: (Filer::_probed(Filer::Probe*, > object_t const&, unsigned long, std::chrono::time_point<ceph::real_clock, > std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, > std::unique_lock<std::mutex>&)+0x1781) [0x564cbe4c0f91] > Jan 21 19:52:23 s11 bash[3097096]: 4: (Filer::C_Probe::finish(int)+0x91) > [0x564cbe4c2c71] > Jan 21 19:52:23 s11 bash[3097096]: 5: (Context::complete(int)+0xd) > [0x564cbe09c5fd] > Jan 21 19:52:23 s11 bash[3097096]: 6: > (Finisher::finisher_thread_entry()+0x18d) [0x7f6ce8526abd] > Jan 21 19:52:23 s11 bash[3097096]: 7: /lib64/libpthread.so.0(+0x81ca) > [0x7f6ce72281ca] > Jan 21 19:52:23 s11 bash[3097096]: 8: clone() > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.152+0000 > 7f6cd7e74700 -1 > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/osdc/Filer.cc: > In function 'bool Filer::_probed(Filer::Probe*, const object_t&, uint64_t, > ceph::real_time, Filer::Probe::unique_lock&)' thread 7f6cd7e74700 time > 2026-01-21T19:52:23.152720+0000 > Jan 21 19:52:23 s11 bash[3097096]: > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/osdc/Filer.cc: > 224: FAILED ceph_assert(probe->known_size[p->oid] <= shouldbe) > Jan 21 19:52:23 s11 bash[3097096]: ceph version 18.2.2 > (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) > Jan 21 19:52:23 s11 bash[3097096]: 1: (ceph::__ceph_assert_fail(char > const*, char const*, int, char const*)+0x135) [0x7f6ce8482e15] > Jan 21 19:52:23 s11 bash[3097096]: 2: > /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f6ce8482fdb] > Jan 21 19:52:23 s11 bash[3097096]: 3: (Filer::_probed(Filer::Probe*, > object_t const&, unsigned long, std::chrono::time_point<ceph::real_clock, > std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, > std::unique_lock<std::mutex>&)+0x1781) [0x564cbe4c0f91] > Jan 21 19:52:23 s11 bash[3097096]: 4: (Filer::C_Probe::finish(int)+0x91) > [0x564cbe4c2c71] > Jan 21 19:52:23 s11 bash[3097096]: 5: (Context::complete(int)+0xd) > [0x564cbe09c5fd] > Jan 21 19:52:23 s11 bash[3097096]: 6: > (Finisher::finisher_thread_entry()+0x18d) [0x7f6ce8526abd] > Jan 21 19:52:23 s11 bash[3097096]: 7: /lib64/libpthread.so.0(+0x81ca) > [0x7f6ce72281ca] > Jan 21 19:52:23 s11 bash[3097096]: 8: clone() > Jan 21 19:52:23 s11 bash[3097096]: *** Caught signal (Aborted) ** > Jan 21 19:52:23 s11 bash[3097096]: in thread 7f6cd7e74700 > thread_name:MR_Finisher > Jan 21 19:52:23 s11 bash[3097096]: ceph version 18.2.2 > (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) > Jan 21 19:52:23 s11 bash[3097096]: 1: /lib64/libpthread.so.0(+0x12d20) > [0x7f6ce7232d20] > Jan 21 19:52:23 s11 bash[3097096]: 2: gsignal() > Jan 21 19:52:23 s11 bash[3097096]: 3: abort() > Jan 21 19:52:23 s11 bash[3097096]: 4: (ceph::__ceph_assert_fail(char > const*, char const*, int, char const*)+0x18f) [0x7f6ce8482e6f] > Jan 21 19:52:23 s11 bash[3097096]: 5: > /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f6ce8482fdb] > Jan 21 19:52:23 s11 bash[3097096]: 6: (Filer::_probed(Filer::Probe*, > object_t const&, unsigned long, std::chrono::time_point<ceph::real_clock, > std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, > std::unique_lock<std::mutex>&)+0x1781) [0x564cbe4c0f91] > Jan 21 19:52:23 s11 bash[3097096]: 7: (Filer::C_Probe::finish(int)+0x91) > [0x564cbe4c2c71] > Jan 21 19:52:23 s11 bash[3097096]: 8: (Context::complete(int)+0xd) > [0x564cbe09c5fd] > Jan 21 19:52:23 s11 bash[3097096]: 9: > (Finisher::finisher_thread_entry()+0x18d) [0x7f6ce8526abd] > Jan 21 19:52:23 s11 bash[3097096]: 10: /lib64/libpthread.so.0(+0x81ca) > [0x7f6ce72281ca] > Jan 21 19:52:23 s11 bash[3097096]: 11: clone() > Jan 21 19:52:23 s11 bash[3097096]: debug 2026-01-21T19:52:23.154+0000 > 7f6cd7e74700 -1 *** Caught signal (Aborted) ** > Jan 21 19:52:23 s11 bash[3097096]: in thread 7f6cd7e74700 > thread_name:MR_Finisher > Jan 21 19:52:23 s11 bash[3097096]: ceph version 18.2.2 > (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) > Jan 21 19:52:23 s11 bash[3097096]: 1: /lib64/libpthread.so.0(+0x12d20) > [0x7f6ce7232d20] > Jan 21 19:52:23 s11 bash[3097096]: 2: gsignal() > Jan 21 19:52:23 s11 bash[3097096]: 3: abort() > Jan 21 19:52:23 s11 bash[3097096]: 4: (ceph::__ceph_assert_fail(char > const*, char const*, int, char const*)+0x18f) [0x7f6ce8482e6f] > Jan 21 19:52:23 s11 bash[3097096]: 5: > /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f6ce8482fdb] > Jan 21 19:52:23 s11 bash[3097096]: 6: (Filer::_probed(Filer::Probe*, > object_t const&, unsigned long, std::chrono::time_point<ceph::real_clock, > std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, > std::unique_lock<std::mutex>&)+0x1781) [0x564cbe4c0f91] > Jan 21 19:52:23 s11 bash[3097096]: 7: (Filer::C_Probe::finish(int)+0x91) > [0x564cbe4c2c71] > Jan 21 19:52:23 s11 bash[3097096]: 8: (Context::complete(int)+0xd) > [0x564cbe09c5fd] > Jan 21 19:52:23 s11 bash[3097096]: 9: > (Finisher::finisher_thread_entry()+0x18d) [0x7f6ce8526abd] > Jan 21 19:52:23 s11 bash[3097096]: 10: /lib64/libpthread.so.0(+0x81ca) > [0x7f6ce72281ca] > Jan 21 19:52:23 s11 bash[3097096]: 11: clone() > Jan 21 19:52:23 s11 bash[3097096]: NOTE: a copy of the executable, or > `objdump -rdS <executable>` is needed to interpret this. > > Kind regards, > Andrei > > чт, 29 янв. 2026 г. в 16:02, Dhairya Parmar <dparmar(a)redhat.com>: > >> Hi, Can you share the crash backtrace and mds debug logs? Also is it >> relevant to this issue https://tracker.ceph.com/issues/22550? >> >> *Dhairya Parmar* >> >> Software Engineer, CephFS >> >> >> On Thu, Jan 29, 2026 at 1:05 PM Андрей Муханов via ceph-users < >> ceph-users(a)ceph.io> wrote: >> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>

1 0

clarification on Fast-EC BUG in tentacle
by Sascha Lucas 15 Feb '26

15 Feb '26

Dear ceph users, I like to get some clarification regarding the Fast-EC BUG in tentacle[1]. The clyso article mentions "on CephFS", which gave the impression that just CephFS as a application is affected, but others like RBD are not. I assume and may found out, that RDB is also affected. In a cephadm test environment I started a qemu-VM on an EC-data-pool with ec-overwrite and ec-optimisations set and issued fstrim (qemu discard=unmap via librbd) which immediately brings down some OSDs (see log below). Also the clyso article points to issue 71642 "crash if allow_ec_optimizations set but with allow_ec_overwrites not set"[2], which from its title seems misleading because others and I have allow_ec_overwrites set. Thanks to clyso's known-bugs collection! Otherwise it's hard to get an overview what problem one might hit currently ;-). And also thanks for providing fixed images on harbor.clyso.com/custom-ceph/ceph/ceph:v20.2.0-fast-ec-path-hf.2. It seems the fix[3] is from Jul 11, 2025, but still awaiting backport to, in the mean time released, stable version tentacle? Sounds odd. Thanks, Sascha. [1] https://docs.clyso.com/docs/kb/known-bugs/tentacle/#osd-crash-when-enabling… [2] https://tracker.ceph.com/issues/71642 [3] https://github.com/ceph/ceph/commit/51679b948bd9bb45cce9d778b0555f45ff808cd8 ODS crash log ======8<-------------------------- osd-1[197069]: terminate called after throwing an instance of 'std::out_of_range' osd-1[197069]: what(): Key not found osd-1[197069]: *** Caught signal (Aborted) ** osd-1[197069]: in thread 7f2dd5c65640 thread_name:tp_osd_tp osd-1[197069]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) osd-1[197069]: 1: /lib64/libc.so.6(+0x3fc30) [0x7f2df4da8c30] osd-1[197069]: 2: /lib64/libc.so.6(+0x8d03c) [0x7f2df4df603c] osd-1[197069]: 3: raise() osd-1[197069]: 4: abort() osd-1[197069]: 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f2df5b45b21] osd-1[197069]: 6: /lib64/libstdc++.so.6(+0xad53c) [0x7f2df5b5153c] osd-1[197069]: 7: /lib64/libstdc++.so.6(+0xad5a7) [0x7f2df5b515a7] osd-1[197069]: 8: /lib64/libstdc++.so.6(+0xad809) [0x7f2df5b51809] osd-1[197069]: 9: /usr/bin/ceph-osd(+0x41b012) [0x55ef3f2b1012] osd-1[197069]: 10: (ECTransaction::WritePlanObj::WritePlanObj(hobject_t const&, PGTransaction::ObjectOperation const&, ECU> osd-1[197069]: 11: /usr/bin/ceph-osd(+0x99cdaa) [0x55ef3f832daa] osd-1[197069]: 12: (ECCommon::get_write_plan(ECUtil::stripe_info_t const&, PGTransaction&, ECCommon::ReadPipeline&, ECComm> osd-1[197069]: 13: (ECBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::uniq> osd-1[197069]: 14: /usr/bin/ceph-osd(+0x7d980f) [0x55ef3f66f80f] osd-1[197069]: 15: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x3ae) [0x55ef3f5f002e] osd-1[197069]: 16: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xf6a) [0x55ef3f5cd63a] osd-1[197069]: 17: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x2d5f) [0x55ef3f5bdf5f] osd-1[197069]: 18: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x19> osd-1[197069]: 19: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)> osd-1[197069]: 20: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x8bc) [0x55ef3f51611c] osd-1[197069]: 21: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x23a) [0x55ef3fa98e1a] osd-1[197069]: 22: /usr/bin/ceph-osd(+0xc033d4) [0x55ef3fa993d4] osd-1[197069]: 23: /lib64/libc.so.6(+0x8b2fa) [0x7f2df4df42fa] osd-1[197069]: 24: /lib64/libc.so.6(+0x110400) [0x7f2df4e79400] osd-1[197069]: 2026-02-15T00:10:48.350+0000 7f2dd5c65640 -1 *** Caught signal (Aborted) ** osd-1[197069]: in thread 7f2dd5c65640 thread_name:tp_osd_tp --------------------------------------

1 0

tracker.ceph.com blocks edge browsers
by Sake Ceph 13 Feb '26

13 Feb '26

Hi! Is there a reason for tracker.ceph.com to block edge browsers? I use it on my Android phone and when I remove in the user agent the edge part, everything works fine. Or use a chrome user agent, everything works. Any help would be appreciated :) Kind regards Sake

1 0

Three Days Left For Ceph Days Raleigh CFP Submissions!
by Anthony Middleton 13 Feb '26

13 Feb '26

ACT NOW! Ceph Days Raleigh CFP Closes Feb. 16th! This Ceph Days offers a full day dedicated to sharing Ceph's transformative power and fostering the vibrant Ceph community in Raleigh, NC! Submit your talk here: https://ceph.io/en/community/events/2026/ceph-days-raleigh/

1 0

Ceph MDS super slow
by Martin Gerhard Loschwitz 13 Feb '26

13 Feb '26

Folks, we are seeing very slow MDSes on a CephFS that has nearly nothing to do. Slow as in „200 metadata requests per second“ slow when putting a bit of load onto it. We managed to get a wallclock of the MDS: [root@node01 gdbpmp]# ./gdbpmp.py -i gdbpmp.data Thread: 7 (mds-rank-fin) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% Finisher::finisher_thread_entry() + 94.12% MDSIOContextBase::complete |+ 88.24% MDSRank::respawn ||+ 88.24% Context::complete || + 88.24% MDSDaemon::respawn || + 88.24% ceph::logging::Log::dump_recent() || + 52.94% ceph::logging::Log::_flush(std::vector<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >&, bool) || |+ 23.53% __snprintf_chk || ||+ 23.53% __vsnprintf_internal || || + 17.65% __vfprintf_internal || || + 5.88% __strchrnul_evex || || + 5.88% _IO_default_xsputn || |+ 17.65% ceph::logging::Log::_flush_logbuf() || ||+ 17.65% ceph::logging::Log::_log_safe_write(std::basic_string_view<char, std::char_traits<char> >) || || + 17.65% safe_write || || + 17.65% write || |+ 5.88% __tz_convert || | + 5.88% __tzfile_compute || | + 5.88% __strcmp_evex || + 5.88% __memmove_avx512_unaligned_erms |+ 5.88% std::lock_guard<ceph::fair_mutex>::lock_guard | + 5.88% ceph::fair_mutex::lock | + 5.88% std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&, ceph::fair_mutex::lock()::{lambda()#1}) | + 5.88% std::condition_variable::wait(std::unique_lock<std::mutex>&) | + 5.88% pthread_cond_wait@@GLIBC_2.3.2 | + 5.88% __futex_abstimed_wait_common + 5.88% std::condition_variable::wait(std::unique_lock<std::mutex>&) + 5.88% pthread_cond_wait@@GLIBC_2.3.2 + 5.88% __futex_abstimed_wait_common Thread: 10 (mds-rank-progr) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% MDSRank::ProgressThread::entry + 100.00% std::_V2::condition_variable_any::wait<std::unique_lock<ceph::fair_mutex>, MDSRank::ProgressThread::entry()::<lambda()> > + 100.00% std::_V2::condition_variable_any::wait<std::unique_lock<ceph::fair_mutex> > + 88.24% std::condition_variable::wait(std::unique_lock<std::mutex>&) |+ 88.24% pthread_cond_wait@@GLIBC_2.3.2 | + 88.24% __futex_abstimed_wait_common + 11.76% std::_V2::condition_variable_any::_Unlock<std::unique_lock<ceph::fair_mutex> >::~_Unlock + 11.76% std::unique_lock<ceph::fair_mutex>::lock + 11.76% ceph::fair_mutex::lock + 11.76% std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&, ceph::fair_mutex::lock()::{lambda()#1}) + 11.76% std::condition_variable::wait(std::unique_lock<std::mutex>&) + 11.76% pthread_cond_wait@@GLIBC_2.3.2 + 11.76% __futex_abstimed_wait_common Thread: 11 (mds-log-trim) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% execute_native_thread_routine + 100.00% MDLog::log_trim_upkeep + 100.00% std::_V2::condition_variable_any::wait_for<std::unique_lock<ceph::fair_mutex>, long, std::ratio<1l, 1000l> > + 100.00% std::_V2::condition_variable_any::wait_until<std::unique_lock<ceph::fair_mutex>, std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > + 82.35% std::_V2::condition_variable_any::_Unlock<std::unique_lock<ceph::fair_mutex> >::~_Unlock |+ 82.35% std::unique_lock<ceph::fair_mutex>::lock | + 82.35% ceph::fair_mutex::lock | + 82.35% std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&, ceph::fair_mutex::lock()::{lambda()#1}) | + 82.35% std::condition_variable::wait(std::unique_lock<std::mutex>&) | + 82.35% pthread_cond_wait@@GLIBC_2.3.2 | + 82.35% __futex_abstimed_wait_common + 17.65% std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > + 17.65% std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > + 17.65% std::__condvar::wait_until + 17.65% pthread_cond_clockwait(a)GLIBC_2.30 + 17.65% __futex_abstimed_wait_common Thread: 12 (mds-cache-trim) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% execute_native_thread_routine + 100.00% MDCache::upkeep_main + 70.59% std::scoped_lock<ceph::fair_mutex>::scoped_lock |+ 70.59% ceph::fair_mutex::lock | + 70.59% std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&, ceph::fair_mutex::lock()::{lambda()#1}) | + 70.59% std::condition_variable::wait(std::unique_lock<std::mutex>&) | + 70.59% pthread_cond_wait@@GLIBC_2.3.2 | + 70.59% __futex_abstimed_wait_common + 23.53% std::condition_variable::wait_for<long, std::ratio<1l, 1000000000l> > |+ 23.53% std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > | + 23.53% std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > | + 23.53% std::__condvar::wait_until | + 23.53% pthread_cond_clockwait(a)GLIBC_2.30 | + 23.53% __futex_abstimed_wait_common + 5.88% MDCache::check_memory_usage + 5.88% MemoryModel::sample + 5.88% MemoryModel::_sample + 5.88% std::getline<char, std::char_traits<char>, std::allocator<char> > + 5.88% std::basic_istream<char, std::char_traits<char> >& std::getline<char, std::char_traits<char>, std::allocator<char> >(std::basic_istream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char , std::char_traits<char>, std::allocator<char> >&, char) + 5.88% std::basic_filebuf<char, std::char_traits<char> >::underflow() + 5.88% std::__basic_file<char>::xsgetn(char*, long) + 5.88% read Thread: 15 (safe_timer) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% CommonSafeTimerThread<std::mutex>::entry() + 100.00% CommonSafeTimer<std::mutex>::timer_thread() + 88.24% pthread_cond_clockwait(a)GLIBC_2.30 |+ 88.24% __futex_abstimed_wait_common + 11.76% std::condition_variable::wait(std::unique_lock<std::mutex>&) + 11.76% pthread_cond_wait@@GLIBC_2.3.2 + 11.76% __futex_abstimed_wait_common Thread: 17 (safe_timer) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% CommonSafeTimerThread<ceph::fair_mutex>::entry() + 100.00% CommonSafeTimer<ceph::fair_mutex>::timer_thread() + 70.59% std::_V2::condition_variable_any::_Unlock<std::unique_lock<ceph::fair_mutex> >::~_Unlock() |+ 70.59% std::condition_variable::wait(std::unique_lock<std::mutex>&) | + 70.59% pthread_cond_wait@@GLIBC_2.3.2 | + 70.59% __futex_abstimed_wait_common + 29.41% pthread_cond_clockwait(a)GLIBC_2.30 + 29.41% __futex_abstimed_wait_common Thread: 20 (ms_dispatch) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% DispatchQueue::DispatchThread::entry() + 100.00% DispatchQueue::entry() + 100.00% MDSDaemon::ms_dispatch2 + 94.12% std::lock_guard<ceph::fair_mutex>::lock_guard |+ 94.12% ceph::fair_mutex::lock | + 94.12% std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&, ceph::fair_mutex::lock()::{lambda()#1}) | + 94.12% std::condition_variable::wait(std::unique_lock<std::mutex>&) | + 94.12% pthread_cond_wait@@GLIBC_2.3.2 | + 94.12% __futex_abstimed_wait_common + 5.88% MDSRankDispatcher::ms_dispatch + 5.88% MDSRank::_dispatch + 5.88% MDSRank::_advance_queues + 5.88% MDSContext::complete + 5.88% Context::complete + 5.88% Server::handle_client_readdir + 5.88% CInode::encode_inodestat + 5.88% encode<optmetadata_singleton<optmetadata_client_t<std::allocator>, std::allocator>, std::allocator> + 5.88% optmetadata_multiton<optmetadata_singleton<optmetadata_client_t<std::allocator>, std::allocator>, std::allocator>::encode + 5.88% ceph::buffer::v15_2_0::list::append_hole(unsigned int) + 5.88% ceph::buffer::v15_2_0::list::refill_append_space(unsigned int) + 5.88% mempool::pool_t::adjust_count(long, long) Thread: 21 (io_context_pool) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% execute_native_thread_routine + 100.00% std::thread::_State_impl<std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::i o_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> > >::_M_run() + 100.00% std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(sh ort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> >::operator()() + 100.00% std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(s hort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) + 100.00% std::__invoke<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1 }&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&, ceph::async::io_context_pool::start(sho rt)::{lambda()#1}&&) + 100.00% std::__invoke_impl<void, make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short): :{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, make_named_thread<ceph::async::io_context_pool::start(short)::{la mbda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}&&, ceph::async::io_context_pool::sta rt(short)::{lambda()#1}&&) + 100.00% make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambd a(auto:1&&, (auto:2&&)...)#1}::operator()<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) const + 100.00% std::invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) + 100.00% std::__invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) + 100.00% std::__invoke_impl<void, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, ceph::async::io_context_pool::start(short)::{lambda()#1}&&) + 100.00% ceph::async::io_context_pool::start(short)::{lambda()#1}::operator()() const + 100.00% boost::asio::io_context::run + 100.00% boost::asio::detail::scheduler::run(boost::system::error_code&) [clone .constprop.0] [clone .isra.0] + 100.00% boost::asio::detail::scheduler::do_run_one + 94.12% boost::asio::detail::conditionally_enabled_event::wait |+ 94.12% boost::asio::detail::posix_event::wait<boost::asio::detail::conditionally_enabled_mutex::scoped_lock> | + 94.12% pthread_cond_wait@@GLIBC_2.3.2 | + 94.12% __futex_abstimed_wait_common + 5.88% boost::asio::detail::scheduler_operation::complete + 5.88% boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>, std::allocator<voi d>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) + 5.88% boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>::operator()() + 5.88% boost::asio::detail::executor_op<boost::asio::detail::binder0<MonClient::handle_config(MConfig*)::{lambda()#2}>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete (void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) [clone .lto_priv.0] + 5.88% ceph::common::ConfigProxy::set_mon_vals(ceph::common::CephContext*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<cha r, std::char_traits<char>, std::allocator<char> >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__c xx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > co nst&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>) [clone .isra.0] + 5.88% md_config_t::set_mon_vals(ceph::common::CephContext*, ConfigValues&, ConfigTracker const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_ _cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocato r<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>) + 5.88% md_config_t::update_legacy_vals(ConfigValues&) + 5.88% md_config_t::update_legacy_val(ConfigValues&, Option const&, std::variant<long ConfigValues::*, unsigned long ConfigValues::*, std::__cxx11::basic_string<char, std::char_traits<char>, s td::allocator<char> > ConfigValues::*, double ConfigValues::*, bool ConfigValues::*, entity_addr_t ConfigValues::*, entity_addrvec_t ConfigValues::*, uuid_d ConfigValues::*>) + 5.88% md_config_t::_get_val(ConfigValues const&, Option const&, boost::container::small_vector<std::pair<Option const*, std::variant<std::monostate, std::__cxx11::basic_string<char, std::cha r_traits<char>, std::allocator<char> >, unsigned long, long, double, bool, entity_addr_t, entity_addrvec_t, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long , std::ratio<1l, 1000l> >, Option::size_t, uuid_d> const*>, 4ul, void, void>*, std::basic_ostream<char, std::char_traits<char> >*) const + 5.88% md_config_t::_get_val_nometa[abi:cxx11](ConfigValues const&, Option const&) const Thread: 22 (io_context_pool) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% execute_native_thread_routine + 100.00% std::thread::_State_impl<std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::i o_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> > >::_M_run() + 100.00% std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(sh ort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> >::operator()() + 100.00% std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(s hort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) + 100.00% std::__invoke<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1 }&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&, ceph::async::io_context_pool::start(sho rt)::{lambda()#1}&&) + 100.00% std::__invoke_impl<void, make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short): :{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, make_named_thread<ceph::async::io_context_pool::start(short)::{la mbda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1}&&, ceph::async::io_context_pool::sta rt(short)::{lambda()#1}&&) + 100.00% make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambd a(auto:1&&, (auto:2&&)...)#1}::operator()<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) const + 100.00% std::invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) + 100.00% std::__invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&) + 100.00% std::__invoke_impl<void, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, ceph::async::io_context_pool::start(short)::{lambda()#1}&&) + 100.00% ceph::async::io_context_pool::start(short)::{lambda()#1}::operator()() const + 100.00% boost::asio::io_context::run + 100.00% boost::asio::detail::scheduler::run(boost::system::error_code&) [clone .constprop.0] [clone .isra.0] + 100.00% boost::asio::detail::scheduler::do_run_one + 100.00% boost::asio::detail::conditionally_enabled_event::wait + 100.00% boost::asio::detail::posix_event::wait<boost::asio::detail::conditionally_enabled_mutex::scoped_lock> + 100.00% pthread_cond_wait@@GLIBC_2.3.2 + 100.00% __futex_abstimed_wait_common Thread: 26 (msgr-worker-2) - 17 samples + 100.00% clone + 100.00% start_thread + 100.00% execute_native_thread_routine + 100.00% std::_Function_handler<void (), NetworkStack::add_thread(Worker*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) + 100.00% EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*) + 94.12% EpollDriver::event_wait(std::vector<FiredFileEvent, std::allocator<FiredFileEvent> >&, timeval*) |+ 94.12% epoll_wait + 5.88% AsyncConnection::process() + 5.88% ProtocolV2::run_continuation(Ct<ProtocolV2>&) + 5.88% ProtocolV2::handle_message() + 5.88% decode_message(ceph::common::CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&, boost::intrusive_ptr<Conne ction>) + 5.88% MMgrMap::decode_payload() + 5.88% MgrMap::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&) + 5.88% MgrMap::ModuleInfo::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&) + 5.88% std::enable_if<denc_traits<std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocat or<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, void>::supported&&(!denc_traits<std::set<std::__cxx11::basic_string<char, std::cha r_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_trait s<char>, std::allocator<char> > > >, void>::need_contiguous), void>::type ceph::decode<std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cx x11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, denc_traits<std::set<s td::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__c xx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, void> >(std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx1 1::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, ceph::buffer::v15_2_0:: list::iterator_impl<true>&) + 5.88% ceph::buffer::v15_2_0::ptr::release() Sidenote: while recording this, the MDS became so slow that the MDS cluster declared it dead … As can be seen, the MDS is waiting in futex to no end, but how can we find out what it is waiting for? FS has many small files but nearly no activity …. Thanks in advance Martin -- Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH Phone: +49 2433 5253130 Mobile: +49 176 61832178 Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany Legal: HRB 21985, Amtsgericht Mönchengladbach VAT: DE363893844 True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.

1 0

reef 18.2.8 QE validation status
by Yuri Weinstein 13 Feb '26

13 Feb '26

Note: this is an attempt to test and publish post EOL reef point release Details of this release are summarized here: https://tracker.ceph.com/issues/73906#note-1 Release Notes - TBD LRC upgrade - N/A distro => jammy focal centos9 windows bookworm Seeking approvals/reviews for: smoke - Laura approved? rados - Radek, Laura approved? Travis? Nizamudeen? Adam King approved? (rados is also running on build 2 == build 1 + https://github.com/ceph/ceph/pull/66056 per Igor's request) rgw - Adam E approved? fs - Venky approved? orch - Adam King approved? rbd - Ilya approved? krbd - Ilya approved? upgrade/pacific-x (reef) - can this be deprecated? Josh? Neha? (same as in 18.2.5) upgrade/quincy-x (reef) - Laura, Prashant please take a look. (same as in 18.2.5) powercycle - Brad approved? ceph-volume - see orch, Guillaume approved? Notably upgrades suites show same results as for 18.2.5 (tested in March 2025) To compare, see 18.2.5 tests https://tracker.ceph.com/issues/70563#note-1 TIA

4 5

Re: mgmt-gateway and oauth2-proxy for radosgw?
by Casey Bodley 13 Feb '26

13 Feb '26

hi Florian, On Thu, Feb 12, 2026 at 11:05 AM Florian Haas via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi everyone, > > a question about the mgmt-gateway and oauth2-proxy services, because we > want to incorporate one or both of them into our Ceph training course. > > I understand that as of Tentacle, mgmt-gateway exists as an > administrative simplification for access to the Dashboard and the > orchestrated monitoring stack,[0] and that it has OpenID Connect (OIDC) > support via oauth2-proxy. The docs[1] assert that oauth2-proxy serves as > "an advanced method for managing authentication and access control for > Ceph applications". > > We've also had OIDC support in radosgw since Octopus (I believe), > however it is tied to an implementation of a subset of STS[2], > configuration is rather non-trivial[3], and as far as I understand it is > presently not integrated with oauth2-proxy. > > Now my question is: will oauth2-proxy eventually be integrated with > radosgw to replace and remove the STS dependency, or are the two OIDC > integrations expected to coexist in parallel? i don't know that radosgw has any plans for oidc integration outside of sts and AssumeRoleWithWebIdentity. the benefit of sts for this integration is that it works with unmodified aws clients the user account feature in squid added fine-grained api-based control over oidc providers, roles, and their associated iam policy: https://docs.ceph.com/en/squid/radosgw/account/ if configuration was your main obstacle, i would hope that the aws-compatible tooling like https://docs.aws.amazon.com/cli/latest/reference/iam/create-open-id-connect… would make that easier > > Thanks! > > Cheers, > Florian > > [0] https://docs.ceph.com/en/latest/cephadm/services/mgmt-gateway/ > [1] https://docs.ceph.com/en/latest/cephadm/services/oauth2-proxy/ > [2] https://docs.ceph.com/en/latest/radosgw/oidc/ > [3] > https://community.ibm.com/community/user/blogs/deepak-thorat/2024/03/20/cep… > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: Data is still actively corrupted!
by Eugen Block 13 Feb '26

13 Feb '26

Hi, you seem to hit this issue [0] with the read_from_replica option, see the announcement in [1]. But I haven't looked in detail, not sure if there's a way to fix it and if read_from_replica=balance has the same effect. Regards, Eugen [0] https://tracker.ceph.com/issues/73997 [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/JI2ZRF7A3PW… Am Do., 12. Feb. 2026 um 10:46 Uhr schrieb dominik.baack via ceph-users < ceph-users(a)ceph.io>: > Hi, > > thanks for your reply. > > Mounting was done without the 'root_squash' option, here is the > corresponding fstab entry: > > > 192.168.251.2,192.168.251.3,192.168.251.4,192.168.251.5,192.168.251.6,192.168.251.7:/ > > /cephfs ceph > name=gpu01,secretfile=/etc/ceph/gpu01.key,noatime,_netdev,recover_session=clean,read_from_replica=balance > > 0 0 > > Dominik > > > Am 2026-02-12 09:51, schrieb goetze: > > Hi ! > > > > Have you mounted cephfs with the 'root_squash' option set? If so, > > remove that option. I may be wrong here, but as far as I know, this is > > still considered unsafe and can lead to data corruption since the > > necessary code changes have not yet made it into the mainstream linux > > kernel. > > > > Carsten > > ------------------------------------------------------------------ > > Carsten Goetze > > Computer Graphics tel: +49 531 391-2109 > > TU Braunschweig fax: +49 531 391-2103 > > Muehlenpfordtstr. 23 eMail: c.goetze(a)cg.cs.tu-bs.de > > D-38106 Braunschweig http://www.cg.cs.tu-bs.de/people/goetze > > > >> Am 12.02.2026 um 07:15 schrieb Dominik Baack via ceph-users > >> <ceph-users(a)ceph.io>: > >> > >> Hi, > >> > >> I noticed now that files are still actively corrupted / replaced by > >> empty files when open saved. > >> > >> Access was done via Ceph 19.2.1 from Ubuntu 24 with kernel mount. > >> Ceph servers are still running 20.2 Tentacle (deployed via cephadm) > >> with an Ubuntu 24.04 host. > >> > >> Options currently set are to noout, norebalance, noscrub and > >> nodeep-scrub > >> > >> I am currently setting up a read only mount to copy all existing > >> data for a backup. > >> > >> I have currently no clue whats going on, I currently was able to > >> observe this behavior only on the nodes. As those were upgraded as > >> well (MLNX driver, nvidia-fs,...) can it be network? > >> > >> Any Idea how to recover from this? > >> > >> Cheers > >> Dominik > >> > >> Am 11.02.2026 um 18:44 schrieb dominik.baack via ceph-users: > >> > >>> Hi, > >>> > >>> after an controlled shutdown of the whole cluster do to external > >>> circumstances we decided to update from 19.2 to 20.2 after the > >>> restart. The system was health before and after the update. > >>> The nodes mounting the filesystem were not equally lucky and were > >>> partially shutdown hard. Storage was kept running additional > >>> ~30min after node shutdown, all inflight operations should have > >>> finished. > >>> > >>> Now we discover the some of the user files seem to be replaced > >>> with zeros. For example: > >>> > >>> stat .gitignore > >>> File: .gitignore > >>> Size: 4429 Blocks: 9 IO Block: 4194304 > >>> regular file > >>> Device: 0,48 Inode: 1100241384598 Links: 1 > >>> > >>> hexdump -C .gitignore > >>> 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > >>> |................| > >>> * > >>> 00001140 00 00 00 00 00 00 00 00 00 00 00 00 00 |.............| > >>> 0000114d > >>> > >>> Scanning for files containing only zeros show several issues of > >>> files that were likely accessed before or during the shutdown of > >>> the nodes. > >>> > >>> How should I progress from here? > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: Why did this situation occur?
by Jayanth Reddy 13 Feb '26

13 Feb '26

Hi, Just restart your manager daemons using `ceph orch restart mgr` and see the updated status. AFAIK Quincy and earlier versions had problems with mgrs where stale data was shown and a simple daemon restart would fix it. Additionally, it is extremely recommended to use a monitoring system for the Operating System as well as Ceph. Ceph dashboard also should shows you problems with Ceph and your monitoring system for the OS should catch any abnormal disk usage. Thanks ________________________________ From: 苏察哈尔灿 via ceph-users <ceph-users(a)ceph.io> Sent: Friday, February 13, 2026 7:55:51 AM To: ceph-users <ceph-users(a)ceph.io> Subject: [ceph-users] Why did this situation occur? Please help me, I have a CEPH cluster with six nodes. Yesterday, one of the nodes reported insufficient disk space (image01.png), but using the command "ceph -s", it kept showing normal (image02.png). Before this, there was also a situation where the disk space of a certain node was insufficient. At that time, using the "ceph -s" command could normally display the alarm and the involved nodes. Why is this situation occurring now? Filesystem                    Type  Size  Used Avail Use% Mounted on tmpfs                         tmpfs   3.2G  3.6M  3.1G   1% /run /dev/mapper/ubuntu--vg-ubuntu--lv ext4    98G   93G   863M  100% / tmpfs                         tmpfs   16G     0     16G    0% /dev/shm tmpfs                         tmpfs   5.0M    0     5.0M   0% /run/lock /dev/sdj2                     ext4    2.0G  253M  1.6G   14% /boot /dev/sdj1                     vfat    1.1G  6.1M  1.1G   1% /boot/efi tmpfs                         tmpfs   3.2G  4.0K  3.2G   1% /run/user/0 overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/b5a55615f548ae182d48dee0eced64eb2ed43c37d696d951618e66b01b700/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/3c6ecfc4d8a3561835504c3aaee7f69ea7a117c26d3ec59ea1ceabe5994fd57/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/a504698e7cc386ebefd506402638c2f8af83ad9a35a8d0c49a7c287d8d26c24/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/380d370bc8313724d95b8fc7c70d7283c69439b014ce82936b245ee6b810c/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/71ba8a75e186df7542a2fa9667580e4b2b0db2fb78eacfea3711abe6c9d22ad/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/8daea1eaddbc8c3b0f7afa35b58524ec3adcfcf48a304f988e5adbec2c4ffa/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/04f8e8635d8bc88c953b00815bc3234f26499f6eb066ae998121bf0dd4a03b4/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/bafbe7cbad3f4ab94a92eb5f098591ac316de90def61c2ca1b69559d96b62/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/4a0aa4bad8692e397ca98ed91a042e22caeb7509d3825f597bb37e9fb98a1cca/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/9c2e8075696b5edb2a38e4f8d7e866ada67b5d238ff73f8ea7b0f0669cf06660a/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/8251b5ef9df5394947e5d8018a163822aac69ab1fe378cfa7359cb42d1788db/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/0762e8c175c6b568042a8e8a12b90bb46dc76b9bc41ccc972ee56c6670e2f5/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/c7f232a9a0abe297fbcfe58f5242754078a669b826726a4ed7146d1e5619cefc/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/8186f868a3e87299a360a60f780cef452556fe00ace340433f11e7f922c12124/merged cluster:   id: f68c4e2e-e880-11ef-b13e-512ad216ec40   health: HEALTH_OK services:   mon: 5 daemons, quorum ceph-node001,ceph-node006,ceph-node002,ceph-node003,ceph-node005 (age 44h)   mgr: ceph-node001.zcyayo(active, since 12H), standbys: ceph-node002.wvpazs   mds: 1/1 daemons up, 1 standby   osd: 60 osds: 60 up (since 5w), 60 in (since 7M) data:   volumes: 1/1 healthy   pools: 6 pools, 2625 pg   objects: 42.59M objects, 160 TiB   usage: 477 TiB used, 396 TiB / 873 TiB avail   pg:     2591 active+clean     28 active+clean+scrubbing+deep     6 active+clean+scrubbing ceph version 17.2.7 (b12291d11b049b2f35a32e0de30d70e9a4c060d2), quincy (stable) CLASS    SIZE      AVAIL     USED      RAW USED  %RAW USED hdd      873 TiB   396 TiB   477 TiB   477 TiB   54.68 TOTAL    873 TiB   396 TiB   477 TiB   477 TiB   54.68 POOL               ID    PGS     STORAGED   OBJECTS   USED       %USED    MAX AVAIL .mgr               1     1       978 MiB    161       2.9 GiB    0        108 TiB vmware-server      2     1024    155 TiB    41.5M     464 TiB    58.79    108 TiB cephfs-metadata    4     32      195 MiB    69        584 MiB    0        108 TiB cephfs-data        5     512     1.9 GiB    490       5.7 GiB    0        108 TiB .nfs               6     32      0 B        0         0 B        0        108 TiB backup-pool        8     1024    4.1 TiB    1.0M      12 TiB     3.63     108 TiB NAME                     PORTS            RUNNING  REFRESHED  AGE   PLACEMENT alertmanager             ?:9093,9094      1/1      12s ago    12M   count:1 crash                    -                6/6      6m ago     12M   * graphana                 ?:3000           1/1      12s ago    12M   count:1 mds.cephfs-vmware-server -                2/2      4m ago     12M   count:2 mgr                      -                2/2      4m ago     12M   count:2 mon                      -                5/5      6m ago     12M   count:5 node-exporter            ?:9100           6/6      6m ago     12M   * osd                      -                60       -          -     <unmanaged> prometheus               ?:9095           1/1      12s ago    12M   count:1 NAME                              HOST             PORTS                  STATUS      REFRESHED  AGE   MEM USE  MEM LIM  VERSTION  IMAGE ID              CONTAINER ID alertmanager.ceph-node001         ceph-node001     *:9093,9094            running (12M)  10m ago  12M   32.2K    -        0.25.0                c8568f9114cd2         0bda2adb384 crash.ceph-node001                ceph-node001     -                      running (12M)  10m ago  12M   3479k    -        17.2.8                259b35566514          6bdcadd825e5 crash.ceph-node002                ceph-node002     -                      running (5M)   3m ago   12M   5227k    -        17.2.8                259b35566514          4f315378f8fb crash.ceph-node003                ceph-node003     -                      running (5M)   3m ago   12M   4328k    -        17.2.8                259b35566514          c89f20d5244 crash.ceph-node004                ceph-node004     -                      running (7M)   5m ago   7M    10.9M    -        17.2.8                259b35566514          e3f0a0a49d6a crash.ceph-node005                ceph-node005     -                      running (5M)   3m ago   12M   4551k    -        17.2.8                259b35566514          a3028d149a9c crash.ceph-node006                ceph-node006     -                      running (12M)  5m ago   12M   4219k    -        17.2.8                259b35566514          5458ecc46771 grafana.ceph-node001              ceph-node001     *:3000                 running (12M)  10m ago  12M   133M     -        9.4.7                 954c88fa6188          5536bc4858e4 mds.cephfs-vmware-server.ceph-node002.gryudx ceph-node002     -                      running (5M)   3m ago   12M   15.3M    -        17.2.8                259b35566514          b7bd424d9996 mds.cephfs-vmware-server.ceph-node005.qyxswr ceph-node005     -                      running (5M)   3m ago   12M   25.8M    -        17.2.8                259b35566514          3be6f2e7c7ce mgr.ceph-node001.zcyayo           ceph-node001     *:9283                 running (12M)  10m ago  12M   891M     -        17.2.8                259b35566514          daa1ea0d1168 mgr.ceph-node002.uvpazs           ceph-node002     *:8443,9283            running (5M)   3m ago   12M   57.3M    -        17.2.8                259b35566514          81a8cd1fe2c3 mon.ceph-node001                  ceph-node001     -                      running (7M)   10m ago  12M   477M     2048M    17.2.8                259b35566514          3e5c22bfedd4 mon.ceph-node002                  ceph-node002     -                      running (5M)   10m ago  12M   398M     2048M    17.2.8                259b35566514          c15e5b4c21c8 mon.ceph-node003                  ceph-node003     -                      running (12M)  5m ago   12M   290M     2048M    17.2.8                259b35566514          72facfdf82a8 mon.ceph-node005                  ceph-node005     -                      running (7M)   3m ago   7M    832M     2048M    17.2.8                259b35566514          3f94eb72a5e3 mon.ceph-node006                  ceph-node006     -                      running (12M)  5m ago   12M   285M     2048M    17.2.8                259b35566514          97a4a1d76116 node-exporter.ceph-node001        ceph-node001     *:9100                 running (12M)  10m ago  12M   17.3M    -        1.5.0                 0da6a335fe13          81b476bbe6fc node-exporter.ceph-node002        ceph-node002     *:9100                 running (5M)   3m ago   12M   20.2M    -        1.5.0                 0da6a335fe13          ba20a215dd70 node-exporter.ceph-node003        ceph-node003     *:9100                 running (5M)   3m ago   12M   17.4M    -        1.5.0                 0da6a335fe13          ef75144ce2b3 node-exporter.ceph-node004        ceph-node004     *:9100                 running (7M)   5m ago   7M    20.7M    -        1.5.0                 0da6a335fe13          a04852da78e8 node-exporter.ceph-node005        ceph-node005     *:9100                 running (12M)  3m ago   12M   17.1M    -        1.5.0                 0da6a335fe13          d07533213e5c node-exporter.ceph-node006        ceph-node006     *:9100                 running (12M)  5m ago   12M   16.8M    -        1.5.0                 0da6a335fe13          23f1ceec45a osd.0                             ceph-node001     -                      running (12M)  10m ago  12M   1885M    1287M    17.2.8                259b35566514          9b33a19ff54c osd.1                             ceph-node001     -                      running (12M)  10m ago  12M   1711M    1287M    17.2.8                259b35566514          49eea8cdb06d osd.2                             ceph-node001     -                      running (12M)  10m ago  12M   2118M    1287M    17.2.8                259b35566514          25cdd9efdd95 osd.3                             ceph-node001     -                      running (12M)  10m ago  12M   1639M    1287M    17.2.8                259b35566514          ed50fbe841f osd.4                             ceph-node001     -                      running (12M)  10m ago  12M   1828M    1287M    17.2.8                259b35566514          56109a380a52 osd.5                             ceph-node001     -                      running (8w)   10m ago  12M   2335M    1287M    17.2.8                259b35566514          b1f65b1df237 osd.6                             ceph-node001     -                      running (12M)  10m ago  12M   2088M    1287M    17.2.8                259b35566514          ca7d1a2835ab osd.7                             ceph-node001     -                      running (12M)  10m ago  12M   2055M    1287M    17.2.8                259b35566514          fafd75e72d5f osd.8                             ceph-node001     -                      running (12M)  10m ago  12M   2033M    1287M    17.2.8                259b35566514          1cdc6b774429 osd.9                             ceph-node002     -                      running (5M)   3m ago   12M   2307M    1185M    17.2.8                259b35566514          117c76b6d10 osd.10                            ceph-node002     -                      running (5M)   3m ago   12M   1683M    1185M    17.2.8                259b35566514          d1ce2269d143 osd.11                            ceph-node002     -                      running (5M)   3m ago   12M   2008M    1185M    17.2.8                259b35566514          b69fbf088ee osd.12                            ceph-node002     -                      running (5M)   3m ago   12M   2442M    1185M    17.2.8                259b35566514          df71fa4dd602 osd.13                            ceph-node002     -                      running (5M)   3m ago   12M   1803M    1185M    17.2.8                259b35566514          86d54722e9ab osd.14                            ceph-node002     -                      running (5M)   3m ago   12M   1999M    1185M    17.2.8                259b35566514          ab1f5734cf6f My cluster is version 17.2.7 (quincy). The basic information of the Ceph cluster is provided in the other attached pictures. Thank you! _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Why did this situation occur?
by 苏察哈尔灿 13 Feb '26

13 Feb '26

Please help me, I have a CEPH cluster with six nodes. Yesterday, one of the nodes reported insufficient disk space (image01.png), but using the command "ceph -s", it kept showing normal (image02.png). Before this, there was also a situation where the disk space of a certain node was insufficient. At that time, using the "ceph -s" command could normally display the alarm and the involved nodes. Why is this situation occurring now? Filesystem                    Type  Size  Used Avail Use% Mounted on tmpfs                         tmpfs   3.2G  3.6M  3.1G   1% /run /dev/mapper/ubuntu--vg-ubuntu--lv ext4    98G   93G   863M  100% / tmpfs                         tmpfs   16G     0     16G    0% /dev/shm tmpfs                         tmpfs   5.0M    0     5.0M   0% /run/lock /dev/sdj2                     ext4    2.0G  253M  1.6G   14% /boot /dev/sdj1                     vfat    1.1G  6.1M  1.1G   1% /boot/efi tmpfs                         tmpfs   3.2G  4.0K  3.2G   1% /run/user/0 overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/b5a55615f548ae182d48dee0eced64eb2ed43c37d696d951618e66b01b700/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/3c6ecfc4d8a3561835504c3aaee7f69ea7a117c26d3ec59ea1ceabe5994fd57/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/a504698e7cc386ebefd506402638c2f8af83ad9a35a8d0c49a7c287d8d26c24/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/380d370bc8313724d95b8fc7c70d7283c69439b014ce82936b245ee6b810c/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/71ba8a75e186df7542a2fa9667580e4b2b0db2fb78eacfea3711abe6c9d22ad/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/8daea1eaddbc8c3b0f7afa35b58524ec3adcfcf48a304f988e5adbec2c4ffa/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/04f8e8635d8bc88c953b00815bc3234f26499f6eb066ae998121bf0dd4a03b4/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/bafbe7cbad3f4ab94a92eb5f098591ac316de90def61c2ca1b69559d96b62/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/4a0aa4bad8692e397ca98ed91a042e22caeb7509d3825f597bb37e9fb98a1cca/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/9c2e8075696b5edb2a38e4f8d7e866ada67b5d238ff73f8ea7b0f0669cf06660a/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/8251b5ef9df5394947e5d8018a163822aac69ab1fe378cfa7359cb42d1788db/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/0762e8c175c6b568042a8e8a12b90bb46dc76b9bc41ccc972ee56c6670e2f5/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/c7f232a9a0abe297fbcfe58f5242754078a669b826726a4ed7146d1e5619cefc/merged overlay                       overlay 98G   93G   863M  100% /var/lib/docker/overlay2/8186f868a3e87299a360a60f780cef452556fe00ace340433f11e7f922c12124/merged cluster:   id: f68c4e2e-e880-11ef-b13e-512ad216ec40   health: HEALTH_OK services:   mon: 5 daemons, quorum ceph-node001,ceph-node006,ceph-node002,ceph-node003,ceph-node005 (age 44h)   mgr: ceph-node001.zcyayo(active, since 12H), standbys: ceph-node002.wvpazs   mds: 1/1 daemons up, 1 standby   osd: 60 osds: 60 up (since 5w), 60 in (since 7M) data:   volumes: 1/1 healthy   pools: 6 pools, 2625 pg   objects: 42.59M objects, 160 TiB   usage: 477 TiB used, 396 TiB / 873 TiB avail   pg:     2591 active+clean     28 active+clean+scrubbing+deep     6 active+clean+scrubbing ceph version 17.2.7 (b12291d11b049b2f35a32e0de30d70e9a4c060d2), quincy (stable) CLASS    SIZE      AVAIL     USED      RAW USED  %RAW USED hdd      873 TiB   396 TiB   477 TiB   477 TiB   54.68 TOTAL    873 TiB   396 TiB   477 TiB   477 TiB   54.68 POOL               ID    PGS     STORAGED   OBJECTS   USED       %USED    MAX AVAIL .mgr               1     1       978 MiB    161       2.9 GiB    0        108 TiB vmware-server      2     1024    155 TiB    41.5M     464 TiB    58.79    108 TiB cephfs-metadata    4     32      195 MiB    69        584 MiB    0        108 TiB cephfs-data        5     512     1.9 GiB    490       5.7 GiB    0        108 TiB .nfs               6     32      0 B        0         0 B        0        108 TiB backup-pool        8     1024    4.1 TiB    1.0M      12 TiB     3.63     108 TiB NAME                     PORTS            RUNNING  REFRESHED  AGE   PLACEMENT alertmanager             ?:9093,9094      1/1      12s ago    12M   count:1 crash                    -                6/6      6m ago     12M   * graphana                 ?:3000           1/1      12s ago    12M   count:1 mds.cephfs-vmware-server -                2/2      4m ago     12M   count:2 mgr                      -                2/2      4m ago     12M   count:2 mon                      -                5/5      6m ago     12M   count:5 node-exporter            ?:9100           6/6      6m ago     12M   * osd                      -                60       -          -     <unmanaged> prometheus               ?:9095           1/1      12s ago    12M   count:1 NAME                              HOST             PORTS                  STATUS      REFRESHED  AGE   MEM USE  MEM LIM  VERSTION  IMAGE ID              CONTAINER ID alertmanager.ceph-node001         ceph-node001     *:9093,9094            running (12M)  10m ago  12M   32.2K    -        0.25.0                c8568f9114cd2         0bda2adb384 crash.ceph-node001                ceph-node001     -                      running (12M)  10m ago  12M   3479k    -        17.2.8                259b35566514          6bdcadd825e5 crash.ceph-node002                ceph-node002     -                      running (5M)   3m ago   12M   5227k    -        17.2.8                259b35566514          4f315378f8fb crash.ceph-node003                ceph-node003     -                      running (5M)   3m ago   12M   4328k    -        17.2.8                259b35566514          c89f20d5244 crash.ceph-node004                ceph-node004     -                      running (7M)   5m ago   7M    10.9M    -        17.2.8                259b35566514          e3f0a0a49d6a crash.ceph-node005                ceph-node005     -                      running (5M)   3m ago   12M   4551k    -        17.2.8                259b35566514          a3028d149a9c crash.ceph-node006                ceph-node006     -                      running (12M)  5m ago   12M   4219k    -        17.2.8                259b35566514          5458ecc46771 grafana.ceph-node001              ceph-node001     *:3000                 running (12M)  10m ago  12M   133M     -        9.4.7                 954c88fa6188          5536bc4858e4 mds.cephfs-vmware-server.ceph-node002.gryudx ceph-node002     -                      running (5M)   3m ago   12M   15.3M    -        17.2.8                259b35566514          b7bd424d9996 mds.cephfs-vmware-server.ceph-node005.qyxswr ceph-node005     -                      running (5M)   3m ago   12M   25.8M    -        17.2.8                259b35566514          3be6f2e7c7ce mgr.ceph-node001.zcyayo           ceph-node001     *:9283                 running (12M)  10m ago  12M   891M     -        17.2.8                259b35566514          daa1ea0d1168 mgr.ceph-node002.uvpazs           ceph-node002     *:8443,9283            running (5M)   3m ago   12M   57.3M    -        17.2.8                259b35566514          81a8cd1fe2c3 mon.ceph-node001                  ceph-node001     -                      running (7M)   10m ago  12M   477M     2048M    17.2.8                259b35566514          3e5c22bfedd4 mon.ceph-node002                  ceph-node002     -                      running (5M)   10m ago  12M   398M     2048M    17.2.8                259b35566514          c15e5b4c21c8 mon.ceph-node003                  ceph-node003     -                      running (12M)  5m ago   12M   290M     2048M    17.2.8                259b35566514          72facfdf82a8 mon.ceph-node005                  ceph-node005     -                      running (7M)   3m ago   7M    832M     2048M    17.2.8                259b35566514          3f94eb72a5e3 mon.ceph-node006                  ceph-node006     -                      running (12M)  5m ago   12M   285M     2048M    17.2.8                259b35566514          97a4a1d76116 node-exporter.ceph-node001        ceph-node001     *:9100                 running (12M)  10m ago  12M   17.3M    -        1.5.0                 0da6a335fe13          81b476bbe6fc node-exporter.ceph-node002        ceph-node002     *:9100                 running (5M)   3m ago   12M   20.2M    -        1.5.0                 0da6a335fe13          ba20a215dd70 node-exporter.ceph-node003        ceph-node003     *:9100                 running (5M)   3m ago   12M   17.4M    -        1.5.0                 0da6a335fe13          ef75144ce2b3 node-exporter.ceph-node004        ceph-node004     *:9100                 running (7M)   5m ago   7M    20.7M    -        1.5.0                 0da6a335fe13          a04852da78e8 node-exporter.ceph-node005        ceph-node005     *:9100                 running (12M)  3m ago   12M   17.1M    -        1.5.0                 0da6a335fe13          d07533213e5c node-exporter.ceph-node006        ceph-node006     *:9100                 running (12M)  5m ago   12M   16.8M    -        1.5.0                 0da6a335fe13          23f1ceec45a osd.0                             ceph-node001     -                      running (12M)  10m ago  12M   1885M    1287M    17.2.8                259b35566514          9b33a19ff54c osd.1                             ceph-node001     -                      running (12M)  10m ago  12M   1711M    1287M    17.2.8                259b35566514          49eea8cdb06d osd.2                             ceph-node001     -                      running (12M)  10m ago  12M   2118M    1287M    17.2.8                259b35566514          25cdd9efdd95 osd.3                             ceph-node001     -                      running (12M)  10m ago  12M   1639M    1287M    17.2.8                259b35566514          ed50fbe841f osd.4                             ceph-node001     -                      running (12M)  10m ago  12M   1828M    1287M    17.2.8                259b35566514          56109a380a52 osd.5                             ceph-node001     -                      running (8w)   10m ago  12M   2335M    1287M    17.2.8                259b35566514          b1f65b1df237 osd.6                             ceph-node001     -                      running (12M)  10m ago  12M   2088M    1287M    17.2.8                259b35566514          ca7d1a2835ab osd.7                             ceph-node001     -                      running (12M)  10m ago  12M   2055M    1287M    17.2.8                259b35566514          fafd75e72d5f osd.8                             ceph-node001     -                      running (12M)  10m ago  12M   2033M    1287M    17.2.8                259b35566514          1cdc6b774429 osd.9                             ceph-node002     -                      running (5M)   3m ago   12M   2307M    1185M    17.2.8                259b35566514          117c76b6d10 osd.10                            ceph-node002     -                      running (5M)   3m ago   12M   1683M    1185M    17.2.8                259b35566514          d1ce2269d143 osd.11                            ceph-node002     -                      running (5M)   3m ago   12M   2008M    1185M    17.2.8                259b35566514          b69fbf088ee osd.12                            ceph-node002     -                      running (5M)   3m ago   12M   2442M    1185M    17.2.8                259b35566514          df71fa4dd602 osd.13                            ceph-node002     -                      running (5M)   3m ago   12M   1803M    1185M    17.2.8                259b35566514          86d54722e9ab osd.14                            ceph-node002     -                      running (5M)   3m ago   12M   1999M    1185M    17.2.8                259b35566514          ab1f5734cf6f My cluster is version 17.2.7 (quincy). The basic information of the Ceph cluster is provided in the other attached pictures. Thank you!

1 0

ceph
by 苏察哈尔灿 13 Feb '26

13 Feb '26

Please help me, I have a CEPH cluster with six nodes. Yesterday, one of the nodes reported insufficient disk space (image01.png), but using the command "ceph -s", it kept showing normal (image02.png). Before this, there was also a situation where the disk space of a certain node was insufficient. At that time, using the "ceph -s" command could normally display the alarm and the involved nodes. Why is this situation occurring now?  My cluster is version 17.2.7 (quincy). The basic information of the Ceph cluster is provided in the other attached pictures. Thank you!

1 0

Why did this situation occur?
by 苏察哈尔灿 13 Feb '26

13 Feb '26

Please help me, I have a CEPH cluster with six nodes. Yesterday, one of the nodes reported insufficient disk space (image01.png), but using the command "ceph -s", it kept showing normal (image02.png). Before this, there was also a situation where the disk space of a certain node was insufficient. At that time, using the "ceph -s" command could normally display the alarm and the involved nodes. Why is this situation occurring now?  My cluster is version 17.2.7 (quincy). The basic information of the Ceph cluster is provided in the other attached pictures. Thank you!

1 0

Re: reef 18.2.8 QE validation status
by Ilya Dryomov 12 Feb '26

12 Feb '26

On Thu, Feb 12, 2026 at 4:58 PM Yuri Weinstein via Dev <dev(a)ceph.io> wrote: > > Still seeking approvals for: > > (smoke - FAILED, was reviewed and approved by Laura and Patrick, see Slack thread for details) > rados - FAILED, Afreen, Travis? ( Laura approved for core) > > rgw - FAILED, Adam E approved? > > (rbd, krbd - FAILED, Ilya approved) I approved only krbd. For rbd, I highlighted a test failure that hasn't been root caused and is therefore deemed concerning (even if it's likely to go unnoticed by actual users). Thanks, Ilya

2 1

mgmt-gateway and oauth2-proxy for radosgw?
by Florian Haas 12 Feb '26

12 Feb '26

Hi everyone, a question about the mgmt-gateway and oauth2-proxy services, because we want to incorporate one or both of them into our Ceph training course. I understand that as of Tentacle, mgmt-gateway exists as an administrative simplification for access to the Dashboard and the orchestrated monitoring stack,[0] and that it has OpenID Connect (OIDC) support via oauth2-proxy. The docs[1] assert that oauth2-proxy serves as "an advanced method for managing authentication and access control for Ceph applications". We've also had OIDC support in radosgw since Octopus (I believe), however it is tied to an implementation of a subset of STS[2], configuration is rather non-trivial[3], and as far as I understand it is presently not integrated with oauth2-proxy. Now my question is: will oauth2-proxy eventually be integrated with radosgw to replace and remove the STS dependency, or are the two OIDC integrations expected to coexist in parallel? Thanks! Cheers, Florian [0] https://docs.ceph.com/en/latest/cephadm/services/mgmt-gateway/ [1] https://docs.ceph.com/en/latest/cephadm/services/oauth2-proxy/ [2] https://docs.ceph.com/en/latest/radosgw/oidc/ [3] https://community.ibm.com/community/user/blogs/deepak-thorat/2024/03/20/cep…

1 0

Re: reef 18.2.8 QE validation status
by Yuri Weinstein 12 Feb '26

12 Feb '26

Still seeking approvals for: (smoke - FAILED, was reviewed and approved by Laura and Patrick, see Slack thread for details) rados - FAILED, Afreen, Travis? ( Laura approved for core) rgw - FAILED, Adam E approved? (rbd, krbd - FAILED, Ilya approved) fs - FAILED, Venky approved? orch - FAILED, Adam K approved? upgrade/pacific-x (reef) - FAILED Laura, approved? upgrade/quincy-x (reef) - FAILED Laura, approved? upgrade/reef-p2p - FAILED Laura, approved? (powercycle - FAILED Brad, approved) ceph-volume - Approved Guillaume? On Wed, Feb 11, 2026 at 12:10 PM Laura Flores via Dev <dev(a)ceph.io> wrote: > The rados suite is approved: > https://tracker.ceph.com/projects/rados/wiki/REEF#reef-v1828-httpstrackerce… > > On Tue, Feb 10, 2026 at 6:33 PM Brad Hubbard via Dev <dev(a)ceph.io> wrote: > >> Powercycle approved. >> >> On Tue, Feb 10, 2026 at 10:35 AM Brad Hubbard via Dev <dev(a)ceph.io> >> wrote: >> >>> >>> >>> On Fri, Feb 6, 2026 at 12:36 PM Brad Hubbard <bhubbard(a)redhat.com> >>> wrote: >>> >>>> >>>> >>>> On Fri, Feb 6, 2026 at 8:33 AM Brad Hubbard via ceph-users < >>>> ceph-users(a)ceph.io> wrote: >>>> >>>>> On Fri, Feb 6, 2026 at 7:59 AM Yuri Weinstein via ceph-users < >>>>> ceph-users(a)ceph.io> wrote: >>>>> >>>>> > We are continuing on this release. >>>>> > >>>>> > >>>>> > powercycle - FAILED Brad, approved? >>>>> > >>>>> >>>>> Unfortunately, I don't see any logs anywhere where I have access to >>>>> them so >>>>> I have no way of checking this? >>>>> >>>> >>>> OK, I have the latest logs. The three failing jobs appear to have all >>>> timed out after running for about six hours. At this time I don't know why >>>> but I continue to investigate. >>>> >>> >>> One point of commonality of the three tests that failed is that they are >>> all running tests using ceph fuse. I'll reach out to the FS guys to see if >>> they can help me get to the bottom of these failures. >>> >>> >>>> >>>> >>>>> >>>>> >>>>> > ceph-volume - Approved Guillaume? >>>>> > >>>>> > Patrick was reviewing some tests, please see his comments in the same >>>>> > tracker. >>>>> > TIA >>>>> > >>>>> > >>>>> > On Sat, Dec 13, 2025 at 1:19 AM Ilya Dryomov via ceph-users < >>>>> > ceph-users(a)ceph.io> wrote: >>>>> > >>>>> > > On Mon, Dec 8, 2025 at 11:31 PM Yuri Weinstein via Dev < >>>>> dev(a)ceph.io> >>>>> > > wrote: >>>>> > > > >>>>> > > > This test branch was built loner then 2.5 weeks ago >>>>> > > > I have to rebuild it first and then will schedule krbd >>>>> > > >>>>> > > krbd approved based on >>>>> > > >>>>> > > >>>>> > > >>>>> > >>>>> https://pulpito.ceph.com/yuriw-2025-12-10_21:26:56-krbd-reef-release-testin… >>>>> > > >>>>> > > >>>>> > >>>>> https://pulpito.ceph.com/yuriw-2025-12-12_18:20:32-krbd-reef-release-testin… >>>>> > > >>>>> > > Thanks, >>>>> > > >>>>> > > Ilya >>>>> > > >>>>> > > > >>>>> > > > On Mon, Dec 8, 2025 at 1:39 PM Ilya Dryomov via ceph-users < >>>>> > > ceph-users(a)ceph.io> wrote: >>>>> > > >> >>>>> > > >> On Mon, Dec 8, 2025 at 8:50 PM Ilya Dryomov <idryomov(a)gmail.com >>>>> > >>>>> > wrote: >>>>> > > >> > >>>>> > > >> > On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users >>>>> > > >> > <ceph-users(a)ceph.io> wrote: >>>>> > > >> > > >>>>> > > >> > > Hi Yuri, >>>>> > > >> > > >>>>> > > >> > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein < >>>>> > yweinste(a)redhat.com> >>>>> > > wrote: >>>>> > > >> > > > >>>>> > > >> > > > Note: this is an attempt to test and publish post EOL >>>>> reef >>>>> > point >>>>> > > release >>>>> > > >> > > > >>>>> > > >> > > > Details of this release are summarized here: >>>>> > > >> > > > >>>>> > > >> > > > https://tracker.ceph.com/issues/73906#note-1 >>>>> > > >> > > > Release Notes - TBD >>>>> > > >> > > > LRC upgrade - N/A >>>>> > > >> > > > >>>>> > > >> > > > distro => jammy focal centos9 windows bookworm >>>>> > > >> > > > >>>>> > > >> > > > Seeking approvals/reviews for: >>>>> > > >> > > > >>>>> > > >> > > > smoke - Laura approved? >>>>> > > >> > > > >>>>> > > >> > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam >>>>> King >>>>> > > approved? >>>>> > > >> > > > (rados is also running on build 2 == build 1 + >>>>> > > >> > > > https://github.com/ceph/ceph/pull/66056 per Igor's >>>>> request) >>>>> > > >> > > > >>>>> > > >> > > > rgw - Adam E approved? >>>>> > > >> > > > >>>>> > > >> > > > fs - Venky approved? >>>>> > > >> > > >>>>> > > >> > > There are lot many failures in the fs suite resembling >>>>> > > >> > > >>>>> > > >> > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: >>>>> wrong >>>>> > fs >>>>> > > >> > > type, bad option ... >>>>> > > >> > > >>>>> > > >> > > which is an issue with the testing kernel. Adding Alex who >>>>> was >>>>> > > looking >>>>> > > >> > > into this issue. >>>>> > > >> > > >>>>> > > >> > > As far as the run validation is concerned, it's risky to >>>>> ignore >>>>> > the >>>>> > > >> > > large number of failures. We have two options: >>>>> > > >> > > >>>>> > > >> > > - fix the testing kernel issue and rerun >>>>> > > >> > > - rerun with `for-linus` branch (with `-k for-linus` in the >>>>> > > >> > > teuthology-suite schedule command) >>>>> > > >> > > >>>>> > > >> > > The latter would be using an older kernel branch, but I >>>>> think it >>>>> > > >> > > should suffice for this case. >>>>> > > >> > >>>>> > > >> > Hi Venky, >>>>> > > >> > >>>>> > > >> > With your report of the fs suite being affected as well, I >>>>> carved >>>>> > out >>>>> > > >> > some time to take a look. >>>>> > > >> > >>>>> > > >> > I think it's happening because XFS maintainers disabled >>>>> support for >>>>> > V4 >>>>> > > >> > filesystems that was deprecated 5 years ago -- as of 6.18 >>>>> kernel >>>>> > it's >>>>> > > >> > no longer compiled in by default. The proper fix would be to >>>>> create >>>>> > > an >>>>> > > >> > image with/move to V5 on-disk format; I have opened [1] to >>>>> try to >>>>> > > >> > achieve that. We can also re-enable V4 on-disk format in our >>>>> kernel >>>>> > > >> > builds as a workaround. >>>>> > > >> > >>>>> > > >> > [1] https://github.com/ceph/ceph-cm-ansible/pull/799 >>>>> > > >> >>>>> > > >> This should be fixed now. Yuri, please schedule a rerun for >>>>> krbd. >>>>> > > >> >>>>> > > >> Thanks, >>>>> > > >> >>>>> > > >> Ilya >>>>> > > >> _______________________________________________ >>>>> > > >> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> > > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> > > > >>>>> > > > _______________________________________________ >>>>> > > > Dev mailing list -- dev(a)ceph.io >>>>> > > > To unsubscribe send an email to dev-leave(a)ceph.io >>>>> > > _______________________________________________ >>>>> > > ceph-users mailing list -- ceph-users(a)ceph.io >>>>> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> > > >>>>> > _______________________________________________ >>>>> > ceph-users mailing list -- ceph-users(a)ceph.io >>>>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> > >>>>> >>>>> >>>>> -- >>>>> Cheers, >>>>> Brad >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> >>>> >>>> >>>> -- >>>> Cheers, >>>> Brad >>>> >>> >>> >>> -- >>> Cheers, >>> Brad >>> _______________________________________________ >>> Dev mailing list -- dev(a)ceph.io >>> To unsubscribe send an email to dev-leave(a)ceph.io >>> >> >> >> -- >> Cheers, >> Brad >> _______________________________________________ >> Dev mailing list -- dev(a)ceph.io >> To unsubscribe send an email to dev-leave(a)ceph.io >> > > > -- > > Laura Flores > > She/Her/Hers > > Software Engineer, Ceph Storage <https://ceph.io> > > Chicago, IL > > lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> > M: +17087388804 > > > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io >

1 0

Re: How to disable an alertmanager alert
by Ankush Behl 12 Feb '26

12 Feb '26

Thanks for info, it would need a fix and we need to filter the suppressed alerts. On Thu, Feb 5, 2026 at 10:11 PM Chris Palmer via ceph-users < ceph-users(a)ceph.io> wrote: > Hi > > As a result of applying a Silence, they correctly appear on the > Observability\Alerts screen as "suppressed". > > > However they still appear on the home page Dashboard, in the top-centre > Status panel: > > Thanks, Chris > > > On 05/02/2026 16:10, Ankush Behl wrote: > > Hi Chris, do you mean they show up in overview dashboard? Can you show > > via some screenshots where all they are coming up? > > > > > > On Thu, 5 Feb 2026 at 6:39 PM, Chris Palmer via ceph-users > > <ceph-users(a)ceph.io> wrote: > > > > cephadm (squid & tentacle) deploys alertmanager with some default > > alerts. One is CephNodeInconsistentMTU which checks if the MTU of > > each > > interface with the same name is the same across the cluster. > > > > However if your cluster nodes are not identical, and have differing > > interface names, we end up with lots of permanent alerts. Although > > I can > > silence the alert (matching the alert name) for a long time, they > > still > > show up in the dashboard. > > > > The alerts are different permutations of: > > > > Node ceph3 has a different MTU size (1500) than the median of > > devices named enp4s0. > > > > Since that check is inappropriate in a heterogeneous cluster, > > what's the > > recommended way of disabling it, which won't get overwritten with > > redeployment of the monitoring stack? > > > > Thanks, Chris > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

[rgw multiple zonegroups] ERROR: S3 error: 404 (NoSuchKey) while copy
by Alexander NA 12 Feb '26

12 Feb '26

Hello everyone. I'm trying to set up one realm ("r1"), two zone groups ("zg1" - master, "zg2"), and one zone in each zone group ("z1" - master and "z2", respectively). The first zone group and its zone are masters. Endpoint "zg1" - 10.1.1.1 Endpoint "zg2" - 10.1.1.2 "zg1" has bucket "buc1" "zg2" has bucket "buc2" If I copy the file like this: s3cmd put test_file s3://buc1 --region=zg2 --host=10.1.1.2 I get the correct error code (as per the documentation): "ERROR: S3 error: 301 (Moved Permanently)" BUT, if I copy it like this: s3cmd put test_file s3://buc2 --region=zg1 --host=10.1.1.1 I get this code: ERROR: S3 error: 404 (NoSuchKey) Can you point me to something that might be wrong? Thanks

1 0

Re: Debian Packages for Trixie
by Albert Shih 12 Feb '26

12 Feb '26

Le 09/02/2026 à 16:16:30+0000, Stephane DUGRAVOT via ceph-users a écrit Hi, > ----- Le 13 Nov 25, à 17:32, Albert Shih <Albert.Shih(a)obspm.fr> a écrit : > > > Le 28/08/2025 à 18:52:05+1000, Andrew a écrit > > Hi, > > > > > I can see that there are official packages for Debian Old-Stable (Bookworm) > > > available on the official site: > > > https://download.ceph.com/debian-19.2.3/dists/ > > > > Does anyone know if there is an approximate (or likely) ETA for the Debian > > > Packages to be released for current stable version (Trixie)? > > > > Juste notice it's still not there.... > > > It would be super nice if the pkg for Debian 13 are available. > Just to notice, we installed 18.2.7 on Debian 13, then after upgrade with cephadm : > > ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.0 > > and 19.2.3, etc ... until 20.1.1. On a test plaform. > It might not be that easy to add nodes :-/ ;-) ;-) In fact the point I'm not very sure is what difference between the package ceph-common from Debian and the one from ceph.com. Because the ceph «himself» (at least with cephadm) are in container so should not have big link with the version of the OS. In other term what append if we install the ceph-common from Debian 13 (18.2.7) with cephadm 20.x.y (or whatever which version) in container. Regards. -- Albert SHIH 🦫 🐸 France Heure locale/Local time: jeu. 12 févr. 2026 14:26:02 CET

1 0

Re: Data is still actively corrupted!
by dominik.baack 12 Feb '26

12 Feb '26

Hi, thanks for your reply. Mounting was done without the 'root_squash' option, here is the corresponding fstab entry: 192.168.251.2,192.168.251.3,192.168.251.4,192.168.251.5,192.168.251.6,192.168.251.7:/ /cephfs ceph name=gpu01,secretfile=/etc/ceph/gpu01.key,noatime,_netdev,recover_session=clean,read_from_replica=balance 0 0 Dominik Am 2026-02-12 09:51, schrieb goetze: > Hi ! > > Have you mounted cephfs with the 'root_squash' option set? If so, > remove that option. I may be wrong here, but as far as I know, this is > still considered unsafe and can lead to data corruption since the > necessary code changes have not yet made it into the mainstream linux > kernel. > > Carsten > ------------------------------------------------------------------ > Carsten Goetze > Computer Graphics tel: +49 531 391-2109 > TU Braunschweig fax: +49 531 391-2103 > Muehlenpfordtstr. 23 eMail: c.goetze(a)cg.cs.tu-bs.de > D-38106 Braunschweig http://www.cg.cs.tu-bs.de/people/goetze > >> Am 12.02.2026 um 07:15 schrieb Dominik Baack via ceph-users >> <ceph-users(a)ceph.io>: >> >> Hi, >> >> I noticed now that files are still actively corrupted / replaced by >> empty files when open saved. >> >> Access was done via Ceph 19.2.1 from Ubuntu 24 with kernel mount. >> Ceph servers are still running 20.2 Tentacle (deployed via cephadm) >> with an Ubuntu 24.04 host. >> >> Options currently set are to noout, norebalance, noscrub and >> nodeep-scrub >> >> I am currently setting up a read only mount to copy all existing >> data for a backup. >> >> I have currently no clue whats going on, I currently was able to >> observe this behavior only on the nodes. As those were upgraded as >> well (MLNX driver, nvidia-fs,...) can it be network? >> >> Any Idea how to recover from this? >> >> Cheers >> Dominik >> >> Am 11.02.2026 um 18:44 schrieb dominik.baack via ceph-users: >> >>> Hi, >>> >>> after an controlled shutdown of the whole cluster do to external >>> circumstances we decided to update from 19.2 to 20.2 after the >>> restart. The system was health before and after the update. >>> The nodes mounting the filesystem were not equally lucky and were >>> partially shutdown hard. Storage was kept running additional >>> ~30min after node shutdown, all inflight operations should have >>> finished. >>> >>> Now we discover the some of the user files seem to be replaced >>> with zeros. For example: >>> >>> stat .gitignore >>> File: .gitignore >>> Size: 4429 Blocks: 9 IO Block: 4194304 >>> regular file >>> Device: 0,48 Inode: 1100241384598 Links: 1 >>> >>> hexdump -C .gitignore >>> 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >>> |................| >>> * >>> 00001140 00 00 00 00 00 00 00 00 00 00 00 00 00 |.............| >>> 0000114d >>> >>> Scanning for files containing only zeros show several issues of >>> files that were likely accessed before or during the shutdown of >>> the nodes. >>> >>> How should I progress from here? >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: tentacle and libvirt
by Eugen Block 12 Feb '26

12 Feb '26

I upgraded my Ceph test cluster to Tentacle last night and OpenStack (Ubuntu 24) still works as before. So there must be something else going on with your cluster, I assume. Am Di., 10. Feb. 2026 um 00:06 Uhr schrieb Piotr Pisz via ceph-users < ceph-users(a)ceph.io>: > Hello, > > > > We upgraded Ceph to Tentacle, and everything went fine, but all hosts > running Ubuntu 22 and 24 lost connectivity to RBD at the libvirt level. > > What has changed in the tentacle? > > > > Best regards, > > Piotr > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Data is still actively corrupted!
by Carsten Götze 12 Feb '26

12 Feb '26

Hi ! Have you mounted cephfs with the 'root_squash' option set? If so, remove that option. I may be wrong here, but as far as I know, this is still considered unsafe and can lead to data corruption since the necessary code changes have not yet made it into the mainstream linux kernel. Carsten ------------------------------------------------------------------ Carsten Goetze Computer Graphics tel: +49 531 391-2109 TU Braunschweig fax: +49 531 391-2103 Muehlenpfordtstr. 23 eMail: c.goetze(a)cg.cs.tu-bs.de D-38106 Braunschweig http://www.cg.cs.tu-bs.de/people/goetze > Am 12.02.2026 um 07:15 schrieb Dominik Baack via ceph-users <ceph-users(a)ceph.io>: > > Hi, > > I noticed now that files are still actively corrupted / replaced by empty files when open saved. > > Access was done via Ceph 19.2.1 from Ubuntu 24 with kernel mount. > Ceph servers are still running 20.2 Tentacle (deployed via cephadm) with an Ubuntu 24.04 host. > > Options currently set are to noout, norebalance, noscrub and nodeep-scrub > > I am currently setting up a read only mount to copy all existing data for a backup. > > > I have currently no clue whats going on, I currently was able to observe this behavior only on the nodes. As those were upgraded as well (MLNX driver, nvidia-fs,...) can it be network? > > Any Idea how to recover from this? > > Cheers > Dominik > > > Am 11.02.2026 um 18:44 schrieb dominik.baack via ceph-users: >> Hi, >> >> after an controlled shutdown of the whole cluster do to external circumstances we decided to update from 19.2 to 20.2 after the restart. The system was health before and after the update. >> The nodes mounting the filesystem were not equally lucky and were partially shutdown hard. Storage was kept running additional ~30min after node shutdown, all inflight operations should have finished. >> >> Now we discover the some of the user files seem to be replaced with zeros. For example: >> >> stat .gitignore >> File: .gitignore >> Size: 4429 Blocks: 9 IO Block: 4194304 regular file >> Device: 0,48 Inode: 1100241384598 Links: 1 >> >> >> hexdump -C .gitignore >> 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| >> * >> 00001140 00 00 00 00 00 00 00 00 00 00 00 00 00 |.............| >> 0000114d >> >> >> Scanning for files containing only zeros show several issues of files that were likely accessed before or during the shutdown of the nodes. >> >> How should I progress from here? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: tentacle and libvirt
by Alfonso Escribano Merino 12 Feb '26

12 Feb '26

1 0

Re: Data is still actively corrupted!
by Dominik Baack 12 Feb '26

12 Feb '26

Hi, I noticed now that files are still actively corrupted / replaced by empty files when open saved. Access was done via Ceph 19.2.1 from Ubuntu 24 with kernel mount. Ceph servers are still running 20.2 Tentacle (deployed via cephadm) with an Ubuntu 24.04 host. Options currently set are to noout, norebalance, noscrub and nodeep-scrub I am currently setting up a read only mount to copy all existing data for a backup. I have currently no clue whats going on, I currently was able to observe this behavior only on the nodes. As those were upgraded as well (MLNX driver, nvidia-fs,...) can it be network? Any Idea how to recover from this? Cheers Dominik Am 11.02.2026 um 18:44 schrieb dominik.baack via ceph-users: > Hi, > > after an controlled shutdown of the whole cluster do to external > circumstances we decided to update from 19.2 to 20.2 after the > restart. The system was health before and after the update. > The nodes mounting the filesystem were not equally lucky and were > partially shutdown hard. Storage was kept running additional ~30min > after node shutdown, all inflight operations should have finished. > > Now we discover the some of the user files seem to be replaced with > zeros. For example: > > stat .gitignore > File: .gitignore > Size: 4429 Blocks: 9 IO Block: 4194304 regular file > Device: 0,48 Inode: 1100241384598 Links: 1 > > > hexdump -C .gitignore > 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > |................| > * > 00001140 00 00 00 00 00 00 00 00 00 00 00 00 00 |.............| > 0000114d > > > Scanning for files containing only zeros show several issues of files > that were likely accessed before or during the shutdown of the nodes. > > How should I progress from here?

1 0

Excessive OSD memory bloat during Sequential Read (Tentacle v20.2.0, EC 6+2)
by Chanu Yu 12 Feb '26

12 Feb '26

Hi all, I am experiencing a critical memory issue with OSD daemons during sequential read tests. I would appreciate your advice on whether this is a known issue or if there is a flaw in my methodology. Environment Ceph Version: Tentacle v20.2.0Deployment: cephadmOSD Nodes: 8 nodes (128GB RAM each)OSD Configuration: 10 OSDs per node (Total 80 OSDs)Pool Type: Erasure Coding 6+2 (allow_ec_optimizations: true, allow_ec_overwrites: true)Workload: fio Sequential Read (12 clients, 40 jobs/client, BS=4M, QD=128, direct=1)Key Observations Random OSD Spikes: In every test run, a small number of OSD daemons exhibit a massive memory spike. However, these few daemons consume enough memory to exhaust the entire host’s RAM and swap space.Lack of Memory Reclamation: The memory is never reclaimed automatically after the workload stops. It remains occupied until I manually intervene (e.g., by restarting daemons or clearing heap free list of tcmalloc).Production Concern: My primary concern is that if this behavior occurs in a live production environment, it will inevitably lead to severe system-wide thrashing and service instability.Configuration Efforts: I have set osd_memory_target to 10G (with autotune off) and bluestore_cache_size to 7G, but the OSDs continue to ignore these targets during the read workload.Attempted Workarounds TCMALLOC_RELEASE_RATE=1000: No effect.TCMALLOC_HEAP_LIMIT_MB / Container Memory Limits: Problematic OSDs are repeatedly killed by OOM.TCMALLOC_AGGRESSIVE_DECOMMIT=true: Keeps memory stable but causes a 20% performance drop.Bluestore Cache Autotune: Controls memory but results in a 40% performance drop.Questions Is this behavior normal, or is there a flaw in my experimental methodology? Given that the memory usage seems abnormal, I am wondering if my fio configuration or OSD setup is fundamentally misconfigured for this version.Why does this spike occur on random OSDs during reads? Could this be related to how specific OSDs handle EC primary shards or is it a known TCMalloc fragmentation issue in v20.2.0?Is there a way to strictly enforce memory limits without such significant performance degradation?Any insights or suggestions for further debugging would be greatly appreciated. Best regards, Chanu Yu

1 0

Re: reef 18.2.8 QE validation status
by Laura Flores 11 Feb '26

11 Feb '26

The rados suite is approved: https://tracker.ceph.com/projects/rados/wiki/REEF#reef-v1828-httpstrackerce… On Tue, Feb 10, 2026 at 6:33 PM Brad Hubbard via Dev <dev(a)ceph.io> wrote: > Powercycle approved. > > On Tue, Feb 10, 2026 at 10:35 AM Brad Hubbard via Dev <dev(a)ceph.io> wrote: > >> >> >> On Fri, Feb 6, 2026 at 12:36 PM Brad Hubbard <bhubbard(a)redhat.com> wrote: >> >>> >>> >>> On Fri, Feb 6, 2026 at 8:33 AM Brad Hubbard via ceph-users < >>> ceph-users(a)ceph.io> wrote: >>> >>>> On Fri, Feb 6, 2026 at 7:59 AM Yuri Weinstein via ceph-users < >>>> ceph-users(a)ceph.io> wrote: >>>> >>>> > We are continuing on this release. >>>> > >>>> > >>>> > powercycle - FAILED Brad, approved? >>>> > >>>> >>>> Unfortunately, I don't see any logs anywhere where I have access to >>>> them so >>>> I have no way of checking this? >>>> >>> >>> OK, I have the latest logs. The three failing jobs appear to have all >>> timed out after running for about six hours. At this time I don't know why >>> but I continue to investigate. >>> >> >> One point of commonality of the three tests that failed is that they are >> all running tests using ceph fuse. I'll reach out to the FS guys to see if >> they can help me get to the bottom of these failures. >> >> >>> >>> >>>> >>>> >>>> > ceph-volume - Approved Guillaume? >>>> > >>>> > Patrick was reviewing some tests, please see his comments in the same >>>> > tracker. >>>> > TIA >>>> > >>>> > >>>> > On Sat, Dec 13, 2025 at 1:19 AM Ilya Dryomov via ceph-users < >>>> > ceph-users(a)ceph.io> wrote: >>>> > >>>> > > On Mon, Dec 8, 2025 at 11:31 PM Yuri Weinstein via Dev <dev(a)ceph.io >>>> > >>>> > > wrote: >>>> > > > >>>> > > > This test branch was built loner then 2.5 weeks ago >>>> > > > I have to rebuild it first and then will schedule krbd >>>> > > >>>> > > krbd approved based on >>>> > > >>>> > > >>>> > > >>>> > >>>> https://pulpito.ceph.com/yuriw-2025-12-10_21:26:56-krbd-reef-release-testin… >>>> > > >>>> > > >>>> > >>>> https://pulpito.ceph.com/yuriw-2025-12-12_18:20:32-krbd-reef-release-testin… >>>> > > >>>> > > Thanks, >>>> > > >>>> > > Ilya >>>> > > >>>> > > > >>>> > > > On Mon, Dec 8, 2025 at 1:39 PM Ilya Dryomov via ceph-users < >>>> > > ceph-users(a)ceph.io> wrote: >>>> > > >> >>>> > > >> On Mon, Dec 8, 2025 at 8:50 PM Ilya Dryomov <idryomov(a)gmail.com> >>>> > wrote: >>>> > > >> > >>>> > > >> > On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users >>>> > > >> > <ceph-users(a)ceph.io> wrote: >>>> > > >> > > >>>> > > >> > > Hi Yuri, >>>> > > >> > > >>>> > > >> > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein < >>>> > yweinste(a)redhat.com> >>>> > > wrote: >>>> > > >> > > > >>>> > > >> > > > Note: this is an attempt to test and publish post EOL reef >>>> > point >>>> > > release >>>> > > >> > > > >>>> > > >> > > > Details of this release are summarized here: >>>> > > >> > > > >>>> > > >> > > > https://tracker.ceph.com/issues/73906#note-1 >>>> > > >> > > > Release Notes - TBD >>>> > > >> > > > LRC upgrade - N/A >>>> > > >> > > > >>>> > > >> > > > distro => jammy focal centos9 windows bookworm >>>> > > >> > > > >>>> > > >> > > > Seeking approvals/reviews for: >>>> > > >> > > > >>>> > > >> > > > smoke - Laura approved? >>>> > > >> > > > >>>> > > >> > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam >>>> King >>>> > > approved? >>>> > > >> > > > (rados is also running on build 2 == build 1 + >>>> > > >> > > > https://github.com/ceph/ceph/pull/66056 per Igor's >>>> request) >>>> > > >> > > > >>>> > > >> > > > rgw - Adam E approved? >>>> > > >> > > > >>>> > > >> > > > fs - Venky approved? >>>> > > >> > > >>>> > > >> > > There are lot many failures in the fs suite resembling >>>> > > >> > > >>>> > > >> > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: >>>> wrong >>>> > fs >>>> > > >> > > type, bad option ... >>>> > > >> > > >>>> > > >> > > which is an issue with the testing kernel. Adding Alex who >>>> was >>>> > > looking >>>> > > >> > > into this issue. >>>> > > >> > > >>>> > > >> > > As far as the run validation is concerned, it's risky to >>>> ignore >>>> > the >>>> > > >> > > large number of failures. We have two options: >>>> > > >> > > >>>> > > >> > > - fix the testing kernel issue and rerun >>>> > > >> > > - rerun with `for-linus` branch (with `-k for-linus` in the >>>> > > >> > > teuthology-suite schedule command) >>>> > > >> > > >>>> > > >> > > The latter would be using an older kernel branch, but I >>>> think it >>>> > > >> > > should suffice for this case. >>>> > > >> > >>>> > > >> > Hi Venky, >>>> > > >> > >>>> > > >> > With your report of the fs suite being affected as well, I >>>> carved >>>> > out >>>> > > >> > some time to take a look. >>>> > > >> > >>>> > > >> > I think it's happening because XFS maintainers disabled >>>> support for >>>> > V4 >>>> > > >> > filesystems that was deprecated 5 years ago -- as of 6.18 >>>> kernel >>>> > it's >>>> > > >> > no longer compiled in by default. The proper fix would be to >>>> create >>>> > > an >>>> > > >> > image with/move to V5 on-disk format; I have opened [1] to try >>>> to >>>> > > >> > achieve that. We can also re-enable V4 on-disk format in our >>>> kernel >>>> > > >> > builds as a workaround. >>>> > > >> > >>>> > > >> > [1] https://github.com/ceph/ceph-cm-ansible/pull/799 >>>> > > >> >>>> > > >> This should be fixed now. Yuri, please schedule a rerun for >>>> krbd. >>>> > > >> >>>> > > >> Thanks, >>>> > > >> >>>> > > >> Ilya >>>> > > >> _______________________________________________ >>>> > > >> ceph-users mailing list -- ceph-users(a)ceph.io >>>> > > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> > > > >>>> > > > _______________________________________________ >>>> > > > Dev mailing list -- dev(a)ceph.io >>>> > > > To unsubscribe send an email to dev-leave(a)ceph.io >>>> > > _______________________________________________ >>>> > > ceph-users mailing list -- ceph-users(a)ceph.io >>>> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> > > >>>> > _______________________________________________ >>>> > ceph-users mailing list -- ceph-users(a)ceph.io >>>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> > >>>> >>>> >>>> -- >>>> Cheers, >>>> Brad >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>> >>> >>> -- >>> Cheers, >>> Brad >>> >> >> >> -- >> Cheers, >> Brad >> _______________________________________________ >> Dev mailing list -- dev(a)ceph.io >> To unsubscribe send an email to dev-leave(a)ceph.io >> > > > -- > Cheers, > Brad > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io > -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

Re: Kernel NULL dereference in ceph_mds_auth_match (Linux 6.17, CephFS kernel client)
by Ilya Dryomov 11 Feb '26

11 Feb '26

On Wed, Feb 11, 2026 at 5:29 PM Fabien Sirjean via ceph-users <ceph-users(a)ceph.io> wrote: > > Hello, > > We are encountering a reproducible issue with the CephFS kernel client > on Ubuntu 24.04 and would appreciate your feedback. > > Context: > We are preparing an updated Ubuntu 24.04 image for our scientific users. > With kernel |6.17.0-14-generic|, the CephFS mount succeeds and the > filesystem is visible via |df -h|, but any attempt to access the > directory contents (e.g. |ls|) immediately triggers a kernel crash. > > From userland, the process is simply killed: > > |# ls /net/serdon/illdata Killed| > > The mount appears healthy and the filesystem is correctly reported: > > |# mount | grep ceph 192.168.170.61,192.168.170.69,192.168.170.70:/ on > /net/serdon/illdata type ceph (rw,relatime,name=visa,acl) # df -h | grep > serdon 192.168.170.61,192.168.170.69,192.168.170.70:/ 4.1P 2.0P 2.1P 49% > /net/serdon/illdata| > > However, any access to the mount point triggers a kernel NULL pointer > dereference in the Ceph module: > > |BUG: kernel NULL pointer dereference, address: 0000000000000000 RIP: > strcmp+0x10/0x40 Call Trace: ceph_mds_auth_match ceph_mds_check_access > ceph_open| > > Environment: > > |Linux 6.17.0-14-generic #14~24.04.1-Ubuntu CephFS mounted via kernel > client (mount -t ceph)| > > The issue is fully reproducible: any access to the mount point causes > the same kernel crash. > > Full stack trace: > https://pastebin.com/ZgQHpN0q <https://pastebin.com/ZgQHpN0q> > > Questions: > - Is this a known issue with the CephFS kernel client on Linux 6.17? Hi Fabien, Yes, it has just been fixed: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/f… > - Does this look like an upstream CephFS bug, or something specific to > the Ubuntu 6.17 kernel? It's an upstream bug; the fix should be trivial to backport to any Ubuntu kernel that is affected. Thanks, Ilya

1 0

Zeroed files after shutdown + update
by dominik.baack 11 Feb '26

11 Feb '26

Hi, after an controlled shutdown of the whole cluster do to external circumstances we decided to update from 19.2 to 20.2 after the restart. The system was health before and after the update. The nodes mounting the filesystem were not equally lucky and were partially shutdown hard. Storage was kept running additional ~30min after node shutdown, all inflight operations should have finished. Now we discover the some of the user files seem to be replaced with zeros. For example: stat .gitignore File: .gitignore Size: 4429 Blocks: 9 IO Block: 4194304 regular file Device: 0,48 Inode: 1100241384598 Links: 1 hexdump -C .gitignore 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00001140 00 00 00 00 00 00 00 00 00 00 00 00 00 |.............| 0000114d Scanning for files containing only zeros show several issues of files that were likely accessed before or during the shutdown of the nodes. How should I progress from here? Cheers Dominik

1 0

Kernel NULL dereference in ceph_mds_auth_match (Linux 6.17, CephFS kernel client)
by Fabien Sirjean 11 Feb '26

11 Feb '26

Hello, We are encountering a reproducible issue with the CephFS kernel client on Ubuntu 24.04 and would appreciate your feedback. Context: We are preparing an updated Ubuntu 24.04 image for our scientific users. With kernel |6.17.0-14-generic|, the CephFS mount succeeds and the filesystem is visible via |df -h|, but any attempt to access the directory contents (e.g. |ls|) immediately triggers a kernel crash. From userland, the process is simply killed: |# ls /net/serdon/illdata Killed| The mount appears healthy and the filesystem is correctly reported: |# mount | grep ceph 192.168.170.61,192.168.170.69,192.168.170.70:/ on /net/serdon/illdata type ceph (rw,relatime,name=visa,acl) # df -h | grep serdon 192.168.170.61,192.168.170.69,192.168.170.70:/ 4.1P 2.0P 2.1P 49% /net/serdon/illdata| However, any access to the mount point triggers a kernel NULL pointer dereference in the Ceph module: |BUG: kernel NULL pointer dereference, address: 0000000000000000 RIP: strcmp+0x10/0x40 Call Trace: ceph_mds_auth_match ceph_mds_check_access ceph_open| Environment: |Linux 6.17.0-14-generic #14~24.04.1-Ubuntu CephFS mounted via kernel client (mount -t ceph)| The issue is fully reproducible: any access to the mount point causes the same kernel crash. Full stack trace: https://pastebin.com/ZgQHpN0q <https://pastebin.com/ZgQHpN0q> Questions: - Is this a known issue with the CephFS kernel client on Linux 6.17? - Does this look like an upstream CephFS bug, or something specific to the Ubuntu 6.17 kernel? We can provide additional details (Ceph cluster version, MDS version, etc.) if needed. Thank you for your help. Best regards, Fabien

1 0

Re: tentacle and libvirt
by Joachim Kraftmayer 11 Feb '26

11 Feb '26

Hi, my first question is, what kind of ceph client version is used by libvirt? Regards, Joachim www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Eugen Block via ceph-users <ceph-users(a)ceph.io> schrieb am Mi., 11. Feb. 2026, 12:24: > Hi, > > your question is a bit "thin", I'd say. A lot has changed in Tentacle [0], > but nothing of the mentioned changes looks critical wrt libvirt from my > perspective. Do you have any logs or more detailed descriptions what > exactly fails? We haven't upgraded to v20 yet, at least not our prod > cluster, we also use RBD heavily (OpenStack on Ubuntu), so this is quite > relevant for us. > > Thanks, > Eugen > > [0] https://ceph.io/en/news/blog/2025/v20-2-0-tentacle-released/#rbd > > Am Di., 10. Feb. 2026 um 00:06 Uhr schrieb Piotr Pisz via ceph-users < > ceph-users(a)ceph.io>: > > > Hello, > > > > > > > > We upgraded Ceph to Tentacle, and everything went fine, but all hosts > > running Ubuntu 22 and 24 lost connectivity to RBD at the libvirt level. > > > > What has changed in the tentacle? > > > > > > > > Best regards, > > > > Piotr > > > > > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: tentacle and libvirt
by Eugen Block 11 Feb '26

11 Feb '26

Hi, your question is a bit "thin", I'd say. A lot has changed in Tentacle [0], but nothing of the mentioned changes looks critical wrt libvirt from my perspective. Do you have any logs or more detailed descriptions what exactly fails? We haven't upgraded to v20 yet, at least not our prod cluster, we also use RBD heavily (OpenStack on Ubuntu), so this is quite relevant for us. Thanks, Eugen [0] https://ceph.io/en/news/blog/2025/v20-2-0-tentacle-released/#rbd Am Di., 10. Feb. 2026 um 00:06 Uhr schrieb Piotr Pisz via ceph-users < ceph-users(a)ceph.io>: > Hello, > > > > We upgraded Ceph to Tentacle, and everything went fine, but all hosts > running Ubuntu 22 and 24 lost connectivity to RBD at the libvirt level. > > What has changed in the tentacle? > > > > Best regards, > > Piotr > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: reef 18.2.8 QE validation status
by Brad Hubbard 11 Feb '26

11 Feb '26

Powercycle approved. On Tue, Feb 10, 2026 at 10:35 AM Brad Hubbard via Dev <dev(a)ceph.io> wrote: > > > On Fri, Feb 6, 2026 at 12:36 PM Brad Hubbard <bhubbard(a)redhat.com> wrote: > >> >> >> On Fri, Feb 6, 2026 at 8:33 AM Brad Hubbard via ceph-users < >> ceph-users(a)ceph.io> wrote: >> >>> On Fri, Feb 6, 2026 at 7:59 AM Yuri Weinstein via ceph-users < >>> ceph-users(a)ceph.io> wrote: >>> >>> > We are continuing on this release. >>> > >>> > >>> > powercycle - FAILED Brad, approved? >>> > >>> >>> Unfortunately, I don't see any logs anywhere where I have access to them >>> so >>> I have no way of checking this? >>> >> >> OK, I have the latest logs. The three failing jobs appear to have all >> timed out after running for about six hours. At this time I don't know why >> but I continue to investigate. >> > > One point of commonality of the three tests that failed is that they are > all running tests using ceph fuse. I'll reach out to the FS guys to see if > they can help me get to the bottom of these failures. > > >> >> >>> >>> >>> > ceph-volume - Approved Guillaume? >>> > >>> > Patrick was reviewing some tests, please see his comments in the same >>> > tracker. >>> > TIA >>> > >>> > >>> > On Sat, Dec 13, 2025 at 1:19 AM Ilya Dryomov via ceph-users < >>> > ceph-users(a)ceph.io> wrote: >>> > >>> > > On Mon, Dec 8, 2025 at 11:31 PM Yuri Weinstein via Dev <dev(a)ceph.io> >>> > > wrote: >>> > > > >>> > > > This test branch was built loner then 2.5 weeks ago >>> > > > I have to rebuild it first and then will schedule krbd >>> > > >>> > > krbd approved based on >>> > > >>> > > >>> > > >>> > >>> https://pulpito.ceph.com/yuriw-2025-12-10_21:26:56-krbd-reef-release-testin… >>> > > >>> > > >>> > >>> https://pulpito.ceph.com/yuriw-2025-12-12_18:20:32-krbd-reef-release-testin… >>> > > >>> > > Thanks, >>> > > >>> > > Ilya >>> > > >>> > > > >>> > > > On Mon, Dec 8, 2025 at 1:39 PM Ilya Dryomov via ceph-users < >>> > > ceph-users(a)ceph.io> wrote: >>> > > >> >>> > > >> On Mon, Dec 8, 2025 at 8:50 PM Ilya Dryomov <idryomov(a)gmail.com> >>> > wrote: >>> > > >> > >>> > > >> > On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users >>> > > >> > <ceph-users(a)ceph.io> wrote: >>> > > >> > > >>> > > >> > > Hi Yuri, >>> > > >> > > >>> > > >> > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein < >>> > yweinste(a)redhat.com> >>> > > wrote: >>> > > >> > > > >>> > > >> > > > Note: this is an attempt to test and publish post EOL reef >>> > point >>> > > release >>> > > >> > > > >>> > > >> > > > Details of this release are summarized here: >>> > > >> > > > >>> > > >> > > > https://tracker.ceph.com/issues/73906#note-1 >>> > > >> > > > Release Notes - TBD >>> > > >> > > > LRC upgrade - N/A >>> > > >> > > > >>> > > >> > > > distro => jammy focal centos9 windows bookworm >>> > > >> > > > >>> > > >> > > > Seeking approvals/reviews for: >>> > > >> > > > >>> > > >> > > > smoke - Laura approved? >>> > > >> > > > >>> > > >> > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam >>> King >>> > > approved? >>> > > >> > > > (rados is also running on build 2 == build 1 + >>> > > >> > > > https://github.com/ceph/ceph/pull/66056 per Igor's request) >>> > > >> > > > >>> > > >> > > > rgw - Adam E approved? >>> > > >> > > > >>> > > >> > > > fs - Venky approved? >>> > > >> > > >>> > > >> > > There are lot many failures in the fs suite resembling >>> > > >> > > >>> > > >> > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: >>> wrong >>> > fs >>> > > >> > > type, bad option ... >>> > > >> > > >>> > > >> > > which is an issue with the testing kernel. Adding Alex who was >>> > > looking >>> > > >> > > into this issue. >>> > > >> > > >>> > > >> > > As far as the run validation is concerned, it's risky to >>> ignore >>> > the >>> > > >> > > large number of failures. We have two options: >>> > > >> > > >>> > > >> > > - fix the testing kernel issue and rerun >>> > > >> > > - rerun with `for-linus` branch (with `-k for-linus` in the >>> > > >> > > teuthology-suite schedule command) >>> > > >> > > >>> > > >> > > The latter would be using an older kernel branch, but I think >>> it >>> > > >> > > should suffice for this case. >>> > > >> > >>> > > >> > Hi Venky, >>> > > >> > >>> > > >> > With your report of the fs suite being affected as well, I >>> carved >>> > out >>> > > >> > some time to take a look. >>> > > >> > >>> > > >> > I think it's happening because XFS maintainers disabled support >>> for >>> > V4 >>> > > >> > filesystems that was deprecated 5 years ago -- as of 6.18 kernel >>> > it's >>> > > >> > no longer compiled in by default. The proper fix would be to >>> create >>> > > an >>> > > >> > image with/move to V5 on-disk format; I have opened [1] to try >>> to >>> > > >> > achieve that. We can also re-enable V4 on-disk format in our >>> kernel >>> > > >> > builds as a workaround. >>> > > >> > >>> > > >> > [1] https://github.com/ceph/ceph-cm-ansible/pull/799 >>> > > >> >>> > > >> This should be fixed now. Yuri, please schedule a rerun for krbd. >>> > > >> >>> > > >> Thanks, >>> > > >> >>> > > >> Ilya >>> > > >> _______________________________________________ >>> > > >> ceph-users mailing list -- ceph-users(a)ceph.io >>> > > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> > > > >>> > > > _______________________________________________ >>> > > > Dev mailing list -- dev(a)ceph.io >>> > > > To unsubscribe send an email to dev-leave(a)ceph.io >>> > > _______________________________________________ >>> > > ceph-users mailing list -- ceph-users(a)ceph.io >>> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> > > >>> > _______________________________________________ >>> > ceph-users mailing list -- ceph-users(a)ceph.io >>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> > >>> >>> >>> -- >>> Cheers, >>> Brad >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >> >> >> -- >> Cheers, >> Brad >> > > > -- > Cheers, > Brad > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io > -- Cheers, Brad

1 0

Announcing go-ceph v0.38.0
by Sven Anderson 10 Feb '26

10 Feb '26

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.38.0 The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Sven

1 0

Re: MGR Restful not accessible when doing mgr failover (squid v19.2.3)
by Andreas Eriksson 10 Feb '26

10 Feb '26

Hi, Thanks! This worked. On 2/10/26 12:38 PM, Eugen Block via ceph-users wrote: > Hi, > > did you configure the additional MGR(s) as well with server_addr, > server_port and certs [0]? > > I haven't used the restful module before, but it works for me on Squid > after creating a self-signed cert and a user/key. > > Then I used the existing crt (it's a test environment) and set the > config-key for the other mgr as well: > > # get cert and key > ceph config-key get mgr/restful/nautilus/crt > nautilus-restful.crt > ceph config-key get mgr/restful/nautilus/key > nautilus-restful.key > > # set cert and key > ceph config-key set mgr/restful/nautilus2/crt -i nautilus-restful.crt > ceph config-key set mgr/restful/nautilus2/key -i nautilus-restful.key > > # configure per host server_addr and server_port > ceph config set mgr mgr/restful/nautilus/server_addr 192.168.168.111 > ceph config set mgr mgr/restful/nautilus/server_port 8003 > > ceph config set mgr mgr/restful/nautilus2/server_addr 192.168.168.112 > ceph config set mgr mgr/restful/nautilus2/server_port 8003 > > # restart restful if necessary > ceph restful restart > > ceph mgr services | grep rest > "restful":"https://192.168.168.111:8003/" > > # after failover > ceph mgr services | grep rest > "restful":"https://192.168.168.112:8003/" > > Hope this helps! > Eugen > > [0]https://docs.ceph.com/en/squid/mgr/restful/#configuring-ip-and-port > > Am So., 8. Feb. 2026 um 19:43 Uhr schrieb Andreas Eriksson via ceph-users< ceph-users(a)ceph.io>: > >> Hi, >> >> I'm doing cluster monitoring using the restful api on the ceph-mgr on >> port 8003. >> The requests are working fine as long as I have my *mgr-0* as primary. >> Doing a `ceph mgr fail` so that the *mgr-1* becomes active, the mgr >> daemon doesn't listen on port 8003. >> >> Is there any workaround for this? >> _______________________________________________ >> ceph-users mailing list --ceph-users(a)ceph.io >> To unsubscribe send an email toceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list --ceph-users(a)ceph.io > To unsubscribe send an email toceph-users-leave(a)ceph.io

1 0

Re: Repeated crashes of mgr module in Tentacle
by Eugen Block 10 Feb '26

10 Feb '26

Hi, > even “ceph -s” was hanging that command is contacting the MONs, are you sure the cluster is healthy? If ' ceph -s' hangs, I suspect either a network or a MON (quorum) issue. The default ceph uid/gid has always been 167 since I've been working with Ceph, so more than 10 years now, and it's the same within containers. And yes, there is a difference between cephadm setup and package based installs. Did you ensure that you don't have any other ceph packages installed except cephadm and maybe ceph-common? There can be issues when both deb packages and containers are running for the same daemon. So if only the MGR is affected, I recommend to ensure that no ceph-mgr package is installed. It looks like the telemetry module is responsible for the crash, can you turn that off? In latest releases you can force to turn off a module: ceph mgr module force disable telemetry Can you check if that doesn't crash the MGR? Also some debug (debug_mgr) logs might be helpful for the devs. I don't find an existing tracker issue for this, I'd suggest to create a new one. Regards, Eugen Am So., 8. Feb. 2026 um 14:46 Uhr schrieb Daniel Brown via ceph-users < ceph-users(a)ceph.io>: > > > Greetings — > > Have been seeing repeated crashes on my mgr module. Seems to run for about > 45 to 50 seconds and then boom. Cephadm setup here. Did (try to) enable a > couple modules lately. iostat, stats diskprediction_local, but have toggled > them back off - unfortunately it hasn’t fixed the issue. > > I was seeing “no mgr” with ceph -s — to get around that I’ve tweaked the > “StartLimitInterval” setting in the ceph-[CLUSTER-UID]@.service file down > to 1m, and have 4x mgr’s setup so that I can get a couple commands run > before mgr crashes and another starts. The 30m default there seems… high, > imo - I was having intervals with no mgr which makes it tough to do much > with the cluster - even “ceph -s” was hanging. Everything else in the > cluster seems “normal” - still serving data. > > > One other note — which I think is generally unrelated — I did upgrade one > of my cluster nodes from “Plucky Puffin" (25.04) ubuntu, to “Questing > Quokka” (25.10) ubuntu. After the upgrade, cephadm managed containers > didn’t want to start. I tracked that down to having the ceph user userid in > /etc/passwd set at 64045, but the container seeming to want userid 167. > Most things under /var/lib/ceph/[CLUSTER UID]/ … appear to be owned by > user/group 167:167 — I assume this is a default inside the container. > Workaround here was to manually change the UID/GID for ceph in /etc/passwd > and /etc/group. I’m going to imagine this is some collision between cephadm > managed deployments, and how Ubuntu / apt installs cephadm. > > > > The aforementioned mgr Crashes look like: > > > { > "assert_condition": "cursor != root", > "assert_file": > "/ceph/rpmbuild/BUILD/ceph-20.2.0/src/mgr/PyFormatter.h", > "assert_func": "virtual void PyFormatter::close_section()", > "assert_line": 84, > "assert_msg": "/ceph/rpmbuild/BUILD/ceph-20.2.0/src/mgr/PyFormatter.h: > In function 'virtual void PyFormatter::close_section()' thread ffff34e55700 > time > 2026-02-08T13:10:07.526894+0000\n/ceph/rpmbuild/BUILD/ceph-20.2.0/src/mgr/PyFormatter.h: > 84: FAILED ceph_assert(cursor != root)\n", > "assert_thread_name": "telemetry", > "backtrace": [ > "__kernel_rt_sigreturn()", > "/lib64/libc.so.6(+0x82a78) [0xffff83603a78]", > "raise()", > "abort()", > "(ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x190) [0xffff84039874]", > "/usr/bin/ceph-mgr(+0xcf540) [0xaaaacdf2f540]", > > "(ActivePyModules::get_perf_schema_python(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&)+0xf6c) [0xaaaacdf46ec0]", > "/usr/bin/ceph-mgr(+0x105528) [0xaaaacdf65528]", > "/lib64/libpython3.9.so.1.0(+0xf7bfc) [0xffff84ae9bfc]", > "_PyEval_EvalFrameDefault()", > "/lib64/libpython3.9.so.1.0(+0xda3f0) [0xffff84acc3f0]", > "_PyEval_EvalFrameDefault()", > "/lib64/libpython3.9.so.1.0(+0xc47e8) [0xffff84ab67e8]", > "_PyFunction_Vectorcall()", > "_PyEval_EvalFrameDefault()", > "/lib64/libpython3.9.so.1.0(+0xc47e8) [0xffff84ab67e8]", > "_PyFunction_Vectorcall()", > "_PyEval_EvalFrameDefault()", > "/lib64/libpython3.9.so.1.0(+0xc47e8) [0xffff84ab67e8]", > "_PyFunction_Vectorcall()", > "_PyEval_EvalFrameDefault()", > "/lib64/libpython3.9.so.1.0(+0xc47e8) [0xffff84ab67e8]", > "_PyFunction_Vectorcall()", > "_PyEval_EvalFrameDefault()", > "/lib64/libpython3.9.so.1.0(+0xda3f0) [0xffff84acc3f0]", > "/lib64/libpython3.9.so.1.0(+0xea93c) [0xffff84adc93c]", > "/lib64/libpython3.9.so.1.0(+0xcf304) [0xffff84ac1304]", > "/lib64/libpython3.9.so.1.0(+0x197d78) [0xffff84b89d78]", > "_PyObject_CallMethod_SizeT()", > "(PyModuleRunner::serve()+0x6c) [0xaaaacdfdf6cc]", > "(PyModuleRunner::PyModuleRunnerThread::entry()+0x148) > [0xaaaacdfdff08]" > ], > "ceph_version": "20.2.0", > "crash_id": > "2026-02-08T13:10:07.528739Z_f18e6b74-438b-47db-9438-5a3861fdef2d", > "entity_name": "mgr.hc-945901a5cad1b6e3.mtijbv", > "os_id": "centos", > "os_name": "CentOS Stream", > "os_version": "9", > "os_version_id": "9", > "process_name": "ceph-mgr", > "stack_sig": > "319d76a0d71a4644f9d65f592f6e621cca918d9205fc759ab7acf6944bc77fdd", > "timestamp": "2026-02-08T13:10:07.528739Z", > "utsname_hostname": "hc-945901a5cad1b6e3", > "utsname_machine": "aarch64", > "utsname_release": "6.14.0-1010-raspi", > "utsname_sysname": "Linux", > "utsname_version": "#10-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 15 19:09:05 > UTC 2025" > } > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: MGR Restful not accessible when doing mgr failover (squid v19.2.3)
by Eugen Block 10 Feb '26

10 Feb '26

Hi, did you configure the additional MGR(s) as well with server_addr, server_port and certs [0]? I haven't used the restful module before, but it works for me on Squid after creating a self-signed cert and a user/key. Then I used the existing crt (it's a test environment) and set the config-key for the other mgr as well: # get cert and key ceph config-key get mgr/restful/nautilus/crt > nautilus-restful.crt ceph config-key get mgr/restful/nautilus/key > nautilus-restful.key # set cert and key ceph config-key set mgr/restful/nautilus2/crt -i nautilus-restful.crt ceph config-key set mgr/restful/nautilus2/key -i nautilus-restful.key # configure per host server_addr and server_port ceph config set mgr mgr/restful/nautilus/server_addr 192.168.168.111 ceph config set mgr mgr/restful/nautilus/server_port 8003 ceph config set mgr mgr/restful/nautilus2/server_addr 192.168.168.112 ceph config set mgr mgr/restful/nautilus2/server_port 8003 # restart restful if necessary ceph restful restart ceph mgr services | grep rest "restful": "https://192.168.168.111:8003/" # after failover ceph mgr services | grep rest "restful": "https://192.168.168.112:8003/" Hope this helps! Eugen [0] https://docs.ceph.com/en/squid/mgr/restful/#configuring-ip-and-port Am So., 8. Feb. 2026 um 19:43 Uhr schrieb Andreas Eriksson via ceph-users < ceph-users(a)ceph.io>: > Hi, > > I'm doing cluster monitoring using the restful api on the ceph-mgr on > port 8003. > The requests are working fine as long as I have my *mgr-0* as primary. > Doing a `ceph mgr fail` so that the *mgr-1* becomes active, the mgr > daemon doesn't listen on port 8003. > > Is there any workaround for this? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Missing changelog in Tentacle release notes?
by Eugen Block 10 Feb '26

10 Feb '26

Hi, I was wondering why the Tentacle release notes [0] don't have a changelog section like the previous releases do (Reef [1], Squid [2]). To prepare an upgrade I usually scan the changelog for anything interesting/important. Has that process changed from v20 in general and there won't be any detailed changelogs anymore or was this an oversight? The highlights are great to get an overview, but I kind of miss the details. I'd appreciate any insghts. Thanks! Eugen [0] https://docs.ceph.com/en/latest/releases/tentacle/ [1] https://docs.ceph.com/en/latest/releases/squid/#changelog [2] https://docs.ceph.com/en/latest/releases/reef/#changelog

1 0

RGW period map issues and zone name collision
by Greg O'Neill 10 Feb '26

10 Feb '26

Hey folks, I'm having an issue with name collision of rgw zones across different realms in three multi-site clusters. Clusters are setup with bi-directional multi-site peering in addition to a local realm in each cluster for container images and thanos metrics. All clusters are running Reef v18.2.2 The local realm's `default` zone from the primary cluster's multi-site realm `cloud` was added to the period map by mistake and it propogated to the replica clusters. The zone name collided with the existing `default` zone in the `default` realm. I see the default zone in both pcloud and default realms with the same UUID. When I run `radosgw-admin period get --rgw-realm default` on the replica clusters, it returns with no zones or zonegroups when it should have one each named `default`. In a test cluster, I tried renaming or deleting the `default` zone from the multi-site realm but it knocks out any zones named `default` regardless of realm. ### Primary cluster (west coast) goneill@us-west-2-ceph-mon1:~$ sudo radosgw-admin period get --rgw-realm default | jq -r '.period_map.zonegroups[].zones[] | [.id, .name, .endpoints[]] | @tsv' 37929bb2-2eb3-4827-952a-6b5ca05627a6 default http://us-west-2-ceph-mon1:8080 goneill@us-west-2-ceph-mon1:~$ sudo radosgw-admin period get --rgw-realm savagebucket | jq -r '.period_map.zonegroups[].zones[] | [.id, .name, .endpoints[]] | @tsv' 250a2a20-5e11-486b-bae4-e29b173d2556 thanos 107d29a0-b732-4bf1-a26e-1f64f820e839 us-east-1-prod http://us-east-1-ceph-rgw1:8080 1e27bf9c-3a2f-4845-85b6-33a24bbe1c04 us-west-2-corp http://us-west-2-ceph-rgw1:8080 331d3f1e-1b72-4c56-bb5a-d1d0fcf6d0b8 us-west-1-prod http://us-west-1-ceph-rgw1:8080 37929bb2-2eb3-4827-952a-6b5ca05627a6 default ### Replica cluster (west coast) goneill@us-west-1-ceph-mon1:~$ sudo radosgw-admin period get --rgw-realm default | jq -r '.period_map.zonegroups[].zones[] | [.id, .name, .endpoints[]] | @tsv' goneill@us-west-1-ceph-mon1:~$ sudo radosgw-admin period get --rgw-realm savagebucket | jq -r '.period_map.zonegroups[].zones[] | [.id, .name, .endpoints[]] | @tsv' 250a2a20-5e11-486b-bae4-e29b173d2556 thanos 107d29a0-b732-4bf1-a26e-1f64f820e839 us-east-1-prod http://us-east-1-ceph-rgw1:8080 1e27bf9c-3a2f-4845-85b6-33a24bbe1c04 us-west-2-corp http://us-west-2-ceph-rgw1:8080 331d3f1e-1b72-4c56-bb5a-d1d0fcf6d0b8 us-west-1-prod http://us-west-1-ceph-rgw1:8080 37929bb2-2eb3-4827-952a-6b5ca05627a6 default ### Replica cluster (east coast) goneill@us-east-1-ceph-mon1:~$ sudo radosgw-admin period get --rgw-realm default | jq -r '.period_map.zonegroups[].zones[] | [.id, .name, .endpoints[]] | @tsv' goneill@us-east-1-ceph-mon1:~$ sudo radosgw-admin period get --rgw-realm savagebucket | jq -r '.period_map.zonegroups[].zones[] | [.id, .name, .endpoints[]] | @tsv' 250a2a20-5e11-486b-bae4-e29b173d2556 thanos 107d29a0-b732-4bf1-a26e-1f64f820e839 us-east-1-prod http://us-east-1-ceph-rgw1:8080 1e27bf9c-3a2f-4845-85b6-33a24bbe1c04 us-west-2-corp http://us-west-2-ceph-rgw1:8080 331d3f1e-1b72-4c56-bb5a-d1d0fcf6d0b8 us-west-1-prod http://us-west-1-ceph-rgw1:8080 37929bb2-2eb3-4827-952a-6b5ca05627a6 default

1 1