October 2025 - ceph-users

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

CephFS: no MDS does join the filesystem
by Robert Sander 24 Mar '26

24 Mar '26

Hi, we are having an issue at a customer site where a 3PB CephFS is in failed state. The cluster itself is unhealthy and awaits replacements disks: # ceph -s cluster: id: 28ca2bfa-d87e-11ed-83a3-1070fddda30f health: HEALTH_ERR 4 failed cephadm daemon(s) There are daemons running an older version of ceph 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged 8 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 46 pgs backfill_toofull Possible data damage: 4 pgs inconsistent Degraded data redundancy: 6427646/15858772167 objects degraded (0.041%), 8 pgs degraded, 8 pgs undersized 6 pool(s) nearfull (muted: OSDMAP_FLAGS OSD_SCRUB_ERRORS(2d) PG_NOT_DEEP_SCRUBBED PG_NOT_SCRUBBED) services: mon: 3 daemons, quorum sn01,sn03,sn02 (age 3w) mgr: sn03.crlpzh(active, since 33h), standbys: sn01.tegfya, sn02.mzvgcr mds: 18/19 daemons up, 1 standby osd: 181 osds: 174 up (since 4d), 172 in (since 4d); 206 remapped pgs flags nodeep-scrub data: volumes: 2/3 healthy, 1 recovering; 1 damaged pools: 12 pools, 3585 pgs objects: 1.93G objects, 1.3 PiB usage: 2.5 PiB used, 501 TiB / 3.0 PiB avail pgs: 6427646/15858772167 objects degraded (0.041%) 293845758/15858772167 objects misplaced (1.853%) 2844 active+clean 532 active+clean+scrubbing 147 active+remapped+backfill_wait 28 active+remapped+backfill_toofull 11 active+remapped+backfill_wait+backfill_toofull 10 active+remapped+backfilling 6 active+undersized+degraded+remapped+backfill_toofull 2 active+clean+inconsistent 1 active+clean+scrubbing+deep+inconsistent+repair 1 active+undersized+remapped+backfilling 1 active+undersized+degraded+remapped+backfilling 1 active+recovering+degraded+remapped 1 active+remapped+inconsistent+backfill_toofull io: recovery: 183 MiB/s, 312 objects/s The CephFS metadata pool is not affected by the inconsistent PGs. The MDSs have this line in their logfile: "Monitors have assigned me to become a standby." The filesystem is joinable: # ceph fs lsflags storage_cluster joinable allow_snaps allow_multimds_snaps refuse_client_session But no MDS joins: # ceph fs status storage_cluster - 0 clients =============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed POOL TYPE USED AVAIL cephfs_metadata metadata 490G 12.9T cephfs_data data 970T 54.1T shared_data data 1351T 22.5T STANDBY MDS storage_cluster.sn04.cbvzzu MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) Why? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

6 13

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

Debian Packages for Trixie
by Andrew 09 Feb '26

09 Feb '26

Hello Team, I can see that there are official packages for Debian Old-Stable (Bookworm) available on the official site: https://download.ceph.com/debian-19.2.3/dists/ Does anyone know if there is an approximate (or likely) ETA for the Debian Packages to be released for current stable version (Trixie)? Many thanks, Andrew

6 6

Untrimmed purged_snaps impacts osd startup, possibly oom killer
by Eugen Block 07 Jan '26

07 Jan '26

Hi, this is kind of a follow-up on a two years old thread [0] and I wanted to raise some awareness for the corresponding tracker [1]. Back then we managed to limit the impact on the mon store performance with some paxos configs, but now the OSDs are impacted as well when there are new ones created: Each newly created OSD process grows in RAM usage to around 140 GB within a few minutes, easily causing oom killers on hosts if multiple OSDs are created at once. The residual RAM usage drops to the memory target after it successfully booted. The reason are the purged_snaps that are loaded during OSD boot (snap_mapper.record_purged_snaps purged_snaps), two years ago the customer had more than 42 million purged_snap entries. I don't know how many there are today, I don't have access myself, but I'll try to get a current number. Anyway, the only way to safely create OSDs is one by one, maybe two at once depending on the host's RAM capacity. An automated (unattended) OSD deployment is currently not possible. Unfortunately, the new tool [2] doesn't seem to work as expected, at least in my test cluster it didn't have any impact on the number of purged_snaps in the mon store. That's why we haven't tried it on the customer cluster(s) yet. How do other operators/admins/users deal with this kind of scenario? Having many snapshots can't be a corner case, but I can't remember having read anything like this on the list(s). I'd appreciate any comments, although I'm aware that everybody is probably busy travelling to Cephalocon. ;-) Thanks! Eugen [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/ZEMGKBLMERE… [1] https://tracker.ceph.com/issues/64519 [2] https://github.com/ceph/ceph/pull/57548

1 1

Are osd-secondary and osd-tertiary copys parallel writes?
by jsterr＠proxforge.de 12 Nov '25

12 Nov '25

Hello Ceph-Users, I could not find a information about this in the docs: Are osd-secondary and osd-tertiary parallel writes? Thanks for the help. Greetings -- Jonas Sterr

3 2

pools nearfull (because cephfs is lazy in releasing space for deleted files ?)
by Massimo Sgaravatto 12 Nov '25

12 Nov '25

Dear all We have a portion of a Cephfs file system that maps to a ceph pool called cephfs_data_ssd. If I perform a "du -sh" on this portion of the file system, I see that the value matches the "STORED" field of the "ceph df" output for the cephfs_data_ssd pool. So far so good. I set a quota of 4.5 TB for this file system area. During the weekend, this pool (and other pools of the same device class) became nearfull. A "ceph df" showed that the problem was indeed in the the cephfs_data_ssd pool, with a reported usage of 7 TiB of data (21 TiB in replica 3): cephfs_data_ssd 62 32 7.1 TiB 2.02M 21 TiB 89.06 898 GiB This sounds strange to me because I set a quota of 4.5 TB in that area, and because a "du -sh" of the relevant directory showed a usage of 600 GB. When I lowered the disk quota from 4.5 TB to 600 GB, the jobs writing in that area failed (because of disk quota exceeded) and after a while the space was released. The only explanation I can think of is that, as far as I understand, cephfs can take a while to release the space for deleted files (https://docs.ceph.com/en/reef/dev/delayed-delete/) This would also be consistent with the fact that it looks like some jobs were performing a lot of writes and deletions (they kept writing a ~ 5GB checkpoint file, and deleting the previous one after each iteration). How can I understand from the log files if this was indeed the problem ? Or do you have some other possible explanations for this problem ? And, most important, how can I prevent scenarios such as this one ? Thanks, Massimo

3 8

How important is the "default" data pool being replicated for CephFS
by Mikael Öhman 07 Nov '25

07 Nov '25

The recommendations for cephfs is to make a replicated default data pool, and adding any EC data pools using layouts: https://docs.ceph.com/en/latest/cephfs/createfs/ > If erasure-coded pools are planned for file system data, it is best to configure the default as a replicated pool to improve small-object write and read performance when updating backtraces. I have an cephfs that unfortunately wasn't set up like this: they just made an EC pool on the slow HDDs as the default, which sounds like the worst case scenario to me. I would like to add a NVME data pool to this ceph fs, but recommended gives me pause on if i should instead go through the hassle of creating a new cephfs and migrating all users. I've tried to run some mdtest with small 1k files to see if i could measure this difference, but speed is about the same in my relatively small tests so far. I'm also not sure what impact I should realistically expect here. I don't even know if creating files counts as "updating backtraces", so my testing might just be pointless. I guess my core question is; just how important is this suggestion to keep the default data pool on replicated NVME? Setup: 14 hosts x 42 HDD + 3 NVMEs for db/wal 2*2x25 GbitE bonds 12 hosts x 10 NVME. 2*2x100 GbitE bonds Old CephFS setup: - metadata: replicated NVME - data-pools: EC 10+2 on HDD (i plan to add a EC NVME pool here via layouts) New CephFS setup as recommended: - metadata: replicated NVME - data-pools: replicated NVME (default), EC 8+2 on HDD via layout, EC 8+2 on NVME via layout. Ceph 18.2.7 Best regards, Mikael

4 5

Snaptrim flood
by Lukasz Gomulka 06 Nov '25

06 Nov '25

Dear all, In the previous year, we were forced to migrate all data from one cluster due to a critical issue with snaptrimq length, which caused the cluster to become unstable. Unfortunately, we are now experiencing the same problem again. We've been flooded with a massive queue of snaptrims despite a massive lack of snapshot deletion. Within seconds, every PG received over 2 million snaps to delete. It looks like some kind of database corruption. 1.1 SNAPTRIMQ_LEN 2264067 Last Year it was Octopus cluster (15.2.17). Now it is cluster in Quincy (17.2.7). Originally cluster was created in version 14.2.16 and upgraded -> 15.2.17 -> 17.2.7. Below I attached background/history in chronological order from earliest. I have also attach logs from snaptrimming osd in text file. ---------- BACKGROUND: All was good: > 2025-10-16T21:30:09.776167+0000 mgr.mgr-01-<cluster-fsid> (mgr.8232603159) 47659 : cluster [DBG] pgmap v22191: 8193 pgs: 2 active+clean+scrubbing+deep, 8191 active+clean; 183 TiB data, 542 TiB used, 419 TiB / 960 TiB avail; 491 MiB/s rd, 359 MiB/s wr, 20.99k op/s First laggy pg: > 2025-10-16T21:30:11.789813+0000 mgr.mgr-01-<cluster-fsid> (mgr.8232603159) 47661 : cluster [DBG] pgmap v22192: 8193 pgs: 4 active+clean+laggy, 2 active+clean+scrubbing+deep, 8187 active+clean; 183 TiB data, 542 TiB used, 419 TiB / 960 TiB avail; 328 MiB/s rd, 247 MiB/s wr, 14.01k op/s First hearbeat problem: > 2025-10-16T21:30:21.893+0000 7f578efd7700 -1 osd.yyy 602287 heartbeat_check: no reply from xyx.xyz.xyz.xyz:6802 osd.xxx since back 2025-10-16T21:29:57.827954+0000 front 2025-10-16T21:29:57.828552+0000 (oldest deadline 2025-10-16T21:30:21.328590+0000) First report of failed osd: > 2025-10-16T21:30:21.841714+0000 mon.mon-01-<cluster-fsid> (mon.0) 5235394 : cluster [DBG] osd.16 reported failed by osd.182 First failed osd: > 2025-10-16T21:30:21.841681+0000 mon.mon-01-<cluster-fsid> (mon.0) 5235393 : cluster [INF] osd.15 failed (root=default,rack=rack123,host=host-123) (2 reporters from different host after 23.000420 >= grace 20.000000) Host down: > 2025-10-16T21:30:22.914804+0000 mon.mon-01-<cluster-fsid> (mon.0) 5235451 : cluster [WRN] Health check failed: 1 host (24 osds) down (OSD_HOST_DOWN) Some osd up again: > 2025-10-16T21:30:22.892201+0000 mon.mon-01-<cluster-fsid> (mon.0) 5235450 : cluster [DBG] osd.22 failure report canceled by osd.64 > 2025-10-16T21:30:21.904+0000 7f4c358ca700 1 mon.mon-04-<cluster-fsid>@1(peon).osd e602288 e602288: 283 total, 266 up, 283 in Cluster during recovering: > 2025-10-16T21:30:23.865481+0000 mgr.mgr-01-<cluster-fsid> (mgr.8232603159) 47668 : cluster [DBG] pgmap v22200: 8193 pgs: 34 peering, 648 stale+active+clean, 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 676 active+clean+laggy, 60 active+undersized+degraded, 6773 active+clean; 183 TiB data, 542 TiB used, 419 TiB / 960 TiB avail; 4.7 MiB/s rd, 45 MiB/s wr, 1.28k op/s; 372351/152450688 objects degraded (0.244%) Again all down: > 2025-10-16T21:30:23.936587+0000 mon.mon-01-<cluster-fsid> (mon.0) 5235453 : cluster [DBG] osdmap e602290: 283 total, 259 up, 283 in ... few osds were flapping up and down x few times (about 5) (259->266->259...->259) ... Finally problem was born: >2025-10-16T21:30:34.405+0000 7fe5613fc700 0 log_channel(cluster) log [WRN] : Health check failed: snap trim queue for 4133 pg(s) >= 1024 (mon_osd_snap_trim_queue_warn_on) (PG_SLOW_SNAP_TRIMMING) During problem our monitors with joy were compacting rocksdb: > 2025-10-16T21:30:26.224+0000 7f4c310c1700 4 rocksdb: [db/db_impl/db_impl_compaction_flush.cc:1615] [default] Manual compaction starting (mon-04-<cluster-fsid>.caas.ovh.net) > 2025-10-16T21:30:26.820+0000 7f0fefa75700 4 rocksdb: [db/db_impl/db_impl_compaction_flush.cc:1615] [default] Manual compaction starting (mon-05-<cluster-fsid>.caas.ovh.net) ---------- Of course we ended with 2M+ snaps to remove on every pg. Additionally after few minutes we saw such operation on mon: > 2025-10-16T21:37:11.434+0000 7fe5613fc700 -1 mon.mon-01-<cluster-fsid>@0(leader) e10 get_health_metrics reporting 1 slow ops, oldest is pool_op(delete unmanaged snap pool 1 tid 7113677 name v602484) Do you have any idea what might be the reason for this issue and how to address it? Have you encountered a similar problem elsewhere? Currently, we are running snaptrim, but it may take several weeks to complete, and during this time, the cluster is severely impacted. Maybe do You have any idea what to check next, where can we found a clue. Thank You for any help, Best Regards, Lukasz Lucki Gomulka

2 4

Re: Ceph osdmap inconsistency after upgrade to Squid
by Huseyin Cotuk 05 Nov '25

05 Nov '25

Hi Eugen, I have already tried as the below method I stated before. I got the current osdmap and tried to set-osdmap via ceph-object-store-tool. But the command failed with error: ceph osd getmap 72555 > /tmp/osd_map_72555 CEPH_ARGS="--bluestore-ignore-data-csum" ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-11/ --op set-osdmap --file /tmp/osd_map_72555 osdmap (#-1:9c8e9ef2:::osdmap.72555:0#) does not exist. https://www.mail-archive.com/ceph-users@ceph.io/msg11545.html BR, Huseyin hcotuk(a)gmail.com

4 4

squid 19.2.3 - failed to probe daemon or devices - /var/lib/ceph read-only
by Steven Vacaroaia 31 Oct '25

31 Oct '25

Hi, I just noticed this error "failed to probe daemon or devices" which seems to be related to /var/lib/ceph being read only I checked the host ( ceph-host-1) and it seems fine so it must be a docker instance How do I identify which one and how can I fix it please ? Many thanks Steven root@ceph-host-1:~/utilities# ceph -s cluster: id: 0cfa836d-68b5-11f0-90bf-7cc2558e5ce8 health: HEALTH_WARN failed to probe daemons or devices services: mon: 5 daemons, quorum ceph-host-1,ceph-host-2,ceph-host-3,ceph-host-7,ceph-host-6 (age 8h) mgr: ceph-host-1.lqlece(active, since 5w), standbys: ceph-host-2.suiuxi mds: 19/19 daemons up, 7 standby osd: 185 osds: 185 up (since 4w), 185 in (since 4w) flags noautoscale rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 4/4 healthy pools: 16 pools, 7137 pgs objects: 359.48M objects, 811 TiB usage: 1.2 PiB used, 1.1 PiB / 2.3 PiB avail pgs: 7034 active+clean 72 active+clean+scrubbing 31 active+clean+scrubbing+deep io: client: 30 MiB/s rd, 34 KiB/s wr, 11 op/s rd, 1 op/s wr cephadm shell -- ceph-volume lvm list --format json Inferring fsid 0cfa836d-68b5-11f0-90bf-7cc2558e5ce8 Inferring config /var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/mon.ceph-host-1/config docker: Error response from daemon: error while creating mount source path '/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/crash': mkdir /var/lib/ceph: read-only file system. root@ceph-host-1:~/utilities# ceph health detail HEALTH_WARN failed to probe daemons or devices [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices host ceph-host-1 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/mon.ceph-host-1/config Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE= quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 -e NODE_NAME=ceph-host-1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/run/ceph:z -v /var/log/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/log/ceph:z -v /var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v /tmp/ceph-tmptx7tjpjz:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 inventory --format=json-pretty --filter-for-batch /usr/bin/docker: stderr docker: Error response from daemon: error while creating mount source path '/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/crash': mkdir /var/lib/ceph: read-only file system. Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/__main__.py", line 5581, in <module> File "/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/__main__.py", line 5569, in main File "/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/__main__.py", line 409, in _infer_config File "/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/__main__.py", line 437, in _infer_image File "/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/cephadmlib/call_wrappers.py", line 307, in call_throws RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE= quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 -e NODE_NAME=ceph-host-1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/run/ceph:z -v /var/log/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/log/ceph:z -v /var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v /tmp/ceph-tmptx7tjpjz:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: error while creating mount source path '/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/crash': mkdir /var/lib/ceph: read-only file system.

1 2

Help with Jaeger Tracing for Ceph RGW in Kubernetes (Rook-Ceph Deployment)
by Ashok Kumar M 31 Oct '25

31 Oct '25

Hi Ceph Community, I'm currently running a Ceph cluster inside a Kubernetes environment using Rook-Ceph, and I'm working on enabling distributed tracing for Ceph RGW. As per the https://docs.ceph.com/en/latest/jaegertracing/ , I used the following docker run command to deploy a Jaeger all-in-one instance for testing: docker run -d --name jaeger \ -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \ -e COLLECTOR_OTLP_ENABLED=true \ -p 6799:6799/udp \ -p 6832:6832/udp \ -p 5778:5778 \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ -p 14250:14250 \ -p 14268:14268 \ -p 14269:14269 \ -p 9411:9411 \ jaegertracing/all-in-one:latest \ --processor.jaeger-compact.server-host-port=6799 However, when using the latest Jaeger version, I get the following error: unknown flag: --processor.jaeger-compact.server-host-port After researching, I discovered that this flag is only available in older versions of Jaeger (e.g., v1.27) and is no longer supported in Jaeger v2+. I then tried using a specific older Jaeger version (jaegertracing/all-in-one:1.27) that supports the flag. The container runs fine and listens on port 6799 as expected, but I still don’t see any RGW traces appearing in the Jaeger UI. My Setup: Ceph cluster deployed via Rook in Kubernetes Ceph RGW is running as a pod inside the cluster Jaeger all-in-one is running externally (for testing) Jaeger port 6799/udp is exposed and forwarded Tracing is enabled in Ceph config Questions: Could the issue be related to the Ceph RGW pod not being able to send UDP traces to the Jaeger container? Is there a specific configuration needed inside the RGW pod or Rook-Ceph to make sure tracing data is being sent correctly? Should I consider using the default Jaeger port 6831/udp instead of 6799/udp, and configure Ceph accordingly? Are there any known compatibility issues between Ceph RGW tracing and Jaeger v1.x? Could you suggest some troubleshooting steps to help verify whether traces are actually being sent from RGW and received by Jaeger? Any advice or working examples from a similar Kubernetes-based setup would be greatly appreciated. Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

2 2

CephFS object missing on disk.
by Leandro Ferrari 30 Oct '25

30 Oct '25

Hi Cephers, Recently, a split-brain occurred in my cluster and although it has recovered and is now healthy, when attempting to delete an empty directory in CephFS, the MDS server logs: [ERR] dir 0x100215b86b3.011* object missing on disk; some files may be lost ('[path to dir]') After this, the cluster enters in an error state with the message: [ERR] overall HEALTH_ERR 1 MDSs report damaged metadata I have already executed: $ceph tell mds.cephfs:0 scrub start [path to dir] recursive,repair,force As a response, I received information that some inodes, including the inode of the mentioned directory, were repaired and the cluster returned to a healthy state. [ERR] scrub: inode wrongly marked free: 0x100161edd3b [ERR] inode table repaired for inode: 0x100161edd3b [ERR] scrub: inode wrongly marked free: 0x1001c9b8246 [ERR] inode table repaired for inode: 0x1001c9b8246 ..... [ERR] scrub: inode wrongly marked free: 0x100017dc9a3 [ERR] inode table repaired for inode: 0x100017dc9a3 [ERR] scrub: inode wrongly marked free: 0x100113ef73a [ERR] inode table repaired for inode: 0x100113ef73a Part that informs that the directory inode was repaired [WRN] bad backtrace on inode 0x100215b86b3([path to dir]), rewriting it [ERR] scrub: inode wrongly marked free: 0x100215b86b3 [ERR] inode table repaired for inode: 0x100215b86b3 [INF] Scrub repaired inode 0x100215b86b3 [INF] Cluster is now healthy [INF] overall HEALTH_OK If I try to access the same directory again, the MDS server logs: [ERR] dir 0x100215b86b3.011* object missing on disk; Some files may be lost ('[path to dir]') Since the directory is empty, I don't mind losing it. How do I remove this information from MDS? Or what would be the recommended action? Ceph version: ceph version 17.2.8 (f817ceb7f187defb1d021d6328fa833eb8e943b3) quincy (stable) Thanks in advance, Leandro Ferrari

1 0

RGW objects stuck with user.rgw.amz-replication-status PENDING
by Adam Prycki 30 Oct '25

30 Oct '25

Hello, we are having issue with one of our multisite zones failing to synchronize. All replication shards are "busy", we have a lot of errors in `sync error list` on secondary zone. We are running on 19.2.3 (cephadm ubuntu 22.04) After some digging we have found that objects on our master zone have PENDING replication status For example: radosgw-admin object stat --bucket 7c52c9db-de61-4186-a3ce-e95a96df6be1 --object 0000d5d9-0a8c-40cf-88e9-30bf908c7743 "attrs": { "user.rgw.amz-replication-status": "PENDING", "user.rgw.content_type": "application/octet-stream", We have 3 zones in this zonegroup. Both secondary zones have this object replicated. "user.rgw.amz-replication-status": "REPLICA", But on the master zone it never got marked as "COMPLETED". Running data sync init on the bucket didn't help, we cannot fix this "PENDING" status on the ceph side without deleting and recreating affected bucket. Has anyone ever seen similar issue? My current theory is that something may have gone wrong during bucket resharding. I was running tests uploading small objects to a bucket and test was supposed to cause resharding mid benchmark. Best regards Adam Prycki

1 0

squid 19.2.3 - cephfs missing path
by Steven Vacaroaia 29 Oct '25

29 Oct '25

Hi, I have a cephfs subvolume ( out of 4) with " mount path missing " issue I think a user deleted it as it look different than regular data to him/her I am mounting at the group level so all subgroups are available under same path How concerned should I be ? Is there a way to recreate path on a cephfs subvolume ? what are my options for fixing it ? Please find below details Mounting /volumes/projects /projects/cephfs Cephfs status ceph fs status project_ec_ssd project_ec_ssd - 139 clients ============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active project_ec_ssd.ceph-host-5.henbit Reqs: 0 /s 6198k 6054k 1645k 107k 1 active project_ec_ssd.ceph-host-7.emayje Reqs: 0 /s 194 197 340 629 2 active project_ec_ssd.ceph-host-3.qebqlz Reqs: 0 /s 192 196 339 644 3 active project_ec_ssd.ceph-host-1.bwtdlt Reqs: 0 /s 196 199 338 729 4 active project_ec_ssd.ceph-host-4.xsoerf Reqs: 0 /s 196 200 339 93 POOL TYPE USED AVAIL metadata_fs_ssd metadata 18.0G 92.8T ssd_ec_project data 183T 241T ceph fs subvolumegroup ls project_ec_ssd [ { "name": "projects" } ] ceph fs subvolume ls project_ec_ssd --group_name projects [ { "name": "systems_archive" }, { "name": "arescore_moved" }, { "name": "alphacore_moved" }, { "name": "schematics" } ] ceph fs subvolume getpath project_ec_ssd alphacore_moved --group_name projects Error ENOENT: mount path missing for subvolume 'alphacore_moved' ceph fs subvolume getpath project_ec_ssd arescore_moved --group_name projects /volumes/projects/arescore_moved/ff576e35-2363-4e25-aa7a-e4a81693ab20 Many thanks Steven

3 3

Storage-ACK for writes after how many replica?
by jsterr＠proxforge.de 29 Oct '25

29 Oct '25

Hello Ceph-Users, I looked up in the docs but could not find a information about: * When does a storage-client get a write-acknowledgement Example: Replicated Pool 3/2 Will the write-ack come after reaching min-size or when all copys are written. Would be great if this could be added in the docs somewhere (maybe I did not find it - and its already there?). Thanks, Greetings Jonas

3 2

RGW Cascading Failures After Simultaneous Host Loss
by Ramin Najjarbashi 28 Oct '25

28 Oct '25

Hi We recently experienced an incident in one of our production Ceph clusters and would greatly appreciate the community’s input. Our configuration uses Erasure Coding with a 4+2 profile for the main S3 data pools. During the incident, two storage hosts that contained several OSDs each became unavailable simultaneously. As a result, a large number of PGs entered an inactive state because the number of remaining fragments was insufficient for reconstruction under the EC 4+2 profile. This triggered a cascading failure in the RGW service layer: • Incoming S3 requests continued flowing into HAProxy. • RGW attempted to process requests that required I/O to inactive PGs. • I/O stalls caused excessive queue buildup and memory usage on RGWs. • Multiple RGW processes were terminated by the OOM Killer. • RGW daemons repeatedly restarted and were again selected by HAProxy. • The cycle continued, leading to severe service disruption. This behavior continued until the failed hosts were recovered and PGs regained a clean and active state. One interesting observation was that a separate set of RGW daemons dedicated only to internal consumers continued operating normally. Their traffic load was significantly lower, which prevented queue saturation and memory exhaustion. We are seeking guidance on two key questions: 1. In an EC 4+2 pool design, is such a cascading RGW failure considered inevitable if enough hosts fail to cause PG unavailability, or are there established best practices to prevent this behavior? 2. Based on our observation that internal RGWs remained healthy, is isolating customers by assigning separate RGWs to high-traffic and low-traffic groups considered a recommended architectural approach? Thank you in advance for any advice and shared experiences. Best regards, Ramin

2 2

RocksDB corruption on multiple OSDs
by Malcolm Haak 28 Oct '25

28 Oct '25

Hello, I have had a very bad thing happen. 10 of my OSD's were attached to two hosts at the same time. They were automatically mounted at system boot. When I realised I stopped them, one node at a time. However they now wont mount (with the SAS pathing fixed) I'm getting rocksDB corruption issues (Unsurprisingly) that cannot be resolved by ceph-bluestore-tool or ceph-kvstore-tool as they cannot even deal with the rocksdb's. I've lost enough disks that entire PG's are wiped out. I'm hoping there is a way to recover some of the data. I have another 6 4TB disks that are blank and can be used if it will help. Some of the errors I'm seeing are: rocksdb: Corruption: Mismatch in unique ID on table file 41122. Expected: {4769373877066773223,17385751700056157561} Actual: {17358529186729819771,7012539495376980102} in file db/MANIFEST-0411055 rocksdb: verify_sharding unable to list column families: Corruption: checksum mismatch in file db/MANIFEST-062921 And a different one I can't find right now, but it says something about missing a newline.... Any advice or help would be appreciated, Thanks in advance, Mal

2 2

cephadm deploy requirements
by Francesco Di Nucci 28 Oct '25

28 Oct '25

Hi all, I'm working to deploy a new Ceph cluster using cephadm and I have some doubts about the requirements: * how should I setup container networking on OSD nodes? I see that it is advised to have two interfaces, one in the public network and one in the cluster network (for OSD-only communication), do I need to make changes from the default Docker/Podman bridge networking? * Are there any particular steps to do on SELinux-enabled systems? Thanks in advance Francesco -- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples Email:francesco.dinucci@na.infn.it

3 6

tentacle 20.2.0 GA QE validation status
by Yuri Weinstein 27 Oct '25

27 Oct '25

This release is RC1 and cherry picks from the list https://pad.ceph.com/p/tentacle_RC1_must-have Some suites need a rerun, but we are waiting for build fixing. Details of this release are summarized here: https://tracker.ceph.com/issues/73556#note-1 Release Notes - https://github.com/ceph/ceph/pull/65971 LRC upgrade - TBD Dev Leads, please review the list of suites for completeness, as this is a new release. Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Adam King & Guillaume rbd: Ilya krbd: Ilya nvmeof: Aviv, Vallari upgrade/squid-x: NEED RERUN WAITING FOR BUILD upgrade/reef-x: NEED RERUN WAITING FOR BUILD upgrade/telemetry-upgrade: PASSED upgrade/tentacle-p2p: N/A upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED perf-basic: PASSED crimson-rados: Matan powercycle: PASSED TIA

10 12

Re: Ceph osdmap inconsistency after upgrade to Squid
by Huseyin Cotuk 24 Oct '25

24 Oct '25

Hi Kirby, Not at all. It still sends the ticks and you can see the last lines of the ongoing logs below: Oct 24 19:58:35 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:35.798+0000 7e2cb3476640 20 osd.11 70297 check_full_status cur ratio 0.77673, physical ratio 0.77673, new state none Oct 24 19:58:36 ank-backup01 ceph-osd[586118]: osd.11 70297 tick_without_osd_lock Oct 24 19:58:36 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:36.106+0000 7e2cc86b8640 10 osd.11 70297 tick_without_osd_lock Oct 24 19:58:36 ank-backup01 ceph-osd[586118]: osd.11 70297 tick Oct 24 19:58:36 ank-backup01 ceph-osd[586118]: osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 19:58:36 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:36.661+0000 7e2cc8eb9640 10 osd.11 70297 tick Oct 24 19:58:36 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:36.661+0000 7e2cc8eb9640 20 osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 19:58:37 ank-backup01 ceph-osd[586118]: osd.11 70297 tick_without_osd_lock Oct 24 19:58:37 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:37.094+0000 7e2cc86b8640 10 osd.11 70297 tick_without_osd_lock Oct 24 19:58:37 ank-backup01 ceph-osd[586118]: osd.11 70297 tick Oct 24 19:58:37 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:37.681+0000 7e2cc8eb9640 10 osd.11 70297 tick Oct 24 19:58:37 ank-backup01 ceph-osd[586118]: osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 19:58:37 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:37.681+0000 7e2cc8eb9640 20 osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 19:58:38 ank-backup01 ceph-osd[586118]: osd.11 70297 tick_without_osd_lock Oct 24 19:58:38 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:38.074+0000 7e2cc86b8640 10 osd.11 70297 tick_without_osd_lock Oct 24 19:58:38 ank-backup01 ceph-osd[586118]: osd.11 70297 tick Oct 24 19:58:38 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:38.644+0000 7e2cc8eb9640 10 osd.11 70297 tick Oct 24 19:58:38 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:38.644+0000 7e2cc8eb9640 20 osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 19:58:38 ank-backup01 ceph-osd[586118]: osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 19:58:39 ank-backup01 ceph-osd[586118]: osd.11 70297 tick_without_osd_lock Oct 24 19:58:39 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:39.064+0000 7e2cc86b8640 10 osd.11 70297 tick_without_osd_lock Oct 24 19:58:39 ank-backup01 ceph-osd[586118]: osd.11 70297 tick Oct 24 19:58:39 ank-backup01 ceph-osd[586118]: osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 19:58:39 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:39.658+0000 7e2cc8eb9640 10 osd.11 70297 tick Oct 24 19:58:39 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:39.658+0000 7e2cc8eb9640 20 osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 19:58:40 ank-backup01 ceph-osd[586118]: osd.11 70297 tick_without_osd_lock Oct 24 19:58:40 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:40.071+0000 7e2cc86b8640 10 osd.11 70297 tick_without_osd_lock Oct 24 19:58:40 ank-backup01 ceph-osd[586118]: osd.11 70297 reports for 0 queries Oct 24 19:58:40 ank-backup01 ceph-osd[586118]: osd.11 70297 collect_pg_stats Oct 24 19:58:40 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:40.415+0000 7e2cc18ba640 20 osd.11 70297 reports for 0 queries Oct 24 19:58:40 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:40.415+0000 7e2cc18ba640 15 osd.11 70297 collect_pg_stats Oct 24 19:58:40 ank-backup01 ceph-osd[586118]: osd.11 70297 tick Oct 24 19:58:40 ank-backup01 ceph-osd[586118]: osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 19:58:40 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:40.624+0000 7e2cc8eb9640 10 osd.11 70297 tick Oct 24 19:58:40 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:40.624+0000 7e2cc8eb9640 20 osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 19:58:41 ank-backup01 ceph-osd[586118]: osd.11 70297 tick_without_osd_lock Oct 24 19:58:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T16:58:41.048+0000 7e2cc86b8640 10 osd.11 70297 tick_without_osd_lock Selamlar, Huseyin Cotuk hcotuk(a)gmail.com > On 24 Oct 2025, at 19:57, Kirby Haze <kirbyhaze01(a)gmail.com> wrote: > > Are these the end of the logs? This actually looks normal and fine but eventually it will crash? or does the systemd unit stay running and they are stuck on these logs? > > On Fri, Oct 24, 2025 at 9:26 AM Huseyin Cotuk <hcotuk(a)gmail.com <mailto:hcotuk@gmail.com>> wrote: >> journalctl logs of a failing OSD starts like in the link: https://pastes.io/osd11-restart-log >> >> Oct 24 18:50:46 ank-backup01 systemd[1]: Started ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f(a)osd.11.service - Ceph osd.11 for 4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f. >> Oct 24 18:50:46 ank-backup01 ceph-osd[586118]: set uid:gid to 167:167 (ceph:ceph) >> Oct 24 18:50:46 ank-backup01 ceph-osd[586118]: ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable), process ceph-osd, pid 2 >> Oct 24 18:50:46 ank-backup01 ceph-osd[586118]: pidfile_write: ignore empty --pid-file >> >> I can see that it runs with the Ceph version 19.2.3. >> >> After the activation process (it takes some time), it does not crash actually. AFAIR from the logs, it mounts bluefs, starts the rocksdb version 7.9.2 and loads the tables. It evens compacts the rocksdb. >> >> Afterwards, it triggers the boot process as follows. >> >> Oct 24 18:53:41 ank-backup01 ceph-osd[586118]: osd.11 0 journal looks like hdd >> Oct 24 18:53:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:53:41.705+0000 7e2ccf90a740 2 osd.11 0 journal looks like hdd >> Oct 24 18:53:41 ank-backup01 ceph-osd[586118]: osd.11 0 boot >> Oct 24 18:53:41 ank-backup01 ceph-osd[586118]: osd.11 0 configured osd_max_object_name[space]_len looks ok >> Oct 24 18:53:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:53:41.705+0000 7e2ccf90a740 2 osd.11 0 boot >> Oct 24 18:53:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:53:41.705+0000 7e2ccf90a740 20 osd.11 0 configured osd_max_object_name[space]_len looks ok >> >> I noted that at this point, it loads the osdmap 70297 >> >> Oct 24 18:53:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:53:41.706+0000 7e2ccf90a740 20 osd.11 0 get_map 70297 - loading and decoding 0x565a07bb0000 >> >> There are lots of missing logs: >> >> Oct 24 18:53:48 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:53:48.603+0000 7e2ccf90a740 20 read_log_and_missing 68276'8386071 (0'0) modify 23:bf0d9243:::48f964cb-12c5-4689-8aa4-0b8e573f41d5.37815204.2_001%2f184%2f288%2f052.zip:head by client.43525885.0:1306616953 2025-10-09T11:06:44.324360+0000 0 ObjectCleanRegions clean_offsets: [(48911, 18446744073709502704)], clean_omap: true, new_object: false >> >> After all these missing logs, it starts and loads the PGs. >> >> Oct 24 18:54:05 ank-backup01 ceph-osd[586118]: osd.11 70297 load_pgs opened 66 pgs >> Oct 24 18:54:05 ank-backup01 ceph-osd[586118]: osd.11 70297 superblock: I am osd.11 >> Oct 24 18:54:05 ank-backup01 ceph-osd[586118]: osd.11 70297 heartbeat osd_stat(store_statfs(0x19ff4006000/0x0/0x74702400000, data 0x4b4a481b898/0x4b4a4e68000, compress 0x0/0x0/0x0, omap 0x1a50, meta 0xf26958e5b0), peers [] op hist []) >> Oct 24 18:54:05 ank-backup01 ceph-osd[586118]: osd.11 70297 done with init, starting boot process >> Oct 24 18:54:05 ank-backup01 ceph-osd[586118]: osd.11 70297 start_boot >> >> It periodically dumps stats that shows that OSD is running (up) for xxx secs. >> >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Uptime(secs): 600.6 total, 600.0 interval >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Flush(GB): cumulative 0.000, interval 0.000 >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: AddFile(GB): cumulative 0.000, interval 0.000 >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: AddFile(Total Files): cumulative 0, interval 0 >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: AddFile(L0 Files): cumulative 0, interval 0 >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: AddFile(Keys): cumulative 0, interval 0 >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_b> >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Block cache BinnedLRUCache@0x5659dca05350#2 capacity: 1.06 GB usage: 100.56 MB table_size: 0 occupancy: 18446744073709551615 collections: 2 last_copies: 8> >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Block cache entry stats(count,size,portion): DataBlock(24571,97.85 MB,8.99328%) FilterBlock(14,1.17 MB,0.107375%) IndexBlock(20,1.54 MB,0.141681%) Misc(1,> >> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: >> >> >> The problem is, it can not register itself to the mon as UP and running. There are lots of ticks and tick_without_osd_lock messages in the log file. >> >> Oct 24 18:57:14 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:57:14.370+0000 7e2cc8eb9640 10 osd.11 70297 tick >> Oct 24 18:57:14 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:57:14.370+0000 7e2cc8eb9640 20 osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 >> Oct 24 18:57:14 ank-backup01 ceph-osd[586118]: osd.11 70297 tick_without_osd_lock >> Oct 24 18:57:14 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:57:14.729+0000 7e2cc86b8640 10 osd.11 70297 tick_without_osd_lock >> Oct 24 18:57:15 ank-backup01 ceph-osd[586118]: osd.11 70297 reports for 0 queries >> Oct 24 18:57:15 ank-backup01 ceph-osd[586118]: osd.11 70297 collect_pg_stats >> >> It looks like it stays in a loop that send ticks with the older epoch (70297). >> >> Thanks for your help >> >> BR, >> Huseyin Cotuk >> hcotuk(a)gmail.com <mailto:hcotuk@gmail.com> >> >> >> >> >>> On 24 Oct 2025, at 18:36, Kirby Haze <kirbyhaze01(a)gmail.com <mailto:kirbyhaze01@gmail.com>> wrote: >>> >>> if you look at the osd’s logs and follow it while trigger a restart through systemctl >>> >>> systemctl list-units | grep osd >>> >>> journalctl -u <osd unit> >>> >>> When the osd boots up it spits out the version, I would be curious to see how it crashes and the stack trace >>> >>> On Fri, Oct 24, 2025 at 8:31 AM Huseyin Cotuk <hcotuk(a)gmail.com <mailto:hcotuk@gmail.com>> wrote: >>>> Hi Kirby, >>>> >>>> Is there any way to check whether these OSDs have completed the upgrade successfully or not? >>>> >>>> Any help will be appreciated >>>> >>>> BR, >>>> Huseyin >>>> hcotuk(a)gmail.com <mailto:hcotuk@gmail.com> >>>> >>>> >>>> >>>> >>>>> On 24 Oct 2025, at 18:14, Kirby Haze <kirbyhaze01(a)gmail.com <mailto:kirbyhaze01@gmail.com>> wrote: >>>>> >>>>> Osds usually try to load the last clean map on boot, but if these osds finished the upgrade it must have hit an assert I think >>>>> >>>>> On Fri, Oct 24, 2025 at 8:06 AM Huseyin Cotuk <hcotuk(a)gmail.com <mailto:hcotuk@gmail.com>> wrote: >>>>>> Hi Can, >>>>>> >>>>>> No, I did not add or recreate these OSDs. I tried cephadm to run failing OSDs with the help of following commands without success: >>>>>> >>>>>> cephadm --image quay.io/ceph/ceph:v19 <http://quay.io/ceph/ceph:v19> unit --name osd.$i --fsid 4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f restart >>>>>> >>>>>> cephadm --image aade1b12b8e6 run --name osd.$i --fsid 4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f >>>>>> >>>>>> BR, >>>>>> Huseyin >>>>>> hcotuk(a)gmail.com <mailto:hcotuk@gmail.com> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> > On 24 Oct 2025, at 17:55, Can Özyurt <acozyurt(a)gmail.com <mailto:acozyurt@gmail.com>> wrote: >>>>>> > >>>>>> > Hi Huseyin, >>>>>> > >>>>>> > Did you add or recreate these OSDs after the upgrade? >>>>>> > >>>>>> > On Fri, 24 Oct 2025 at 16:42, Huseyin Cotuk <hcotuk(a)gmail.com <mailto:hcotuk@gmail.com> <mailto:hcotuk@gmail.com <mailto:hcotuk@gmail.com>>> wrote: >>>>>> >> More debug logs from a failing OSD: >>>>>> >> >>>>>> >> Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 20 osd.11 70297 reports for 0 queries >>>>>> >> Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 15 osd.11 70297 collect_pg_stats >>>>>> >> Oct 24 17:39:52 ank-backup01 ceph-osd[4100574]: osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish reporting purged_snaps [] >>>>>> >> Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 20 osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish reporting purged_snaps [] >>>>>> >> Oct 24 17:39:52 ank-backup01 ceph-osd[4100574]: osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish publish_stats_to_osd 70297:52872808 >>>>>> >> Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 15 osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish publish_stats_to_osd 70297:52872808 >>>>>> >> Oct 24 17:39:52 ank-backup01 ceph-osd[4100574]: osd.11 pg_epoch: 70297 pg[23.5fs2( v 69081'8395094 (68276'8390628,69081'8395094] local-lis/les=69078/69079 n=8316950 ec=21807/21807 lis/c=69078/68833 les/c/f=69079/68834/39143 sis=70280) [NONE,NONE,11,8,14,16,0,20,17,18,NONE]p11(2) r=2 lpr=70297 pi=[68833,70280)/7 crt=69081'8395094 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish reporting purged_snaps [] >>>>>> >> >>>>>> >> >>>>>> >> Selamlar, >>>>>> >> Huseyin Cotuk >>>>>> >> hcotuk(a)gmail.com <mailto:hcotuk@gmail.com> <mailto:hcotuk@gmail.com <mailto:hcotuk@gmail.com>> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> _______________________________________________ >>>>>> >> ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> <mailto:ceph-users@ceph.io <mailto:ceph-users@ceph.io>> >>>>>> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io> <mailto:ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>> >>>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> >>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io> >>

2 1

Re: Ceph osdmap inconsistency after upgrade to Squid
by Huseyin Cotuk 24 Oct '25

24 Oct '25

journalctl logs of a failing OSD starts like in the link: https://pastes.io/osd11-restart-log Oct 24 18:50:46 ank-backup01 systemd[1]: Started ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f(a)osd.11.service - Ceph osd.11 for 4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f. Oct 24 18:50:46 ank-backup01 ceph-osd[586118]: set uid:gid to 167:167 (ceph:ceph) Oct 24 18:50:46 ank-backup01 ceph-osd[586118]: ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable), process ceph-osd, pid 2 Oct 24 18:50:46 ank-backup01 ceph-osd[586118]: pidfile_write: ignore empty --pid-file I can see that it runs with the Ceph version 19.2.3. After the activation process (it takes some time), it does not crash actually. AFAIR from the logs, it mounts bluefs, starts the rocksdb version 7.9.2 and loads the tables. It evens compacts the rocksdb. Afterwards, it triggers the boot process as follows. Oct 24 18:53:41 ank-backup01 ceph-osd[586118]: osd.11 0 journal looks like hdd Oct 24 18:53:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:53:41.705+0000 7e2ccf90a740 2 osd.11 0 journal looks like hdd Oct 24 18:53:41 ank-backup01 ceph-osd[586118]: osd.11 0 boot Oct 24 18:53:41 ank-backup01 ceph-osd[586118]: osd.11 0 configured osd_max_object_name[space]_len looks ok Oct 24 18:53:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:53:41.705+0000 7e2ccf90a740 2 osd.11 0 boot Oct 24 18:53:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:53:41.705+0000 7e2ccf90a740 20 osd.11 0 configured osd_max_object_name[space]_len looks ok I noted that at this point, it loads the osdmap 70297 Oct 24 18:53:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:53:41.706+0000 7e2ccf90a740 20 osd.11 0 get_map 70297 - loading and decoding 0x565a07bb0000 There are lots of missing logs: Oct 24 18:53:48 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:53:48.603+0000 7e2ccf90a740 20 read_log_and_missing 68276'8386071 (0'0) modify 23:bf0d9243:::48f964cb-12c5-4689-8aa4-0b8e573f41d5.37815204.2_001%2f184%2f288%2f052.zip:head by client.43525885.0:1306616953 2025-10-09T11:06:44.324360+0000 0 ObjectCleanRegions clean_offsets: [(48911, 18446744073709502704)], clean_omap: true, new_object: false After all these missing logs, it starts and loads the PGs. Oct 24 18:54:05 ank-backup01 ceph-osd[586118]: osd.11 70297 load_pgs opened 66 pgs Oct 24 18:54:05 ank-backup01 ceph-osd[586118]: osd.11 70297 superblock: I am osd.11 Oct 24 18:54:05 ank-backup01 ceph-osd[586118]: osd.11 70297 heartbeat osd_stat(store_statfs(0x19ff4006000/0x0/0x74702400000, data 0x4b4a481b898/0x4b4a4e68000, compress 0x0/0x0/0x0, omap 0x1a50, meta 0xf26958e5b0), peers [] op hist []) Oct 24 18:54:05 ank-backup01 ceph-osd[586118]: osd.11 70297 done with init, starting boot process Oct 24 18:54:05 ank-backup01 ceph-osd[586118]: osd.11 70297 start_boot It periodically dumps stats that shows that OSD is running (up) for xxx secs. Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Uptime(secs): 600.6 total, 600.0 interval Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Flush(GB): cumulative 0.000, interval 0.000 Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: AddFile(GB): cumulative 0.000, interval 0.000 Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: AddFile(Total Files): cumulative 0, interval 0 Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: AddFile(L0 Files): cumulative 0, interval 0 Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: AddFile(Keys): cumulative 0, interval 0 Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_b> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Block cache BinnedLRUCache@0x5659dca05350#2 capacity: 1.06 GB usage: 100.56 MB table_size: 0 occupancy: 18446744073709551615 collections: 2 last_copies: 8> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: Block cache entry stats(count,size,portion): DataBlock(24571,97.85 MB,8.99328%) FilterBlock(14,1.17 MB,0.107375%) IndexBlock(20,1.54 MB,0.141681%) Misc(1,> Oct 24 19:03:41 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: The problem is, it can not register itself to the mon as UP and running. There are lots of ticks and tick_without_osd_lock messages in the log file. Oct 24 18:57:14 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:57:14.370+0000 7e2cc8eb9640 10 osd.11 70297 tick Oct 24 18:57:14 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:57:14.370+0000 7e2cc8eb9640 20 osd.11 70297 tick last_purged_snaps_scrub 2025-10-24T08:38:08.454484+0000 next 2025-10-25T18:30:10.329484+0000 Oct 24 18:57:14 ank-backup01 ceph-osd[586118]: osd.11 70297 tick_without_osd_lock Oct 24 18:57:14 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[586110]: 2025-10-24T15:57:14.729+0000 7e2cc86b8640 10 osd.11 70297 tick_without_osd_lock Oct 24 18:57:15 ank-backup01 ceph-osd[586118]: osd.11 70297 reports for 0 queries Oct 24 18:57:15 ank-backup01 ceph-osd[586118]: osd.11 70297 collect_pg_stats It looks like it stays in a loop that send ticks with the older epoch (70297). Thanks for your help BR, Huseyin Cotuk hcotuk(a)gmail.com > On 24 Oct 2025, at 18:36, Kirby Haze <kirbyhaze01(a)gmail.com> wrote: > > if you look at the osd’s logs and follow it while trigger a restart through systemctl > > systemctl list-units | grep osd > > journalctl -u <osd unit> > > When the osd boots up it spits out the version, I would be curious to see how it crashes and the stack trace > > On Fri, Oct 24, 2025 at 8:31 AM Huseyin Cotuk <hcotuk(a)gmail.com <mailto:hcotuk@gmail.com>> wrote: >> Hi Kirby, >> >> Is there any way to check whether these OSDs have completed the upgrade successfully or not? >> >> Any help will be appreciated >> >> BR, >> Huseyin >> hcotuk(a)gmail.com <mailto:hcotuk@gmail.com> >> >> >> >> >>> On 24 Oct 2025, at 18:14, Kirby Haze <kirbyhaze01(a)gmail.com <mailto:kirbyhaze01@gmail.com>> wrote: >>> >>> Osds usually try to load the last clean map on boot, but if these osds finished the upgrade it must have hit an assert I think >>> >>> On Fri, Oct 24, 2025 at 8:06 AM Huseyin Cotuk <hcotuk(a)gmail.com <mailto:hcotuk@gmail.com>> wrote: >>>> Hi Can, >>>> >>>> No, I did not add or recreate these OSDs. I tried cephadm to run failing OSDs with the help of following commands without success: >>>> >>>> cephadm --image quay.io/ceph/ceph:v19 <http://quay.io/ceph/ceph:v19> unit --name osd.$i --fsid 4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f restart >>>> >>>> cephadm --image aade1b12b8e6 run --name osd.$i --fsid 4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f >>>> >>>> BR, >>>> Huseyin >>>> hcotuk(a)gmail.com <mailto:hcotuk@gmail.com> >>>> >>>> >>>> >>>> >>>> > On 24 Oct 2025, at 17:55, Can Özyurt <acozyurt(a)gmail.com <mailto:acozyurt@gmail.com>> wrote: >>>> > >>>> > Hi Huseyin, >>>> > >>>> > Did you add or recreate these OSDs after the upgrade? >>>> > >>>> > On Fri, 24 Oct 2025 at 16:42, Huseyin Cotuk <hcotuk(a)gmail.com <mailto:hcotuk@gmail.com> <mailto:hcotuk@gmail.com <mailto:hcotuk@gmail.com>>> wrote: >>>> >> More debug logs from a failing OSD: >>>> >> >>>> >> Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 20 osd.11 70297 reports for 0 queries >>>> >> Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 15 osd.11 70297 collect_pg_stats >>>> >> Oct 24 17:39:52 ank-backup01 ceph-osd[4100574]: osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish reporting purged_snaps [] >>>> >> Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 20 osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish reporting purged_snaps [] >>>> >> Oct 24 17:39:52 ank-backup01 ceph-osd[4100574]: osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish publish_stats_to_osd 70297:52872808 >>>> >> Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 15 osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish publish_stats_to_osd 70297:52872808 >>>> >> Oct 24 17:39:52 ank-backup01 ceph-osd[4100574]: osd.11 pg_epoch: 70297 pg[23.5fs2( v 69081'8395094 (68276'8390628,69081'8395094] local-lis/les=69078/69079 n=8316950 ec=21807/21807 lis/c=69078/68833 les/c/f=69079/68834/39143 sis=70280) [NONE,NONE,11,8,14,16,0,20,17,18,NONE]p11(2) r=2 lpr=70297 pi=[68833,70280)/7 crt=69081'8395094 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish reporting purged_snaps [] >>>> >> >>>> >> >>>> >> Selamlar, >>>> >> Huseyin Cotuk >>>> >> hcotuk(a)gmail.com <mailto:hcotuk@gmail.com> <mailto:hcotuk@gmail.com <mailto:hcotuk@gmail.com>> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> _______________________________________________ >>>> >> ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> <mailto:ceph-users@ceph.io <mailto:ceph-users@ceph.io>> >>>> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io> <mailto:ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io>

1 0

Re: Ceph osdmap inconsistency after upgrade to Squid
by Huseyin Cotuk 24 Oct '25

24 Oct '25

More debug logs from a failing OSD: Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 20 osd.11 70297 reports for 0 queries Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 15 osd.11 70297 collect_pg_stats Oct 24 17:39:52 ank-backup01 ceph-osd[4100574]: osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish reporting purged_snaps [] Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 20 osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish reporting purged_snaps [] Oct 24 17:39:52 ank-backup01 ceph-osd[4100574]: osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish publish_stats_to_osd 70297:52872808 Oct 24 17:39:52 ank-backup01 ceph-4e7e7d1c-22db-49c7-9f24-5a75cd3a3b9f-osd-11[4100566]: 2025-10-24T14:39:52.271+0000 77f47e35c640 15 osd.11 pg_epoch: 70297 pg[23.b9s0( v 69083'8392506 (68272'8387884,69083'8392506] local-lis/les=69076/69077 n=8314181 ec=21807/21807 lis/c=69076/68833 les/c/f=69077/68834/39143 sis=70280) [11,NONE,NONE,12,31,21,7,NONE,NONE,26,0]p11(0) r=0 lpr=70297 pi=[68833,70280)/5 crt=69083'8392506 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish publish_stats_to_osd 70297:52872808 Oct 24 17:39:52 ank-backup01 ceph-osd[4100574]: osd.11 pg_epoch: 70297 pg[23.5fs2( v 69081'8395094 (68276'8390628,69081'8395094] local-lis/les=69078/69079 n=8316950 ec=21807/21807 lis/c=69078/68833 les/c/f=69079/68834/39143 sis=70280) [NONE,NONE,11,8,14,16,0,20,17,18,NONE]p11(2) r=2 lpr=70297 pi=[68833,70280)/7 crt=69081'8395094 lcod 0'0 mlcod 0'0 unknown mbc={}] PeeringState::prepare_stats_for_publish reporting purged_snaps [] Selamlar, Huseyin Cotuk hcotuk(a)gmail.com

3 5

Ceph osdmap inconsistency after upgrade to Squid
by Huseyin Cotuk 24 Oct '25

24 Oct '25

Hi Cephers, I have recently completed Reef to Squid upgrade in our test environment, After the upgrade, some of the OSDs did not come up. podman containers are up and running but some OSDs did not join the cluster and seem down. After digging into logs, I saw that down OSDs try to boot with osdmap epoch 70297 but the current osdmap epoch is 72555. All the running OSDs use the latest osdmap on the logs, but the failing ones use the same and older osdmap (epcoh 70297). This is a single-node test cluster and has been running for years. All the OSDs are on the same host and all the containers related to OSDs are running. After enabling debug on a failing OSD, I can see that there are several ticks with the given epoch like below. 2025-10-23T21:00:23.945+0000 7f3e05b79640 20 osd.41 70297 tick last_purged_snaps_scrub 2025-10-22T22:20:42.570629+0000 next 2025-10-24T03:13:12.570629+000 I tried to get the latest osdmap via “ceph osd getmap 72555 > /tmp/osdmap.72555” and set with the command: CEPH_ARGS="--bluestore-ignore-data-csum" ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-41/ --op set-osdmap --file /tmp/osd_map_72555" but it fails with the error: osdmap (#-1:9c8e9ef2:::osdmap.72555:0#) does not exist. I can verify all the PGs and the underlying objects are staying safe on the disk by using the ceph-bluestore-tool. Does anyone have a clue why some of the OSDs do not get the latest osdmap from the mon? Thanks in advance, Huseyin hcotuk(a)gmail.com

1 0

Next Cephalocon (2026) - when/where?
by Sebastian 24 Oct '25

24 Oct '25

Hey, I know it is too early, but maybe you already know where it will be? Unfortunately, I have not received permission from the company to visit Cephalocon ‘25 in Vancouver. The rule is simple where I work. Conferences are allowed in the European continent. So when and where do you plan the next Cephalocon to be? I also consider the option to pay for it myself if it will be outside Europe. BTW, too much knowledge to lose IMO :) BR, Sebastian

3 2

Attention: Documentation
by abobakr osman 23 Oct '25

23 Oct '25

Small typo in the sentence[0]: " Baseboard Management Controller (BMC)...Additionally BMCs as of 2023 rarely sport network connections faster than 1 Gb/s, so dedicated and inexpensive 1 Gb/s switches for BMC administrative traffic may reduce costs by wasting fewer expenive ports on faster host switches. " Not sure if sport is intended here as well. [0] https://docs.ceph.com/en/latest/start/hardware-recommendations/#baseboard-m… Best regards Abobakr Mohammed

2 1

Very slow mirroring operation
by GLE, Vivien 23 Oct '25

23 Oct '25

Hi, I'm testing mirroring in ceph and dealt with very slow mirroring operation, Cluster A (primary) is on a Ceph hyperconverged on proxmox and cluster B (secondary) is a Ceph cluster managed by cephadm. These 2 have been peered via bootstrap. Moving 60 GiB VM to the mirrored pool in cluster A it took 5-6 hours to fully sync in the cluster B (SSD device only) Moving 60 GiB VM from cluster A to cluster B in replica 3 SSD pool took 2 minutes (SSD device only) It looks like for this purpose cluster wont be higher than 6-7 MiB/s, is this a normal behaviour ? Ceph -s during mirroring output and rbd info below ceph -s cluster: id: ID health: HEALTH_OK services: mon: 3 daemons, quorum r620-13-1,r620-13-9,r620-13-4 (age 12d) mgr: r620-13-7.zxzajo(active, since 24h), standbys: r620-13-10.wmzodp mds: 1/1 daemons up osd: 24 osds: 24 up (since 6d), 24 in (since 6d) rbd-mirror: 1 daemon active (1 hosts) rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 16 pools, 1287 pgs objects: 151.21k objects, 551 GiB usage: 1.6 TiB used, 6.3 TiB / 7.9 TiB avail pgs: 1287 active+clean io: client: 15 KiB/s rd, 6.5 MiB/s wr, 11 op/s rd, 419 op/s wr -------------------------------------------- rbd info ceph_proxmox/vm-118-disk-0 rbd image 'vm-118-disk-0': size 60 GiB in 15360 objects order 22 (4 MiB objects) snapshot_count: 0 id: abe8c560f69498 block_name_prefix: rbd_data.abe8c560f69498 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Wed Oct 22 07:52:31 2025 access_timestamp: Wed Oct 22 07:52:31 2025 modify_timestamp: Wed Oct 22 07:52:31 2025 journal: abe8c560f69498 mirroring state: enabled mirroring mode: journal mirroring global id: id mirroring primary: false ----------------------------------------- In cluster A ceph.conf rbd_default_features = 125 Thanks ! Vivien

3 3

newbie CRUSH rules / rbd pool
by Alexander Leutz 23 Oct '25

23 Oct '25

Hi everybody, I'm new to ceph. I'm testing 4 proxmox ceph nodes. I researched a lot on CRUSH rules and I build a rbd pool for use with my test-vms. I'm reading a lot in the official docs under docs.ceph.com. But I still have a lot of white space on my personal knowledge map. Are there any other recommendations for getting a full / deeper understanding of ceph? books? Best regards, Alexander Concat AG IT Solutions | Berliner Ring 127 - 129 | 64625 Bensheim | www.concat.de alexander.leutz(a)concat.de | Tel.: +49 6251 7026-143 | Mobil: +49 151 18856009 | Fax: +49 6251 7026-356 Hauptsitz Bensheim a. d. Bergstraße | Vorstand Olaf von Heyer Aufsichtsratsvorsitzender Tim Patronik | Amtsgericht Darmstadt (HRB 25326)

4 6

Ceph RGW Barbican Integration — Tenant isolation issue (Barbican keys from one project usable by another)
by Vignesh Varma 22 Oct '25

22 Oct '25

Hi Team, Lab Environment * Ceph: 19.2.x (Squid) * OpenStack: Caracal * OS: Ubuntu 22.04 LTS * Deployment: `cephadm` --- Setup Summary We have a working Ceph RGW + Barbican integration with: * Keystone auth: `rgw_s3_auth_use_keystone = true` * SSE-KMS backend: `rgw_crypt_s3_kms_backend = barbican` * Single RGW Barbican service user under a single project: ``` rgw_keystone_barbican_user = rgw-barbican rgw_keystone_barbican_project = service rgw_keystone_barbican_domain = default rgw_barbican_url = http://controller:9311 rgw_crypt_s3_kms_backend = barbican rgw_crypt_s3_kms_encryption_keys = {"coke":"<COKE_KEY_ID>","pepsi":"<PEPSI_KEY_ID>"} rgw_crypt_require_ssl = false ``` Each tenant (e.g., Coke, Pepsi) has: * Its own OpenStack project * Separate EC2 credentials * Its own Barbican AES-256 key (with `rgw-barbican` added to the secret ACL for read) S3 operations work for both tenants using their own EC2 credentials. --- Observed Issue Cross-tenant Barbican key usage is allowed: | Tenant | Key Used | Result | | ----------------- | ---------- | -------------- | | Coke → Coke key | ✅ Success | expected | | Pepsi → Pepsi key | ✅ Success | expected | | Coke → Pepsi key | ⚠️ Success | unexpected | | Pepsi → Coke key | ⚠️ Success | unexpected | If a client supplies any valid Barbican key UUID (one that `rgw-barbican` can read via ACL), RGW accepts it—even if the key belongs to a different project. This suggests RGW does not validate that the Barbican key is owned by (or scoped to) the same project as the authenticated S3 user. --- Minimal Repro 1. Create projects `coke` and `pepsi`; create per-project EC2 creds. 2. Store one Barbican key per project; grant `rgw-barbican` read ACL on both secrets. 3. Configure RGW as above with the `rgw_crypt_s3_kms_encryption_keys` mapping. 4. Using Coke EC2 creds, run `put-object` with Pepsi key UUID via `--ssekms-key-id <PEPSI_KEY_ID>`. → Upload succeeds; encryption uses Pepsi key. --- What We Need * A way for RGW to validate that the SSE-KMS key used in the request belongs to the same OpenStack project (tenant) as the authenticated S3 user. * Cross-project use of KMS keys should fail with `AccessDenied` (or similar). --- Questions for Guidance 1. Is there a supported (or planned) way to restrict RGW so the Barbican key must belong to the same project as the authenticated S3 user? 2. Can RGW be configured to derive the project scope from the S3 auth context when requesting the key from Barbican (instead of always using the fixed `rgw_keystone_barbican_project` for the token)? --- Best Regards, Vignesh

3 3

Re: tentacle 20.2.0 GA QE validation status
by Venky Shankar 22 Oct '25

22 Oct '25

(I sent my approval email from ists.ceph.io/hyperkitty as Yuri's original email landed in spam :/ adding users list back to this chain) On Wed, Oct 22, 2025 at 1:00 PM Venky Shankar <vshankar(a)redhat.com> wrote: > > Hi Yuri, > > fs approved. failures are > > https://tracker.ceph.com/projects/cephfs/wiki/Tentacle#2020 > > Cheers, > Venky > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io > -- Cheers, Venky

1 0

rbd volume map
by Iban Cabrillo 21 Oct '25

21 Oct '25

Good morning, I am having trouble mapping volumes using rbd. It used to work with v1 (I haven't tried it in a couple of years), but now we are using Quincy (v17) and I am getting this error: rbd --mon-host 10.10.3.1:3300,10.10.3.2:3300,10.10.3.3:3300 -p volumes map volume-c33b1ea9-c976-438b-8c3c-21cdf2db8bfd rbd: sysfs write failed In some cases useful info is found in syslog - try “dmesg | tail”. The mons are available and responding on port 3300. The volume I want to map is available, and I have disabled features: rbd --mon-host 10.10.3.1:3300,10.10.3.2:3300,10.10.3.3:3300 -p volumes info volume-c33b1ea9-c976-438b-8c3c-21cdf2db8bfd rbd image ‘volume-c33b1ea9-c976-438b-8c3c-21cdf2db8bfd’: size 200 GiB in 51200 objects order 22 (4 MiB objects) snapshot_count: 0 id: f6b4e9515f007c block_name_prefix: rbd_data.f6b4e9515f007c format: 2 features: layering op_features: flags: create_timestamp: Wed Dec 15 20:11:41 2021 access_timestamp: Thu Oct 2 12:04:14 2025 modify_timestamp: Wed Oct 15 17:36:12 2025 It does NOT seem to be an authentication issue, as I have used the admin user and key and the result is the same. Any ideas are welcome. Regards, I -- Jose Ibán Cabrillo Bartolomé cabrillo(a)ifca.unican.es Tlfno (+34) 942200969 Servicio de Computación Avanzada - Advanced Computing Service advancedcomputing.ifca.es Instituto de Física de Cantabria (IFCA) | CSIC · UC Ed. Juan Jordá, Avda. de los Castros s/n - 39005 Santander (Spain) www.ifca.es --- --- --- Políticas del Servicio de Computación Avanzada disponibles en / Advanced Computing Service policies available at : https://confluence.ifca.es/x/HAAt Este mensaje, y los ficheros adjuntos que pueda incluir, puede contener información confidencial o legalmente protegida y está exclusivamente dirigido a la persona o entidad destinataria. Si usted no consta como destinatario final ni es la persona encargada de recibirlo, no está autorizado a leerlo, retenerlo, modificarlo, distribuirlo o copiarlo, ni a revelar su contenido. Si lo ha recibido por error, informe de ello al remitente y elimine del sistema tanto el mensaje como los ficheros adjuntos que pueda contener. This email message and any attachments it carries may contain confidential or legally protected material and are intended solely for the individual or organization to whom they are addressed. If you are not the intended recipient of this message or the person responsible for processing it, then you are not authorized to read, save, modify, send, copy or disclose any part of it. If you have received the message by mistake, please inform the sender of this and eliminate the message and any attachments it carries from your account.

2 2

How to check if a used disk/SSD is ready to become an OSD?
by filip Mutterer 21 Oct '25

21 Oct '25

Hi! How to check if a used disk/SSD is ready to become an OSD? The thing is I don't know for sure if I prepared the disks already to become an OSD. Greetings filip

2 1

OSDs not balancing
by Ryan Sleeth 21 Oct '25

21 Oct '25

Hello, I am new to Ceph and still learning. I setup a test cluster and copied a lot of 'torture' data to it (20 TB with many files under 4KB). I copied this data to different pools configured for different compression plugins (snappy vs. lz4 vs. zstd), required ratios (default 87 vs. 97), and EC plugins (jerasure vs. isa). All totaled, the pools add up to ~440 TB (out of 1.3 PB) all using EC 4+2. Perhaps the autobalancer is misbehaving because many OSDs were 'stuck' at 1 PG until I manually specified them to be 256--now they are catching up. What is not showing any progress is: 1) I have one HDD OSD at 87% full, the next highest at 69%, and the lowest at 12% and 2) a huge number of PGs not deep-scrubbed in time; probably can ignore this as its a result of the dramatic PG changes The cluster is 9x Dell PowerEdge R730xd each with 8x 20 TB HDDs and 2x 2 TB enterprise NVMes configured for db; setup on Ubuntu 22 with Tentacle 20.1.0 RC0 upgraded from Squid and running with Docker/cephadm. Perhaps I am being impatient waiting for it to rebalance? Does the community have any suggestions about how to approach this? PG autobalancer seems fine. cluster: id: mypool health: HEALTH_WARN 1 nearfull osd(s) 1392 pgs not deep-scrubbed in time 217 pgs not scrubbed in time 21 pool(s) nearfull services: mon: 5 daemons, quorum ceph01,ceph02,ceph03,ceph06,ceph05 (age 2w) [leader: ceph01] mgr: ceph01.qfwdee(active, since 2w), standbys: ceph02.imozkz mds: 2/2 daemons up, 1 standby osd: 90 osds: 90 up (since 2w), 90 in (since 3w); 863 remapped pgs data: volumes: 2/2 healthy pools: 25 pools, 3051 pgs objects: 396.39M objects, 326 TiB usage: 440 TiB used, 896 TiB / 1.3 PiB avail pgs: 428953812/1897506717 objects misplaced (22.606%) 2159 active+clean 861 active+remapped+backfill_wait 16 active+clean+scrubbing+deep 13 active+clean+scrubbing 2 active+remapped+backfilling io: recovery: 29 MiB/s, 21 objects/s -- Ryan Sleeth

2 2

What does logrotate configs?
by lejeczek 17 Oct '25

17 Oct '25

Hi guys. In a containerized env what creates: /etc/logrotate.d/cephadm but also other ceph's logorates? That one above seems to be watched constantly for if removed is recreated almost immediately. It also causes OS' 'lograte' to fail for its content conflict with other ceph-created logrotate files. btw. anybody sees this/similar? many thanks, L.

2 1

DriveGroup Spec question
by Robert Sander 17 Oct '25

17 Oct '25

Hi, we have OSD nodes currently consisting of two 605GB SSDs and six 18TB HDDs. The hosts have room for twelve HDDs. We created a drivegroup spec that looks like this: spec: block_db_size: 100GB data_devices: rotational: true size: '18TB:' db_devices: rotational: false size: '550GB:650GB' db_slots: 6 Initially this creates 6 OSDs with their RocksDB+WAL on the SSDs, 3 each which is nice for load balancing. But when we add another HDD it gets a 17.9TB data volume and a 100GB DB volume, both on the HDD: sdm 8:192 0 18T 0 disk ├─ceph--846e1a59--aff6--4ef8--9b71--de7241531677-osd--block--026e8cef--123d--47d9--9b30--211f94edf96c 252:16 0 17.9T 0 lvm └─ceph--846e1a59--aff6--4ef8--9b71--de7241531677-osd--db--88c47d0b--f5c6--4cec--8909--c5f8036ca459 252:17 0 100G 0 lvm I would have assumed that the remaining 305GB on the SSDs would be used. How do we achieve this? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 9

CSC Meeting Notes - 09/29
by Venky Shankar 16 Oct '25

16 Oct '25

- [Ernesto] SCM attack prevention Ceph org repos: request to pin all GH Action deps to SHA-1 commits (instead of version tags). - To decide: deadline (proposed 1 month, end of October) - More context: https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/5OJIR3OI2KES3L3QNW… - https://github.com/ceph/ceph/pull/65720 - [Yuri] Tentacle RC1 status: https://tracker.ceph.com/issues/73123 - [Yuri] Approval in process for many - Yuri to follow up individually with tech leads - DONE - LRC upgrades? - [Neha] should plan for an LRC upgrade since this is the last RC - [Yuri] would RC be GA? - [Neha] RC1 would be GA and would _not_ be a full rebase - [Neha] Reassess next RC if we get community feedback for RC1 - [Yuri] - bookworm: issue fixed but not easy to test (build only) - [David] No teuthology runs for Debian, so build only! - noble support? - [Casey] lack of test coverage for qa suites and will not happen for "T" - [Adam King] https://github.com/ceph/ceph/blob/tentacle/src/cephadm/cephadmlib/constants… - https://github.com/ceph/ceph/pull/65719 - [Neha] Start working on Release Notes - Neha and Anthony D to collaborate on slack (ceph-upstream-release channel) - Lab Update - [David] status: probably a month to get a lab running - sepia lab shutdown is likely to be December '25 - target "T" GA before hardware move - [Patrick] Election? - Anthony M is taking a lead on this - Anthony M to share updates in CSC channel (or in CSC meeting) -- Cheers, Venky

3 2

tls certs per manager - does it work?
by lejeczek 16 Oct '25

16 Oct '25

Hi guys. I thought I had a problem with restarting dashboard - following in docs on ssl certs. But ! I might have other issue - restarting dashboard does what is expected of - but, with these: -> $ ceph dashboard set-ssl-certificate-key podster2.mine.priv -i /root/podster2.mine.priv.key -> $ ceph dashboard set-ssl-certificate podster2.mine.priv -i /root/podster2.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key podster1.mine.priv -i /root/podster1.mine.priv.key -> $ ceph dashboard set-ssl-certificate podster1.mine.priv -i /root/podster1.mine.priv.crt If I go to podster2 I get podster1's cert, perhaps these do not do anything, If I do: -> $ ceph dashboard set-ssl-certificate -i /root/podster2.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key -i /root/podster2.mine.priv.key then podster1 has podster2's cert - naturally - but (weirdly?) if go to podster2 (by FQDN) then I get redirected to IP (as in URL) of when I do: -> $ ceph dashboard set-ssl-certificate -i /root/podster1.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key -i /root/podster1.mine.priv.key and I go to podster2 then, no redirection, "only" warning about cert being of podster1's, podster1 has a good cert! If I do (again) - having podster1 have "correct" cert, with no per-node/manager cert - per-node/mgr cert, as shown earlier, then.. again, podster1 has its correct cert, podster2 has podster1's cert (no ! redirection to IP though). How much of an "issue" this might be - given the fact that I deployed anew cluster twice (lab) - and this reproduces each time. Or, I'm missing some obvious, big picture - and it's simply bad luck, deploying cluster twice. ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) all thoughts are much appreciated. many thanks, L.

2 9

CephFS mount error after successful attach of volume
by Martin Reid 15 Oct '25

15 Oct '25

Hey everyone, I've got a difficult problem with my CephFS that I haven't been able to make any headway with. Maybe you guys can help? The problem I’m trying to provision a volume on a CephFS, using a Ceph cluster installed on Kubernetes (K3s) using Rook, but I’m running into the following error (from the Events in |kubectl describe|: |Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 4m24s default-scheduler Successfully assigned archie/ceph-loader-7989b64fb5-m8ph6 to archie Normal SuccessfulAttachVolume 4m24s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-95b6ca46-cf51-4e58-9bb5-114f00aa4267" Warning FailedMount 3m18s kubelet MountVolume.MountDevice failed for volume "pvc-95b6ca46-cf51-4e58-9bb5-114f00aa4267" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph csi-cephfs-node.1@039a3dba-d55c-476f-90f0-8783a18338aa.main-ceph-fs=/volumes/csi/csi-vol-25d616f5-918f-4e15-bfd6-55b866f9aa9f/4bda56a4-5088-451c-90c8-baa83317d5a5 /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/3e10b46e93bcc2c4d3d1b343af01ee628c736ffee7e562e99d478bc397dab10d/globalmount -o mon_addr=10.43.233.111:3300/10.43.237.205:3300/10.43.39.81:3300,secretfile=/tmp/csi/keys/keyfile-2996214224,_netdev] stderr: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized| I’m kind of new to K8s, and /very/ new to Ceph, so I would love some advice on how to go about debugging this mess. General context *Kubernetes distribution*: K3s *Kubernetes version(s)*: v1.33.4+k3s1 (master), v1.32.7+k3s1 (workers) *Ceph*: installed via Rook *Nodes*: 3 *OS*: Linux (Arch on master, NixOS on workers) What I’ve checked/tried *Note*: Since this is a Rook deployment of Ceph (on Kubernetes), all these checks are performed in the Rook Toolbox <https://rook.io/docs/rook/latest-release/Troubleshooting/ceph-toolbox/> container. MDS status / Ceph cluster health Even I know this is the first go-to when your Ceph cluster is giving you issues. I have the Rook toolbox <https://rook.io/docs/rook/latest-release/Troubleshooting/ceph-toolbox/> running on my K8s cluster, so I went into the toolbox pod and ran: |$ ceph status cluster: id: 039a3dba-d55c-476f-90f0-8783a18338aa health: HEALTH_OK| services: mon: 3 daemons, quorum a,c,b (age 7d) mgr: b(active, since 7d), standbys: a mds: 1/1 daemons up, 1 hot standby osd: 3 osds: 3 up (since 7d), 3 in (since 2w) data: volumes: 1/1 healthy pools: 4 pools, 81 pgs objects: 47 objects, 3.2 MiB usage: 139 MiB used, 502 GiB / 502 GiB avail pgs: 81 active+clean io: client: 1.2 KiB/s rd, 2 op/s rd, 0 op/s wr Since the error we started out with |mount error: no mds (Metadata Server) is up|, I checked the |ceph status| output above for the status of the metadata server. As you can see, all the MDS instances are running. CephFS Status |$ ceph fs status main-ceph-fs - 0 clients ============ RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active main-ceph-fs-a Reqs: 0 /s 143 38 37 0 0-s standby-replay main-ceph-fs-b Evts: 0 /s 159 30 29 0 POOL TYPE USED AVAIL main-ceph-fs-metadata metadata 4176k 158G main-ceph-fs-replicated data 0 158G main-ceph-fs-main-ceph-fs-replicated data 0 158G STANDBY MDS main-ceph-fs-d main-ceph-fs-c MDS version: ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)| Ceph authorizations for MDS Since the other part of the error indicated that I might not be authorized, I wanted to check what the authorizations were: |$ ceph auth ls mds.main-ceph-fs-a # main MDS for my CephFS instance key: <base64 key> caps: [mds] allow caps: [mon] allow profile mds caps: [osd] allow * mds.main-ceph-fs-b # standby MDS for my CephFS instance key: <different base64 key> caps: [mds] allow caps: [mon] allow profile mds caps: [osd] allow * ... client.csi-cephfs-node.1 # the client mentioned in the error message key: <another base64 key> caps: [mds] allow rw caps: [mgr] allow rw caps: [mon] allow r caps: [osd] allow rwx tag cephfs metadata=*, allow rw tag cephfs data=* ... # more after this| Note: |main-ceph-fs| is the name I gave my CephFS file system. It looks like this should be okay, but I’m not sure. Definitely open to some more insight here. PersistentVolumeClaim binding I checked to make sure the PersistentVolume was provisioned successfully from the PersistentVolumeClaim, and that it bound appropriately: |$ kubectl get pvc -n archie jellyfin-ceph-pvc NAME STATUS VOLUME CAPACITY jellyfin-ceph-pvc Bound pvc-95b6ca46-cf51-4e58-9bb5-114f00aa4267 180Gi | Changing the PVC size to something smaller I tried changing the PVC’s size from 180GB to 1GB, since I thought it might be a free space issue,but the error persisted. Turning off firewalls Turned off all firewall to see if it was that, and still no luck. I’m not quite sure where to go from here. What am I missing? What context should I add? What should I try? What should I check? Thank you so much in advance, Martin

2 1

Critical cluster issues after upgrade from Ceph 15.2.17 to 17.2.7 - OSDs failing with "missing pg_pool_t for deleted pool" error
by kefu chai 15 Oct '25

15 Oct '25

Hello Ceph community, I'm writing on behalf of a friend who is experiencing a critical cluster issue after upgrading and would appreciate any assistance. Environment: - 5 MON nodes, 2 MGR nodes, 40 OSD servers (306 OSDs total) - OS: CentOS 8.2 upgraded to 8.4 - Ceph: 15.2.17 upgraded to 17.2.7 - Upgrade method: yum update in rolling batches Timeline: The upgrade started on October 8th at 1:00 PM. We upgraded MON/MGR servers first, and then upgraded OSD nodes in batches of 5 nodes. The process appeared normal initially, but when approximately 10 OSD servers remained, OSDs began going down. MON Quorum Issue: When the OSDs began failing, the monitors failed to form a quorum. In an attempt to recover, we stopped 4 out of 5 monitors. However, the remaining monitor (mbjson20010) then failed to start due to a missing .ldb file. We eventually recovered this single monitor from OSD using the instructions at https://docs.ceph.com/en/quincy/rados/troubleshooting/troubleshooting-mon/#…, so we now have only 1 MON in the cluster instead of the original 5. However, rebuilding the MON store did not help, and restarting the OSD servers also failed to resolve the issue. The cluster status remains problematic. Current Cluster Status: - Only 1 MON daemon active (quorum: mbjson20010) - down from 5 MONs - OSDs: 91 up / 229 in (out of 306 total) - 88.872% of PGs are not active - 4.779% of PGs are unknown - 3,918 PGs down - 1,311 PGs stale+down - Only 12 PGs active+clean Critical Error: When examining OSD logs, we discovered that some OSDs are failing to start with the following error: osd.43 39677784 init missing pg_pool_t for deleted pool 9 for pg 9.3ds3; please downgrade to luminous and allow pg deletion to complete before upgrading Full error context from one of the failing OSDs: # tail /var/log/ceph/ceph-osd.43.log -7> 2025-10-12T13:40:05.987+0800 7fdd13259540 1 bluestore(/var/lib/ceph/osd/ceph-43) _upgrade_super from 4, latest 4 -6> 2025-10-12T13:40:05.987+0800 7fdd13259540 1 bluestore(/var/lib/ceph/osd/ceph-43) _upgrade_super done -5> 2025-10-12T13:40:05.987+0800 7fdd13259540 2 osd.43 0 journal looks like ssd -4> 2025-10-12T13:40:05.987+0800 7fdd13259540 2 osd.43 0 boot -3> 2025-10-12T13:40:05.987+0800 7fdceb2cc700 5 bluestore.MempoolThread(0x55c7b0c66b40) _resize_shards cache_size: 8589934592 kv_alloc: 1717986918 kv_used: 91136 kv_onode_alloc: 343597383 kv_onode_used: 23328 meta_alloc: 6871947673 meta_used: 2984 data_alloc: 0 data_used: 0 -2> 2025-10-12T13:40:05.989+0800 7fdd13259540 -1 osd.43 39677784 init missing pg_pool_t for deleted pool 9 for pg 9.3ds3; please downgrade to luminous and allow pg deletion to complete before upgrading -1> 2025-10-12T13:40:05.991+0800 7fdd13259540 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/osd/OSD.cc: In function 'int OSD::init()' thread 7fdd13259540 time 2025-10-12T13:40:05.990845+0800 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/osd/OSD.cc: 3735: ceph_abort_msg("abort() called") # tail /var/log/ceph/ceph-osd.51.log -7> 2025-10-12T13:39:36.739+0800 7f603e5f7540 1 bluestore(/var/lib/ceph/osd/ceph-51) _upgrade_super from 4, latest 4 -6> 2025-10-12T13:39:36.739+0800 7f603e5f7540 1 bluestore(/var/lib/ceph/osd/ceph-51) _upgrade_super done -5> 2025-10-12T13:39:36.739+0800 7f603e5f7540 2 osd.51 0 journal looks like ssd -4> 2025-10-12T13:39:36.739+0800 7f603e5f7540 2 osd.51 0 boot -3> 2025-10-12T13:39:36.739+0800 7f6016669700 5 bluestore.MempoolThread(0x55e839d4cb40) _resize_shards cache_size: 8589934592 kv_alloc: 1717986918 kv_used: 31232 kv_onode_alloc: 343597383 kv_onode_used: 21584 meta_alloc: 6871947673 meta_used: 1168 data_alloc: 0 data_used: 0 -2> 2025-10-12T13:39:36.741+0800 7f603e5f7540 -1 osd.51 39677784 init missing pg_pool_t for deleted pool 6 for pg 6.1f; please downgrade to luminous and allow pg deletion to complete before upgrading -1> 2025-10-12T13:39:36.742+0800 7f603e5f7540 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/osd/OSD.cc: In function 'int OSD::init()' thread 7f603e5f7540 time 2025-10-12T13:39:36.742527+0800 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/osd/OSD.cc: 3735: ceph_abort_msg("abort() called") Investigation Findings: We examined all OSD instances that failed to start. All of them exhibit the same error pattern in their logs and all contain PG references to non-existent pools. For example, running "ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-51 --op list-pgs" shows PG references to pools that no longer exist (e.g., pool 9, pool 10, pool 4, pool 6, pool 8), while the current pools are numbered 101, 140, 141, 149, 212, 213, 216, 217, 218, 219. Notably, each affected OSD contains only 2-3 PGs referencing these non-existent pools, which is significantly fewer than the hundreds of PGs a regular OSD typically contains. It appears the OSD metadata has been corrupted or overwritten with stale references to deleted pools from previous operations, preventing these OSDs from starting and causing widespread PG state abnormalities across the cluster. 2 PGs referencing non-existent pools were found in osd.51: # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-51 --op list-pgs 1.0 6.1f # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-51 --op list Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available Error getting attr on : 6.1f_head,#6:f8000000::::head#, (61) No data available ["1.0",{"oid":"","key":"","snapid":-2,"hash":0,"max":0,"pool":1,"namespace":"","max":0}] ["1.0",{"oid":"main.db-journal.0000000000000000","key":"","snapid":-2,"hash":1969844440,"max":0,"pool":1,"namespace":"devicehealth","max":0}] ["1.0",{"oid":"main.db.0000000000000000","key":"","snapid":-2,"hash":1315310604,"max":0,"pool":1,"namespace":"devicehealth","max":0}] ["6.1f",{"oid":"","key":"","snapid":-2,"hash":31,"max":0,"pool":6,"namespace":"","max":0}] We also performed a comprehensive check by listing all PGs from all OSD nodes using "ceph-objectstore-tool --op list-pgs" and comparing the results with the output of "ceph pg dump". This comparison revealed that quite a few PGs are missing from the OSD listings. We suspect that some OSDs that previously held these missing PGs may now be corrupted, which would explain both the missing PGs and the widespread cluster degradation. It appears the OSD metadata has been corrupted or overwritten with stale references to deleted pools from previous operations, preventing these OSDs from starting and causing widespread PG state abnormalities across the cluster. It appears the OSD objectstore's metadata has been corrupted or overwritten with stale references to deleted pools from previous operations, preventing these OSDs from starting and causing widespread PG state abnormalities across the cluster. Questions: 1. How can we safely restore the missing PGs from the OSD without data loss? 2. Has anyone encountered similar issues when upgrading from Octopus (15.2.x) to Quincy (17.2.x)? We understand that skipping major versions may not be officially supported, but we urgently need guidance on the safest recovery path at this point. Any help would be greatly appreciated. Thank you in advance. -- Regards Kefu Chai

3 6

Ceph Repository Signing Key and EL10
by Whitehouse, Dan 15 Oct '25

15 Oct '25

Hi, We are currently doing some testing of Rocky 10 and I have noticed that EL10 seems to have tighter standards for repository signing keys: Importing GPG key 0x460F3994: Userid : "" Fingerprint: 08B7 3419 AC32 B4E9 66C1 A330 E84A C2C0 460F 3994 From : https://download.ceph.com/keys/release.asc Is this ok [y/N]: y error: Certificate E84AC2C0460F3994: Policy rejects E84AC2C0460F3994: No binding signature at time 2025-10-15T09:43:58Z Key import failed (code 2). Failing package is: librados2-2:20.1.1-0.el10.x86_64 GPG Keys are configured as: https://download.ceph.com/keys/release.asc Many thanks, Dan Whitehouse

1 0

Announcing go-ceph v0.36.0
by John Mulligan 14 Oct '25

14 Oct '25

We're happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.36.0 Changes include additions to the rbd, rgw, and cephfs packages. More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project.

1 0

How to speed up the backfill on replicated pool?
by Jan Kasprzak 14 Oct '25

14 Oct '25

Hello, Ceph users, on my new cluster which I filled with testing data two weeks ago there are many repmapped PGs in backfill_wait state, probably as result of autoscaling the number of PGs per pool. But the recovery speed is quite low, in order of small MB/s and < 10 obj/s according to ceph -s. The cluster is otherwise idle, no client traffic after initial import, so I wonder why the backfill does not progress faster. Also, it seems like more pgs are getting remapped as existing ones get successfully backfilled - the percentage of misplaced objects is steadily around 6 % for the last two weeks. The PGs waiting for backfill all belong to the biggest pool I have according to "ceph pg dump | grep backfill", no surprise here. The pool has 229 TB of data and currently 128 PGs. It is replicated with k=4 m=2. The second biggest pool has only 23 TB of data: rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR pool_with_backfill 229 TiB 10086940 0 60521640 0 0 0 0 0 B 72545009 54 TiB 0 B 0 B second_biggest_pool 23 TiB 1153174 0 6919044 0 0 0 0 0 B 38506397 16 TiB 0 B 0 B [...] I tried to do "ceph osd pool force-backfill $pool", it helped to speed things up a bit, but it still runs at 50-200 MB/s and 4-20 obj/s. The initial data import ran at around 600 MB/s. Is it normal or can I speed the recovery up a bit somehow? Output of ceph -s: cluster: id: ... health: HEALTH_WARN 2 large omap objects services: mon: 3 daemons, quorum istor11,istor21,istor31 (age 13d) mgr: istor31(active, since 3w), standbys: istor21, istor11 osd: 36 osds: 36 up (since 2w), 36 in (since 3w); 14 remapped pgs data: pools: 45 pools, 1505 pgs objects: 13.39M objects, 198 TiB usage: 303 TiB used, 421 TiB / 724 TiB avail pgs: 5335074/80345832 objects misplaced (6.640%) 1449 active+clean 34 active+clean+scrubbing 11 active+remapped+backfill_wait+forced_backfill 8 active+clean+scrubbing+deep 2 active+remapped+forced_backfill 1 active+remapped+backfilling+forced_backfill io: recovery: 69 MiB/s, 4 objects/s The OSDs are HDD-based with metadata on NVMe, 4 OSDs per node, and all the nodes have load average somewhere between 0.3 and 0.6. Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

6 8

Ceph User+Dev October Meetup Cancelled
by Anthony Middleton 14 Oct '25

14 Oct '25

Hey friends, we're cancelling tomorrow's Ceph User+Dev Meetup to shift our focus to Cephalocon. Please feel free to share any thoughts in the thread.

1 0

Slow deep-scrubs
by Sven Grottke 13 Oct '25

13 Oct '25

Hi, we are operating a Ceph cluster with mostly replicated pools on SSDs, but one of our pools is erasure-coded on 16TB HDDs. Two weeks ago, we upgraded from Reef to Squid (19.2.3). Since then, there has been an ever growing number of PGs that have not been deep-scrubbed or scrubbed in time. All these PGs are located on the HDD pool. I would like to increase the scrubbing and deep-scrubbing intervals for the HDDs, but not for the SSDs. Is there any way to do that other than setting it on a per-OSD base, such as per device class or per sub-tree of the crush map? Thanks in advance for any advice! Ciao, Sven

4 5

ceph pop-up chat / water cooler / happy hour -- today at 12:05 Pacific / 21:05 Europe
by Dan van der Ster 10 Oct '25

10 Oct '25

Fellow Cephalopods, I'm hosting a spontaneous Ceph water cooler chat today at 12:05 Pacific / 21:05 Europe. Wanna look at some code together? Chat about osdmaps? Preview the Cephalocon agenda? ... Grab a sandwich or your beverage of choice and connect here: https://meet.jit.si/ephemeral-quorum Cheers, Dan -- Dan van der Ster Ceph Executive Council | CTO @ CLYSO https://clyso.com | dan.vanderster(a)clyso.com

1 0

IO500 SC25 list call for submission
by IO500 Committee 09 Oct '25

09 Oct '25

Call for Submission Stabilization Period: Sep 29th - Oct 3rd, 2025 Submission Deadline: Nov 3rd, 2025 AoE The IO500 will soon be accepting and encouraging submissions for the upcoming 17th semi-annual IO500 Production and Research lists, in conjunction with SC'25. Once again, we will also be accepting submissions to the 10 Client Node Challenge to encourage the submission of small scale results. View the requirements for submitting to each list on the IO500 Submissions Webpage. The new ranked lists will be announced at the "IO500: High Performance Storage Community" BoF. We hope to see many new results. The tag io500-sc25 should be used for submissions to this version of the benchmark list. Background Following the success of the Top500 in collecting and analyzing historical trends in supercomputer technology and evolution, the IO500 was created in 2017, published its first list at SC17, and has grown continually since then. The benchmarks represent community accepted standards, including being used in Request for Proposals for new HPC platforms. The benchmarks showcase the IO access pattern extremes giving a full picture of storage system potential performance. The list is about much more than just the raw rank; all submissions help the community by collecting and publishing a wider corpus of data. The multi-fold goals of the benchmark suite are as follows: Represent naive and optimized access patterns for the execution of a rich variety of HPC applications, their achievable performance, and the documentation of how the numbers are achieved. Support small to extreme-scale Research and Production HPC systems using flexible storage APIs Maximizing simplicity in running the benchmark suite Specifically, the benchmark suite includes a hero-run of both IOR and mdtest configured however possible to maximize performance and establish an upper-bound for performance. It also includes an IOR and mdtest run with highly prescribed parameters in an attempt to determine a lower performance bound. New in this release is a random 4KB read phase, to better understand the performance of non-sequential IO patterns used in AI workloads. Finally, it includes a namespace search as this has been determined to be a highly sought-after feature in HPC storage systems that has historically not been well-measured. Supported storage APIs are those that are part of IOR and mdtest. Extending these tools with a public pull request can be done to enable new storage APIs. The goals of the community are also multi-fold: Gather historical data for the sake of analysis and to aid predictions of storage futures Collect tuning information to share valuable performance optimizations across the community Encourage vendors and designers to optimize for workloads beyond "hero runs" Establish bounded expectations for users, procurers, and administrators Understand and be able to reproduce performance on storage systems Using the IO500 Reproducibility guidelines, each submission is labeled according to the breadth of details provided and the access to the deployed storage software that enables the community to reproduce the results and study system design changes over time. The IO500 follows a two-staged approach. First, there will be a two-week stabilization period during which we encourage the community to verify that the benchmark runs properly on a variety of storage systems. During this period the benchmark may be updated based upon feedback from the community. The final benchmark will then be released. We expect that runs compliant with the rules made during the stabilization period will be valid as a final submission unless a significant defect is found. 10 Client Node I/O Challenge The 10 Client Node Challenge is conducted using the regular IO500 benchmark, however, with the rule that exactly 10 client nodes must be used to run the benchmark. You may use any shared storage with any number of servers. We will announce the results in the Production and Research lists as well as in separate derived lists. Birds-of-a-Feather Once again, we encourage you to submit to join our community, and to attend the SC25 BoF where we will announce the new IO500 Production and Research lists and their 10 client node counterparts. Be Part of the Community Submissions of all sizes are welcome; the webpage has customizable sorting, so it is possible to submit on a small system and still get a very good per-client score, for example. We will also highlight new and interesting results with invited talk(s) at the BoF.

1 0

Re: Read-Only LUN
by Kardos László 09 Oct '25

09 Oct '25

Hello, Thanks for your response. We must use it currently. We would like to mount the block device as read only for Veeam block based backup. We would like to be make it Read-Only because of security reasons. We found the write_protection option under configFS, but the write_protection option getting overwritten time to time. Best Regards, Laszlo Kardos -----Original Message----- From: Anthony D'Atri <aad(a)dreamsnake.net> Sent: Thursday, October 9, 2025 3:46 PM To: Kardos László <laszlo.kardos(a)acetelecom.hu> Subject: Re: [ceph-users] Read-Only LUN Don't use the iSCSI gateway unless you really really have to. It is not maintained. The NVMeoF gateway is a better idea. Are you providing storage to VMware? > On Oct 9, 2025, at 9:03 AM, Kardos László <laszlo.kardos(a)acetelecom.hu> > wrote: > > Hello, > > > > I would like to ask whether there is a possibility to configure a LUN > and iSCSI initiator to be Read-Only? > > > > > > Best Regards, > > Laszlo Kardos > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io

1 0

Read-Only LUN
by Kardos László 09 Oct '25

09 Oct '25

Hello, I would like to ask whether there is a possibility to configure a LUN and iSCSI initiator to be Read-Only? Best Regards, Laszlo Kardos

1 0

Ceph PG stuck at degraded and very slow
by Sa Pham 09 Oct '25

09 Oct '25

*Hello everyone,* I’m running a Ceph cluster used as an RGW backend, and I’m facing an issue with one particular placement group (PG). - Accessing objects from this PG is *extremely slow*. - Even running ceph pg <pg_id> takes a very long time. - The PG is currently *stuck in a degraded state*, so I’m unable to move it to other OSDs. Current ceph version is reef 18.2.7. Has anyone encountered a similar issue before or have any suggestions on how to troubleshoot and resolve it? Thanks in advance!

5 11

Orphaned CephFS objects
by Trey Palmer 08 Oct '25

08 Oct '25

Hi, Some months ago we deleted about 1.4PB net of CephFS data. Approximately 600TB net of the space we expected to reclaim, did not get reclaimed. The cluster in question is now on Reef, but it was on Pacific when this happened. The cluster has 660 15TB NVMe OSD's on 55 nodes, plus 5 separate mon nodes. We almost exclusively use it for CephFS, though there is a small RBD pool for VM images. At the time we had a single MDS for the entire cluster, but we are now breaking that CephFS instance apart into multiple instances to spread the load (we tried multiple MDS ranks but it regressed performance and caused a near-disaster). Anyway, we often delete ~500TB at a time and we haven't previously run into this orphaned object problem. At least, not at a scale big enough for us to notice -- generally I've gotten back the space I expected to free up. Also we have a 30PB raw spinning disk cluster running Quincy that we deleted 2 PB net from at the same time, and it didn't exhibit this problem. So my assumption is that there is a bug in the version we were running that we exposed by deleting so much data at once. Our support vendor did some digging using some special utilities and found that there are a bunch of orphaned objects from 2 particular snapshots, where the MDS has no reference to the objects, but they still exist on disk. They feel that it would be somewhat difficult and risky to try to find and delete these objects via rados. However, obviously, 600+ *net* TB of all-NVMe storage is quite a lot of money to just let go to waste. That's effectively over 1.1 PB, once you figure EC overhead and the need to not fill the cluster over about 70%. So to reclaim the pool I have created a plan to move everything off of the pool, and then delete the pool. Which we are largely needing to do anyway, due to splitting to multiple CephFS instances. My question is, am I likely to run into problems with this? For example, will I be able to do 'ceph fs rm_data_pool' once there are no longer any objects associated with the CephFS instance on the pool, or will the MDS have ghost object records that cause the command to balk? Thanks a lot for any insight, Trey Palmer

7 15

v20.1.1 Tentacle RC1 released
by Yuri Weinstein 07 Oct '25

07 Oct '25

This is the second release candidate for Tentacle. Ubuntu 22.04, 24.04 and CentOS 9 and Rocky 10 were built for this RC0 Feature highlights: * RADOS: New features include long expected performance optimizations (FastEC) for EC pools, including partial reads and partial writes. There is a new implementation of the Erasure Coding I/O code which provides substantial performance improvements and some capacity improvements. The new code is designed to optimize performance when using Erasure Coding with block storage (RBD) and file storage (CephFS) but will have some benefits for object (RGW) storage, in particular when using smaller sized objects. A new flag allow_ec_optimizations needs to be set on each pool to switch to using the new code, existing pools can be upgraded once the OSD and MON daemons have been updated; there is no need to update the clients. The default erasure code plugin has been switched from Jerasure to ISA-L. Users can also expect to see BlueStore improvements such as compression and a new, faster WAL (write-ahead-log). Additional improvements include switching all components to the faster OMAP iteration interface, bypassable ceph_assert()s, fixed mclock bugs and configuration defaults, and testing improvements for dencoding verification. Data Availability Score: Users can now track data availability score for each pool in their cluster. This feature is currently in tech preview.A pool is considered unavailable if any PG in the pool is not in active state or if there are unfound objects. Otherwise the pool is considered available. The score is updated every one second by default. The feature is on by default. * MGR: Highlights include the ability to force-disable always-on modules and the removal of the restful and zabbix modules (both deprecated since 2020). Note that the dashboard module's richer and better-maintained RESTful API can be used as an alternative to the restful module, and the prometheus module can be used as an alternative monitoring solution for zabbix. * RGW: Multiple fixes: Lua scripts will not run against health checks, properly quoted ETag values returned by S3 CopyPart, PostObject and CompleteMultipartUpload responses. * RGW: IAM policy evaluation now supports conditions ArnEquals and ArnLike, along with their Not and IfExists variants. * RBD: New live migration features: RBD images can now be instantly imported from another Ceph cluster (native format) or from a wide variety of external sources/formats with the help of the new NBD stream and an appropriately capable NBD server such as `qemu-nbd`. Also added support for RBD namespace remapping while mirroring between Ceph clusters, new `rbd group info` and `rbd group snap info` commands and enhanced `rbd group snap ls` command. `rbd device map` command now defaults to msgr2. * CephFS: Directories may now be configured with case-insensitive or normalized directory entry names. This is an inheritable configuration making it apply to an entire directory tree. For more information, see https://docs.ceph.com/en/latest/cephfs/charmap/ * CephFS: Modifying the FS setting variable "max_mds" when a cluster is unhealthy now requires users to pass the confirmation flag (--yes-i-really-mean-it). This has been added as a precaution to tell the users that modifying "max_mds" may not help with troubleshooting or recovery effort. Instead, it might further destabilize the cluster. * CephFS: EOPNOTSUPP (Operation not supported ) is now returned by the CephFS fuse client for `fallocate` for the default case (i.e. mode == 0) since CephFS does not support disk space reservation. The only flags supported are `FALLOC_FL_KEEP_SIZE` and `FALLOC_FL_PUNCH_HOLE`. * Dashboard: Added support for NVMe/TCP (gateway groups, multiple namespaces), multi-cluster management, oAuth2 integration, and enhanced RGW/SMB features including multi-site automation, tiering, policies, lifecycles, notifications, and granular replication. * Integrated SMB support Ceph clusters now support an smb manager module that works similarly to the existing nfs subsystem. The new smb support allows the ceph cluster to automatically create Samba-backed SMB file shares connected to CephFS. The smb module can configure both basic Active Directory domain or standalone user authentication. The Ceph cluster can host one or more virtual smb cluster which can be truly clustered using Samba's CTDB technology. The smb module requires a cephadm enabled Ceph cluster and deploys container images provided by the samba-container project. The Ceph dashboard can be used to configure smb clusters and shares. A new cephfs-proxy daemon is automatically deployed to improve the scalability and memory usage when connecting Samba to CephFS. * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-20.1.1.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: dd9c546413d50a90668289255a256022ea21f0c0

1 0

Follow-up on Bug #68060: _deferred_replay can overwrite BlueFS
by Yuma Ogami 07 Oct '25

07 Oct '25

Hi, I'm writing to follow up on Bug #68060, "_deferred_replay can overwrite BlueFS". Link to the issue: https://tracker.ceph.com/issues/68060 This issue is marked as high priority and has the potential for major data corruption. I'm closely watching its progress. About a month ago, I posted a couple of questions in #note-26 on the tracker, but haven't received a response yet. I understand everyone is very busy, but I would be grateful for any updates on this matter. For convenience, I'm re-posting my questions here: Q1: The following my understanding correct? - If object data is corrupted: If there are other healthy replicas, the corruption will be detected and fixed on scrub or read. - If BlueFS is corrupted: The OSD would crash during operation or fail to start at next BlueFS's open process. Q2: And is there a new fix being developed after the following PR was reverted? https://github.com/ceph/ceph/pull/60753 Any information you could provide would be greatly appreciated. Thank you for your time and help. Best regards, Yuma Ogami

1 0

[RGW] radosgw-admin object stat is very slow on some object
by Sa Pham 06 Oct '25

06 Oct '25

Hi all, I'm running CEPH reef with 2 RGW instances. I have an issue with some object response time being very slow. I checked with command radosgw-admin object stat --bucket <bucket> --object <object> and it responses slow as well. How should I troubleshoot this issue? Thanks & Regards

2 2

Logging recommendation on docker/cephadm pacific
by J-P Methot 06 Oct '25

06 Oct '25

Hi, We've noticed that on some of our older setups (Pacific here), we're getting what appears to be duplicate logs for our ceph docker containers deployed with cephadm. These logs are located at "/var/lib/docker/containers/*containerid*-json.log", are generated by Docker and don't rotate by default. We're wondering what would be the best practice here. Would it be best to setup log rotation for these files or is it safe to just stop Docker from logging at all? -- Mes salutations distinguées | Best regards Jean-Philippe MÉTHOT (Il, Lui | He/Him) Administrateur Infra sénior | Senior Infra administrator PlanetHoster AVIS: Ce courriel est confidentiel et peut être l'objet de secret professionnel. Tout usage ou diffusion de celui-ci par une personne autre que son destinataire est illégal. Si vous recevez ce courriel par erreur, veuillez nous en aviser sans délai et le supprimer de votre système. This message is confidential. It may also be privileged or otherwise protected by work product immunity or other legal rules. If you have received it by mistake, please let us know by e-mail reply and delete it from your system.

2 1

ATTN: Potential data loss from 'radosgw-admin bucket rm --bypass-gc'
by Casey Bodley 03 Oct '25

03 Oct '25

A long-standing data loss bug with --bypass-gc was reported today in https://tracker.ceph.com/issues/73348. If any of the deleted objects had been copied to/from other buckets, --bypass-gc deletes the data of those copies too. As a result, the copies are still visible to ListObjects requests but GetObject requests fail with NoSuchKey. You can follow https://tracker.ceph.com/issues/73348 to track the fix and its backports. In the meantime, use 'radosgw-admin bucket rm' without --bypass-gc which correctly handles copied objects.

1 0

help OSD deploying via yaml
by GLE, Vivien 03 Oct '25

03 Oct '25

Hi, For testing purpose we need to deploy : - 1 pool of 6 SSD OSD - 1 pool of 6 HDD OSD - 1 pool of 6 HDD OSD with 2 SSD for DB+WAL I tried to orch apply this yaml but it doesn't work as expected Osd part of the yaml -> service_type: osd service_id: osd_spec placement: hosts: - host1 - host2 - host3 - host4 - host5 - host6 data_devices: paths: - /dev/sda - /dev/sdc spec: data_devices: all: true filter_logic: AND objectstore: bluestore --- service_type: osd service_id: osd_spec_wall placement: hosts: - host1 - host2 - host3 - host4 - host5 - host6 spec: data_devices: paths: - /dev/sdf db_devices: paths: - /dev/sde limit: 2 db_slots: 3 Only 1 db on /dev/sde from host1 has been created and this OSD showed up as half full at his creation: ceph osd df | grep 25 25 hdd 0.63669 1.00000 652 GiB 373 GiB 1.5 MiB 1 KiB 38 MiB 279 GiB 57.15 35.93 0 up ceph-volume lvm list ====== osd.25 ====== [block] /dev/ceph-23d9297a-d0e1-47be-8650-5c8ccae4fe0e/osd-block-2f009760-fc2b-46d5-984d-e8200dfd9d9d block device /dev/ceph-23d9297a-d0e1-47be-8650-5c8ccae4fe0e/osd-block-2f009760-fc2b-46d5-984d-e8200dfd9d9d block uuid vqJGf8-V5g0-S1cA-BAcN-Qm9D-9VTx-xsc8wk cephx lockbox secret cluster fsid id cluster name ceph crush device class db device /dev/ceph-9c39b87c-a39c-413f-b1ef-07881195fcb8/osd-db-feee5095-b5e7-47a0-ae87-8e5039512661 db uuid Rtj5KA-Qxjk-IFmY-3ffQ-gSAu-Snte-uku8HC encrypted 0 osd fsid 2f009760-fc2b-46d5-984d-e8200dfd9d9d osd id 25 osdspec affinity osd_spec_wall type block vdo 0 with tpm 0 devices /dev/sdf [db] /dev/ceph-9c39b87c-a39c-413f-b1ef-07881195fcb8/osd-db-feee5095-b5e7-47a0-ae87-8e5039512661 block device /dev/ceph-23d9297a-d0e1-47be-8650-5c8ccae4fe0e/osd-block-2f009760-fc2b-46d5-984d-e8200dfd9d9d block uuid vqJGf8-V5g0-S1cA-BAcN-Qm9D-9VTx-xsc8wk cephx lockbox secret cluster fsid id cluster name ceph crush device class db device /dev/ceph-9c39b87c-a39c-413f-b1ef-07881195fcb8/osd-db-feee5095-b5e7-47a0-ae87-8e5039512661 db uuid Rtj5KA-Qxjk-IFmY-3ffQ-gSAu-Snte-uku8HC encrypted 0 osd fsid 2f009760-fc2b-46d5-984d-e8200dfd9d9d osd id 25 osdspec affinity osd_spec_wall type db vdo 0 with tpm 0 devices /dev/sde The other /dev/sde device did show up as data_device instead of db_device (example here on host2) : ceph-volume lvm list ====== osd.17 ====== [block] /dev/ceph-629f98b0-5ed4-4e75-81b9-e85ca76afb15/osd-block-5d43d683-1f7f-4dc1-935e-6a79745252f9 block device /dev/ceph-629f98b0-5ed4-4e75-81b9-e85ca76afb15/osd-block-5d43d683-1f7f-4dc1-935e-6a79745252f9 block uuid HQpp1l-x7IB-kA2W-6gWO-BGlM-VN2k-vYf43R cephx lockbox secret cluster fsid id cluster name ceph crush device class encrypted 0 osd fsid 5d43d683-1f7f-4dc1-935e-6a79745252f9 osd id 17 osdspec affinity osd_spec type block vdo 0 with tpm 0 devices /dev/sde Thx for your help Vivien

3 8

tentacle 20.1.1 RC1 QE validation status
by Yuri Weinstein 03 Oct '25

03 Oct '25

Details of this release are summarized here: https://tracker.ceph.com/issues/73123#note-1 Release Notes - TBD LRC upgrade - TBD Dev Leads, please review the list of suites for completeness, as this is a new release. Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Adam King & Guillaume rbd: Ilya krbd: Ilya nvmeof: Aviv upgrade/squid-x: Laura upgrade/reef-x: Laura upgrade/telemetry-upgrade: PASSED upgrade/tentacle-p2p: N/A upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED perf-basic: PASSED crimson-rados: PASSED powercycle: PASSED TIA

8 13

Re: Ceph PG stuck at degraded and very slow
by Sa Pham 03 Oct '25

03 Oct '25

Hi Anthony, I dont see any error from dmesg log and smart of those OSDs Regards On Fri, 3 Oct 2025 at 07:08 Anthony D'Atri <aad(a)dreamsnake.net> wrote: > Check dmesg / var/log/messages on the hosts with that PG’s OSDs for > errors, and run a `smartctl -a` on each of those OSDs’ drives. > > > On Oct 2, 2025, at 7:55 PM, Sa Pham <saphi070(a)gmail.com> wrote: > > > > Hi Joshua, > > > > No. OSD is still responding. > > > > # ceph tell osd.130 version > > { > > "version": "18.2.7-0-g6b0e988052e", > > "release": "reef", > > "release_type": "stable" > > } > > > > But the primary OSD which includes slow pg (18.773) will respond slower. > > > > > > detailed as below > > > > > > # ceph pg dump_stuck degraded > > PG_STAT STATE UP > > UP_PRIMARY ACTING ACTING_PRIMARY > > 18.773 active+undersized+degraded+remapped+backfilling [302,150,138] > > 302 [130,101] 130 > > > > > > # time ceph tell osd.130 version > > { > > "version": "18.2.7-0-g6b0e988052e", > > "release": "reef", > > "release_type": "stable" > > } > > > > real 0m2.113s > > user 0m0.148s > > sys 0m0.036s > > # time ceph tell osd.101 version > > { > > "version": "18.2.7-0-g6b0e988052e", > > "release": "reef", > > "release_type": "stable" > > } > > > > real 0m0.192s > > user 0m0.152s > > sys 0m0.037s > > > > > > I don't know why. > > > > > > Regards, > > > > > > > > On Fri, Oct 3, 2025 at 4:21 AM Joshua Blanch <joshua.blanch(a)clyso.com> > > wrote: > > > >> could it be an OSD not responding? > >> > >> I would usually do > >> > >> ceph tell osd.* version > >> > >> To test if you can connect with osds > >> > >> > >> On Thu, Oct 2, 2025 at 12:09 PM Sa Pham <saphi070(a)gmail.com> wrote: > >> > >>> *Hello everyone,* > >>> > >>> I’m running a Ceph cluster used as an RGW backend, and I’m facing an > issue > >>> with one particular placement group (PG). > >>> > >>> > >>> - > >>> > >>> Accessing objects from this PG is *extremely slow*. > >>> - > >>> > >>> Even running ceph pg <pg_id> takes a very long time. > >>> - > >>> > >>> The PG is currently *stuck in a degraded state*, so I’m unable to > move > >>> it to other OSDs. > >>> > >>> > >>> Current ceph version is reef 18.2.7. > >>> > >>> Has anyone encountered a similar issue before or have any suggestions > on > >>> how to troubleshoot and resolve it? > >>> > >>> > >>> Thanks in advance! > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users(a)ceph.io > >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >>> > >> > > > > -- > > Sa Pham Dang > > Skype: great_bn > > Phone/Telegram: 0986.849.582 > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 1

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

SQUID ganesha failures
by Patrick Begou 02 Oct '25

02 Oct '25

Hi all, I have a Ceph Cluster using squid 19.2.2. I'm using CephFS storage with kernel mounts on clients and it works fine but for some older computers (still CentOS8 and one CentOS7 because it is running a commercial application) I need to use NFS so I've deployed Ganesha. But a few minutes after the first NFS client mount the volume (Just testing from the first client) and uses it, the Ceph NFS service falls in error on one node (not always the same node) and I have to reboot it. # ceph orch ps --daemon-type nfs NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID nfs.whitaker-nfs.0.2.whitaker04-ceph.itzyqf whitaker04-ceph *:12049 running (49m) 5m ago 3h 26.3M - 5.9 4892a7ef541b 42fd4ab7bba9 nfs.whitaker-nfs.1.0.whitaker05-ceph.vocxmr whitaker05-ceph *:12049 running (63m) 5m ago 23h 85.7M - 5.9 4892a7ef541b ab1c06be2ce7 nfs.whitaker-nfs.2.0.whitaker01-ceph.rhwuth whitaker01-ceph *:12049 running (33m) 5m ago 23h 24.6M - 5.9 4892a7ef541b eb93963f05cd nfs.whitaker-nfs.3.0.whitaker03-ceph.wqfnnq whitaker03-ceph *:12049 error 5m ago 23h - - <unknown> <unknown> <unknown> nfs.whitaker-nfs.4.0.whitaker02-ceph.wjynvn whitaker02-ceph *:12049 running (49m) 4m ago 3h 28.3M - 5.9 4892a7ef541b 5cb71f9d8b89 and the client freeze on the mount point. Any idea about this or how to track the problem ? Thanks Patrick ---------------------------------------------- Details of my NFS CEPH cluster, building commands and status: ---------------------------------------------- # *ceph nfs cluster create whitaker-nfs \ whitaker01-ceph,whitaker02-ceph,whitaker03-ceph,whitaker04-ceph,whitaker05-ceph \ --ingress --virtual_ip 172.30.1.102/24 --ingress-mode haproxy-protocol* *# ceph nfs cluster info whitaker-nfs* { "whitaker-nfs": { "backend": [ { "hostname": "whitaker01-ceph", "ip": "172.30.1.97", "port": 12049 }, { "hostname": "whitaker02-ceph", "ip": "172.30.1.98", "port": 12049 }, { "hostname": "whitaker03-ceph", "ip": "172.30.1.99", "port": 12049 }, { "hostname": "whitaker04-ceph", "ip": "172.30.1.100", "port": 12049 }, { "hostname": "whitaker05-ceph", "ip": "172.30.1.101", "port": 12049 } ], "monitor_port": 9049, "port": 2049, "virtual_ip": "172.30.1.102" } } *# ceph nfs export create cephfs \ --cluster-id whitaker-nfs \ --pseudo-path /data \ --fsname data \ --client_addr 172.30.1.0/24* *# ceph nfs export ls whitaker-nfs --detailed* [ { "access_type": "none", "clients": [ { "access_type": "rw", "addresses": [ "172.30.1.0/24" ], "squash": "none" } ], "cluster_id": "whitaker-nfs", "export_id": 1, "fsal": { "cmount_path": "/", "fs_name": "data", "name": "CEPH", "user_id": "nfs.whitaker-nfs.data.79da3503" }, "path": "/", "protocols": [ 4 ], "pseudo": "/data", "security_label": true, "squash": "none", "transports": [ "TCP" ] } ]

3 6