- ceph-users - lists.ceph.io

demons being stray?
by lejeczek 04 Nov '25

04 Nov '25

Hi guys. Is there a way to "clean" those up, both orderly & not manners would be ok. -> $ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1.02539 root default -3 0.34180 host podster1 9 hdd 0.04880 osd.9 up 1.00000 1.00000 10 hdd 0.29300 osd.10 up 1.00000 1.00000 -7 0.34180 host podster2 2 hdd 0.04880 osd.2 up 1.00000 1.00000 4 hdd 0.29300 osd.4 up 1.00000 1.00000 -5 0.34180 host podster3 1 hdd 0.04880 osd.1 up 1.00000 1.00000 5 hdd 0.29300 osd.5 up 1.00000 1.00000 0 0 osd.0 down 0 1.00000 3 0 osd.3 down 0 1.00000 6 0 osd.6 down 0 1.00000 7 0 osd.7 down 0 1.00000 8 0 osd.8 down 0 1.00000 ID CLASS WEIGHT TYPE NAME -1 1.02539 root default -3 0.34180 host podster1 9 hdd 0.04880 osd.9 10 hdd 0.29300 osd.10 -7 0.34180 host podster2 2 hdd 0.04880 osd.2 4 hdd 0.29300 osd.4 -5 0.34180 host podster3 1 hdd 0.04880 osd.1 5 hdd 0.29300 osd.5 -> $ ceph node ls ... "osd": { "podster1.mine.priv": [ 0, 3, 6, 7, 8, 9, 10 ], "podster2.mine.priv": [ 2, 4 ], "podster3.mine.priv": [ 1, 5 ] }, ... many thanks, L.

2 4

Mismatching Ceph package versions running alongside
by Demian Romeijn 04 Nov '25

04 Nov '25

Hi ceph-users, We're currently in the process of reviewing our current Ceph S3/FS cluster and were debating on adjusting Ceph versions. At this very moment we are running Ubuntu 22.04 with the official Ceph packages from download-ceph-com repository (Reef 18.2.4), however we'd like to change this out for Proxmox VE Ceph, primarily because of the extra bug-fixes & cherry picks that the regular Ceph packages sometimes lack. Also for the built-in support for ZFS which we'd like to change our boot-disks to. Our question at the moment is; 'Can we run these Ceph versions together, probably mismatched between different Reef versions (e.g. Ceph official 18.2.4 - PveCeph 18.2.7)'. Unfortunately we've not got any extra hardware laying around that we could prepare and simply `ceph-volume lvm activate all`, it has to be migrated node-per-node. Is this hypothetically possible to do without any major implecations? Thanks in advance! - Demian

1 0

Pool total sizes increase when deleteing RBDs.
by Konold, Martin 03 Nov '25

03 Nov '25

Hi, I experienced unexpected behaviour. When entirely deleting all rbds on a ceph pool (3/2) not only the used space decreased as expected but the total size of the pool increased significantly. IMHO it is caused by wrong calculations when osds are of slightly different sizes. Kind regards, --martin konold ppa. Martin Konold -- Martin Konold - Prokurist, CTO KONSEC GmbH -⁠ make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany

1 0

squid 19.2.3 - failed to probe daemon or devices - /var/lib/ceph read-only
by Steven Vacaroaia 31 Oct '25

31 Oct '25

Hi, I just noticed this error "failed to probe daemon or devices" which seems to be related to /var/lib/ceph being read only I checked the host ( ceph-host-1) and it seems fine so it must be a docker instance How do I identify which one and how can I fix it please ? Many thanks Steven root@ceph-host-1:~/utilities# ceph -s cluster: id: 0cfa836d-68b5-11f0-90bf-7cc2558e5ce8 health: HEALTH_WARN failed to probe daemons or devices services: mon: 5 daemons, quorum ceph-host-1,ceph-host-2,ceph-host-3,ceph-host-7,ceph-host-6 (age 8h) mgr: ceph-host-1.lqlece(active, since 5w), standbys: ceph-host-2.suiuxi mds: 19/19 daemons up, 7 standby osd: 185 osds: 185 up (since 4w), 185 in (since 4w) flags noautoscale rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 4/4 healthy pools: 16 pools, 7137 pgs objects: 359.48M objects, 811 TiB usage: 1.2 PiB used, 1.1 PiB / 2.3 PiB avail pgs: 7034 active+clean 72 active+clean+scrubbing 31 active+clean+scrubbing+deep io: client: 30 MiB/s rd, 34 KiB/s wr, 11 op/s rd, 1 op/s wr cephadm shell -- ceph-volume lvm list --format json Inferring fsid 0cfa836d-68b5-11f0-90bf-7cc2558e5ce8 Inferring config /var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/mon.ceph-host-1/config docker: Error response from daemon: error while creating mount source path '/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/crash': mkdir /var/lib/ceph: read-only file system. root@ceph-host-1:~/utilities# ceph health detail HEALTH_WARN failed to probe daemons or devices [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices host ceph-host-1 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/mon.ceph-host-1/config Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE= quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 -e NODE_NAME=ceph-host-1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/run/ceph:z -v /var/log/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/log/ceph:z -v /var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v /tmp/ceph-tmptx7tjpjz:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 inventory --format=json-pretty --filter-for-batch /usr/bin/docker: stderr docker: Error response from daemon: error while creating mount source path '/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/crash': mkdir /var/lib/ceph: read-only file system. Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/__main__.py", line 5581, in <module> File "/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/__main__.py", line 5569, in main File "/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/__main__.py", line 409, in _infer_config File "/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/__main__.py", line 324, in _infer_fsid File "/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/__main__.py", line 437, in _infer_image File "/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/__main__.py", line 311, in _validate_fsid File "/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/__main__.py", line 3314, in command_ceph_volume File "/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/cephadmlib/call_wrappers.py", line 307, in call_throws RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE= quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 -e NODE_NAME=ceph-host-1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/run/ceph:z -v /var/log/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/log/ceph:z -v /var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v /tmp/ceph-tmptx7tjpjz:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: error while creating mount source path '/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/crash': mkdir /var/lib/ceph: read-only file system.

1 2

Help with Jaeger Tracing for Ceph RGW in Kubernetes (Rook-Ceph Deployment)
by Ashok Kumar M 31 Oct '25

31 Oct '25

Hi Ceph Community, I'm currently running a Ceph cluster inside a Kubernetes environment using Rook-Ceph, and I'm working on enabling distributed tracing for Ceph RGW. As per the https://docs.ceph.com/en/latest/jaegertracing/ , I used the following docker run command to deploy a Jaeger all-in-one instance for testing: docker run -d --name jaeger \ -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \ -e COLLECTOR_OTLP_ENABLED=true \ -p 6799:6799/udp \ -p 6832:6832/udp \ -p 5778:5778 \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ -p 14250:14250 \ -p 14268:14268 \ -p 14269:14269 \ -p 9411:9411 \ jaegertracing/all-in-one:latest \ --processor.jaeger-compact.server-host-port=6799 However, when using the latest Jaeger version, I get the following error: unknown flag: --processor.jaeger-compact.server-host-port After researching, I discovered that this flag is only available in older versions of Jaeger (e.g., v1.27) and is no longer supported in Jaeger v2+. I then tried using a specific older Jaeger version (jaegertracing/all-in-one:1.27) that supports the flag. The container runs fine and listens on port 6799 as expected, but I still don’t see any RGW traces appearing in the Jaeger UI. My Setup: Ceph cluster deployed via Rook in Kubernetes Ceph RGW is running as a pod inside the cluster Jaeger all-in-one is running externally (for testing) Jaeger port 6799/udp is exposed and forwarded Tracing is enabled in Ceph config Questions: Could the issue be related to the Ceph RGW pod not being able to send UDP traces to the Jaeger container? Is there a specific configuration needed inside the RGW pod or Rook-Ceph to make sure tracing data is being sent correctly? Should I consider using the default Jaeger port 6831/udp instead of 6799/udp, and configure Ceph accordingly? Are there any known compatibility issues between Ceph RGW tracing and Jaeger v1.x? Could you suggest some troubleshooting steps to help verify whether traces are actually being sent from RGW and received by Jaeger? Any advice or working examples from a similar Kubernetes-based setup would be greatly appreciated. Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

2 2

CephFS object missing on disk.
by Leandro Ferrari 30 Oct '25

30 Oct '25

Hi Cephers, Recently, a split-brain occurred in my cluster and although it has recovered and is now healthy, when attempting to delete an empty directory in CephFS, the MDS server logs: [ERR] dir 0x100215b86b3.011* object missing on disk; some files may be lost ('[path to dir]') After this, the cluster enters in an error state with the message: [ERR] overall HEALTH_ERR 1 MDSs report damaged metadata I have already executed: $ceph tell mds.cephfs:0 scrub start [path to dir] recursive,repair,force As a response, I received information that some inodes, including the inode of the mentioned directory, were repaired and the cluster returned to a healthy state. [ERR] scrub: inode wrongly marked free: 0x100161edd3b [ERR] inode table repaired for inode: 0x100161edd3b [ERR] scrub: inode wrongly marked free: 0x1001c9b8246 [ERR] inode table repaired for inode: 0x1001c9b8246 ..... [ERR] scrub: inode wrongly marked free: 0x100017dc9a3 [ERR] inode table repaired for inode: 0x100017dc9a3 [ERR] scrub: inode wrongly marked free: 0x100113ef73a [ERR] inode table repaired for inode: 0x100113ef73a Part that informs that the directory inode was repaired [WRN] bad backtrace on inode 0x100215b86b3([path to dir]), rewriting it [ERR] scrub: inode wrongly marked free: 0x100215b86b3 [ERR] inode table repaired for inode: 0x100215b86b3 [INF] Scrub repaired inode 0x100215b86b3 [INF] Cluster is now healthy [INF] overall HEALTH_OK If I try to access the same directory again, the MDS server logs: [ERR] dir 0x100215b86b3.011* object missing on disk; Some files may be lost ('[path to dir]') Since the directory is empty, I don't mind losing it. How do I remove this information from MDS? Or what would be the recommended action? Ceph version: ceph version 17.2.8 (f817ceb7f187defb1d021d6328fa833eb8e943b3) quincy (stable) Thanks in advance, Leandro Ferrari

1 0

RGW objects stuck with user.rgw.amz-replication-status PENDING
by Adam Prycki 30 Oct '25

30 Oct '25

Hello, we are having issue with one of our multisite zones failing to synchronize. All replication shards are "busy", we have a lot of errors in `sync error list` on secondary zone. We are running on 19.2.3 (cephadm ubuntu 22.04) After some digging we have found that objects on our master zone have PENDING replication status For example: radosgw-admin object stat --bucket 7c52c9db-de61-4186-a3ce-e95a96df6be1 --object 0000d5d9-0a8c-40cf-88e9-30bf908c7743 "attrs": { "user.rgw.amz-replication-status": "PENDING", "user.rgw.content_type": "application/octet-stream", We have 3 zones in this zonegroup. Both secondary zones have this object replicated. "user.rgw.amz-replication-status": "REPLICA", But on the master zone it never got marked as "COMPLETED". Running data sync init on the bucket didn't help, we cannot fix this "PENDING" status on the ceph side without deleting and recreating affected bucket. Has anyone ever seen similar issue? My current theory is that something may have gone wrong during bucket resharding. I was running tests uploading small objects to a bucket and test was supposed to cause resharding mid benchmark. Best regards Adam Prycki

1 0

squid 19.2.3 - cephfs missing path
by Steven Vacaroaia 29 Oct '25

29 Oct '25

Hi, I have a cephfs subvolume ( out of 4) with " mount path missing " issue I think a user deleted it as it look different than regular data to him/her I am mounting at the group level so all subgroups are available under same path How concerned should I be ? Is there a way to recreate path on a cephfs subvolume ? what are my options for fixing it ? Please find below details Mounting /volumes/projects /projects/cephfs Cephfs status ceph fs status project_ec_ssd project_ec_ssd - 139 clients ============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active project_ec_ssd.ceph-host-5.henbit Reqs: 0 /s 6198k 6054k 1645k 107k 1 active project_ec_ssd.ceph-host-7.emayje Reqs: 0 /s 194 197 340 629 2 active project_ec_ssd.ceph-host-3.qebqlz Reqs: 0 /s 192 196 339 644 3 active project_ec_ssd.ceph-host-1.bwtdlt Reqs: 0 /s 196 199 338 729 4 active project_ec_ssd.ceph-host-4.xsoerf Reqs: 0 /s 196 200 339 93 POOL TYPE USED AVAIL metadata_fs_ssd metadata 18.0G 92.8T ssd_ec_project data 183T 241T ceph fs subvolumegroup ls project_ec_ssd [ { "name": "projects" } ] ceph fs subvolume ls project_ec_ssd --group_name projects [ { "name": "systems_archive" }, { "name": "arescore_moved" }, { "name": "alphacore_moved" }, { "name": "schematics" } ] ceph fs subvolume getpath project_ec_ssd alphacore_moved --group_name projects Error ENOENT: mount path missing for subvolume 'alphacore_moved' ceph fs subvolume getpath project_ec_ssd arescore_moved --group_name projects /volumes/projects/arescore_moved/ff576e35-2363-4e25-aa7a-e4a81693ab20 Many thanks Steven

3 3

Storage-ACK for writes after how many replica?
by jsterr＠proxforge.de 29 Oct '25

29 Oct '25

Hello Ceph-Users, I looked up in the docs but could not find a information about: * When does a storage-client get a write-acknowledgement Example: Replicated Pool 3/2 Will the write-ack come after reaching min-size or when all copys are written. Would be great if this could be added in the docs somewhere (maybe I did not find it - and its already there?). Thanks, Greetings Jonas

3 2

RGW Cascading Failures After Simultaneous Host Loss
by Ramin Najjarbashi 28 Oct '25

28 Oct '25

Hi We recently experienced an incident in one of our production Ceph clusters and would greatly appreciate the community’s input. Our configuration uses Erasure Coding with a 4+2 profile for the main S3 data pools. During the incident, two storage hosts that contained several OSDs each became unavailable simultaneously. As a result, a large number of PGs entered an inactive state because the number of remaining fragments was insufficient for reconstruction under the EC 4+2 profile. This triggered a cascading failure in the RGW service layer: • Incoming S3 requests continued flowing into HAProxy. • RGW attempted to process requests that required I/O to inactive PGs. • I/O stalls caused excessive queue buildup and memory usage on RGWs. • Multiple RGW processes were terminated by the OOM Killer. • RGW daemons repeatedly restarted and were again selected by HAProxy. • The cycle continued, leading to severe service disruption. This behavior continued until the failed hosts were recovered and PGs regained a clean and active state. One interesting observation was that a separate set of RGW daemons dedicated only to internal consumers continued operating normally. Their traffic load was significantly lower, which prevented queue saturation and memory exhaustion. We are seeking guidance on two key questions: 1. In an EC 4+2 pool design, is such a cascading RGW failure considered inevitable if enough hosts fail to cause PG unavailability, or are there established best practices to prevent this behavior? 2. Based on our observation that internal RGWs remained healthy, is isolating customers by assigning separate RGWs to high-traffic and low-traffic groups considered a recommended architectural approach? Thank you in advance for any advice and shared experiences. Best regards, Ramin

2 2