- ceph-users - lists.ceph.io

Code of Conduct
by Anthony Middleton 30 Jun '25

30 Jun '25

Hello Ceph Community, I'm reaching out regarding a recent dispute on our public mailing lists. In line with our commitment to uphold the Ceph Code of Conduct ( https://ceph.io/en/code-of-conduct/) I've started a resolution process in close collaboration with the Ceph Steering Committee and Ceph Executive Council. As part of this process, I have contacted those involved to understand their perspectives. The review is ongoing, and no final decisions have been made. I want to assure you that I am handling this matter with impartiality and care, guided by the principles and expectations outlined in our Code of Conduct. The Ceph Executive Council and I value every community member deeply. Our goal is not only to maintain a safe and respectful environment but also to foster a collaborative culture that inspires all of us to contribute toward our shared mission: advancing open-source storage for everyone's benefit. Thank you for your ongoing engagement and for being part of what makes the Ceph community strong. If you have thoughts or concerns you'd like to share, please feel free to contact me directly at community-manager(a)ceph.io. Best, Anthony Middleton Ceph Community Manager

1 0

Re-uploading existing s3 objects slower than initial upload
by Sinan Polat 30 Jun '25

30 Jun '25

Hi all, I am investigating a performance issue related to uploading s3 objects and would appreciate any insights. I have a bucket with ~200 million objects. I noticed that re-uploading files that already exist in the bucket appears to be slower than uploading them for the first time. To verify this behavior, I created a small test case with a new empty bucket: - 500 files with random sizes between 5 KB and 50 KB. - First time upload took 8 seconds: s3cmd put files/* s3://my-bucket/ - Second upload (same files, no changes) took 10 seconds I had expected the second upload to be faster or at least the same. Is this expected behavior? Any guidance on this would be appreciated. Thanks! Sinan

2 1

Get past epoch pg map
by Darrell Enns 30 Jun '25

30 Jun '25

Is there a way to get a pg dump for a previous epoch? I have a situation where I want to make sure all of the PGs that used to be on a specific OSD are deep scrubbed. However, since that OSD is now down and all the PGs have been relocated, I don't know what the PG numbers are. I know the epoch number I want to look at, and I can "ceph osd getmap <epoch>". But I'm not sure if there's a way to use this to get/view the pg map.

3 3

ceph squid - huge difference between capacity reported by "ceph -s" and "ceph df "
by Steven Vacaroaia 29 Jun '25

29 Jun '25

Hi, I just built a new CEPH squid cluster with 7 nodes Since this is brand new, there is no actuall data on it except few test files in the S3 data.bucket Why is "ceph -s" reporting 8 TB of used capacity ? Many thanks Steven [image: image.png]

2 2

Reef: RGW logrotate "breaks" log_to_file
by Eugen Block 27 Jun '25

27 Jun '25

Hi, after upgrading multiple clusters from 18.2.4. some weeks ago, I noticed that the RGWs stop logging to file after the nightly logrotate. Other daemons don't seem to be affected, they continue logging to file. Restarting an RGW daemon helps until the next logrotate. I could reproduce this in a lab cluster, after upgrading from 18.2.4 to 18.2.5, the daemon stops logging to file after a logrotate. Is this a known issue? I can't seem to find anything on tracker. Thanks! Eugen ceph config dump | grep log_to_file global basic log_to_file true global advanced mon_cluster_log_to_file true

5 10

cephfs + SELinux: .snap folder is unlabeled
by Fabian Wenzel 27 Jun '25

27 Jun '25

Hi everyone, when setting up ceph fs (using reef) on my system, I have noticed that there is an issue with an SELinux environment for an smb-gateway: SELinux requires samba shares to be of type samba_share_t. While this can be achieved with e.g. chcon on the mounted ceph fs (since it supports xattr), it is not possible to assign an SELinux type to the .snap subfolder, since it is read-only. Therefore, Is -lZ .snap returns a question mark, and samba access results in an access denied error to the type "unlabeled_t". While it is possible (as a workaround) to implement a rule that allows samba to access folders of "unlabeled_t", I think a proper solution would be to assign a label to the .snap folder as well. For samba, accessing the .snap folder is essential in order to enable "previous versions" functionality of the Windows Explorer with the samba shadow_copy2 vfs, as it can be found in a couple of youtube tutorials covering ceph fs + samba. How could this be done? Thanks, sophonet

2 1

First time configuration advice
by Ryan Sleeth 26 Jun '25

26 Jun '25

I am setting up my first cluster of 9-nodes each with 8x 20T HDDs and 2x 2T NVMes. I plan to partition the NVMes into 5x 300G so that one partition can be used by cephfs_metadata (SSD only), while the other 4x partitions will be paired as db devices for 4x of the HDDs. The cluster will only be used for cephfs and data will only be stored on its EC 4+2 HDD-only pool. Just a simple and large file server, so performance isn't a primary concern. Each node has 2x 10Gb network connections (one public, one cluster). All disks encrypted (encrypted=true on OSD creation on top of LVM). I am new to Ceph, and was hoping to get advice on if these initial configurations are sensible or if I am doing something incorrect that might have a negative impact in the future: https://pastebin.com/yiQS4g3q Thanks!

7 6

Incomplete PG's
by Devender Singh 26 Jun '25

26 Jun '25

Hello Seeking some help to recover these incomplete pgs… I made osd, down then tried but nothing seems working …. Ceph version 18.2.7, we have multiple disk failed in ceph cluster on different nodes… for i in 19.5a 19.5d 19.5e 19.6b 19.6f 19.74 19.7b 34.48 34.69; do ceph pg map $i | grep -q "acting.*1" && \ ceph-objectstore-tool --data-path /var/lib/ceph/15688cb4-044a-11ec-942e-516035adea04/osd.17 --op mark-complete --pgid $i --force done # for i in 19.5a 19.5d 19.5e 19.6b 19.6f 19.74 19.7b 34.48 34.69; do ceph pg map $i | grep -q "acting.*1" && \ ceph-objectstore-tool --data-path /var/lib/ceph/15688cb4-044a-11ec-942e-516035adea04/osd.17 --op repair --pgid $i --force done & pg 16.48 is incomplete, acting [20,9,17] pg 16.50 is incomplete, acting [20,27,5] pg 16.68 is incomplete, acting [2,5,27] pg 16.78 is incomplete, acting [23,14,29] pg 17.4f is incomplete, acting [10,30,18] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 17.7a is incomplete, acting [8,28,27] (reducing pool lv-r3-for-ec-large-disks min_size from 2 may help; search ceph.com/docs for 'incomplete') pg 19.44 is incomplete, acting [10,4,21,29,0] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.45 is stuck inactive since forever, current state incomplete, last acting [10,28,24,4,0] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.49 is incomplete, acting [28,10,26,0,15] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.4a is incomplete, acting [21,1,23,0,26] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.4b is incomplete, acting [6,12,17,30,7] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.4c is incomplete, acting [5,0,8,21,26] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.4f is incomplete, acting [26,14,5,7,9] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.51 is incomplete, acting [20,14,28,26,23] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.52 is incomplete, acting [10,26,22,30,7] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.53 is incomplete, acting [12,24,25,23,22] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.54 is incomplete, acting [10,4,20,15,11] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.57 is incomplete, acting [15,22,9,0,14] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.58 is incomplete, acting [15,0,12,28,20] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.59 is incomplete, acting [6,9,0,21,26] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.5a is incomplete, acting [11,31,20,17,24] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.5d is incomplete, acting [30,19,4,17,15] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.5e is incomplete, acting [3,9,10,17,16] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.5f is remapped+incomplete, acting [22] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.60 is incomplete, acting [25,27,23,4,22] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.61 is remapped+incomplete, acting [23] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.62 is remapped+incomplete, acting [23] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.63 is incomplete, acting [15,4,1,22,7] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.64 is incomplete, acting [4,7,5,29,31] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.65 is incomplete, acting [3,9,15,10,19] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.6b is remapped+incomplete, acting [1] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.6f is incomplete, acting [30,16,17,18,10] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.72 is remapped+incomplete, acting [19] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.73 is incomplete, acting [12,27,13,16,11] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.74 is incomplete, acting [17,18,19,4,29] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.75 is incomplete, acting [8,31,20,15,7] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.76 is incomplete, acting [5,26,18,30,21] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.78 is incomplete, acting [13,26,14,10,16] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.7a is incomplete, acting [15,1,16,12,6] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.7b is incomplete, acting [2,17,15,29,9] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.7c is incomplete, acting [28,19,6,4,26] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 19.7f is remapped+incomplete, acting [23] (reducing pool lvp-ec-large-disks min_size from 3 may help; search ceph.com/docs for 'incomplete') pg 22.4b is incomplete, acting [14,7,23] pg 22.51 is incomplete, acting [11,8,2] pg 22.5a is incomplete, acting [31,15,4] pg 34.48 is incomplete, acting [31,19,17] pg 34.51 is incomplete, acting [15,12,20] pg 34.65 is incomplete, acting [30,18,20] pg 34.6d is incomplete, acting [30,12,21] pg 34.7c is incomplete, acting [31,9,3] # ceph -s cluster: id: 15688cb4-044a-11ec-942e-516035adea04 health: HEALTH_ERR 4 OSD(s) experiencing slow operations in BlueStore 1 failed cephadm daemon(s) 1 filesystem is degraded 1 MDSs report slow metadata IOs 1/16510107 objects unfound (0.000%) Reduced data availability: 449 pgs inactive, 444 pgs incomplete Possible data damage: 1 pg recovery_unfound Degraded data redundancy: 3/60411723 objects degraded (0.000%), 1 pg degraded 343 pgs not deep-scrubbed in time 63 slow ops, oldest one blocked for 262107 sec, daemons [osd.1,osd.15,osd.16,osd.18,osd.19,osd.2,osd.21,osd.25,osd.26,osd.27]... have slow ops. services: mon: 7 daemons, quorum test-host04n,test-host05n,test-host02n,test-host03n,test-host06n,test-host07n,test-host08n (age 5h) mgr: test-host04n.azmqik(active, since 5h), standbys: test-host05n.lybhho mds: 3/3 daemons up, 11 standby osd: 32 osds: 32 up (since 18m), 32 in (since 18m); 36 remapped pgs rbd-mirror: 2 daemons active (2 hosts) data: volumes: 1/2 healthy, 1 recovering pools: 32 pools, 1505 pgs objects: 16.51M objects, 55 TiB usage: 155 TiB used, 246 TiB / 401 TiB avail pgs: 0.332% pgs unknown 29.502% pgs not active 3/60411723 objects degraded (0.000%) 1250071/60411723 objects misplaced (2.069%) 1/16510107 objects unfound (0.000%) 1022 active+clean 438 incomplete 16 active+remapped+backfilling 14 active+remapped+backfill_wait 6 remapped+incomplete 5 unknown 3 active+clean+scrubbing+deep 1 active+recovery_unfound+degraded io: recovery: 1.3 GiB/s, 345 objects/s progress: Global Recovery Event (4h) [===================.........] (remaining: 2h) Regards Dev

2 1

ceph health mute behavior
by Eugen Block 25 Jun '25

25 Jun '25

Hi, I'm trying to understand the "ceph health mute" behavior. In this case, I'm referring to the warning PG_NOT_DEEP_SCRUBBED. If you mute it for a week and the cluster continues deep-scrubbing, the "mute" will clear at some point although there are still PGs not deep-scrubbed in time warnings. I could verify this in a tiny lab with 19.2.2, setting osd_deep_scrub_interval to 10 minutes, the warning pops up. Then I mute that warning, issue deep-scrubs for several pools, and at some point I see this in the mon log: Jun 25 08:53:28 host1 ceph-mon[823315]: log_channel(cluster) log [WRN] : Health check update: 61 pgs not deep-scrubbed in time (PG_NOT_DEEP_SCRUBBED) Jun 25 08:53:28 host1 ceph-mon[823315]: Health check update: 61 pgs not deep-scrubbed in time (PG_NOT_DEEP_SCRUBBED) Jun 25 08:53:29 host1 ceph-mon[823315]: pgmap v164176: 389 pgs: 389 active+clean; 428 MiB data, 57 GiB used, 279 GiB / 336 GiB avail ... Jun 25 08:53:31 host1 ceph-mon[823315]: log_channel(cluster) log [INF] : Health alert mute PG_NOT_DEEP_SCRUBBED cleared (count increased from 60 to 61) Jun 25 08:53:31 host1 ceph-mon[823315]: Health alert mute PG_NOT_DEEP_SCRUBBED cleared (count increased from 60 to 61) I don't really understand what the code does [0] (I'm not a dev): ---snip--- if (!p->second.sticky) { auto q = all.checks.find(p->first); if (q == all.checks.end()) { mon.clog->info() << "Health alert mute " << p->first << " cleared (health alert cleared)"; p = pending_mutes.erase(p); changed = true; continue; } if (p->second.count) { // count-based mute if (q->second.count > p->second.count) { mon.clog->info() << "Health alert mute " << p->first << " cleared (count increased from " << p->second.count << " to " << q->second.count << ")"; p = pending_mutes.erase(p); changed = true; continue; ---snip--- Could anyone shed some light what I'm not understanding? Why would the mute clear although there are still PGs not deep-scrubbed? Thanks! Eugen [0] https://github.com/ceph/ceph/blob/d78ffd1247d6cef5cbd829e77204185dc0d3a8ba/…

4 5

RadosGW buckets stuck in lifecycle "PROCESSING"
by Niklaus Hofer 25 Jun '25

25 Jun '25

Dear all I managed to "CTRL+C" a `radosgw-admin lc process`. And now I have a bucket stuck in PROCESSING. | radosgw-admin lc list | jq '.[] | select(.status == "PROCESSING")' > { > "bucket": ":XXXXXXX:ost-dp1-tier3.7955243979.1209", > "started": "Sun, 22 Jun 2025 16:13:42 GMT", > "status": "PROCESSING" > } It's been like that for > 36 hours now. Is it possible that the "CTRL+C"-ed process has left behind some kind of lock? Can I check that? Maybe even remove the lock? That same question has been asked before on this ML [1] but in that thread, the bucket reverted back to UNINITIAL. I was hoping that maybe it would be the same here, but after waiting for a good while, I've lost hope on that... Many thanks in advance Niklaus Hofer Links: [1] https://www.mail-archive.com/ceph-users@lists.ceph.com/msg36977.html -- stepping stone AG Wasserwerkgasse 7 CH-3011 Bern Telefon: +41 31 332 53 63 www.stepping-stone.ch niklaus.hofer(a)stepping-stone.ch

1 1