- ceph-users - lists.ceph.io

mons with "low on available space" message
by filip Mutterer 20 Nov '25

20 Nov '25

Hi! its about my Lab setup. As I didn't have a thought about the space requirements for the root file system I want to know if there is a rule by thumb how much space a cpeh mon might need? rightnow I only have the "whoami" container running in rook-ceph so it does consume nearly no space in the OSD. But how far can I go with only 1.7 GB left on my Raspberry Pi's sdcard? Greetings filip

5 6

Something changed between RHEL 9.6 and 9.7 - unable to activate HDD OSDs with DB/WAL on NVMe
by Torkil Svensgaard 20 Nov '25

20 Nov '25

Hi We just upgraded a host running containerized squid from RHEL 9.6 to RHEL 9.7 and it failed to activate HDD OSDs with DB/WAL on NVMe when running the 9.7 kernel. SATA SSD OSDs were not affected. We rebooted it on the 9.6 kernel and everything worked again. Anyone know what changed? Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

2 1

cephfs purge queue
by Massimo Sgaravatto 20 Nov '25

20 Nov '25

Dear all In our cephfs installation I saw a couple of times a discrepancy between the "ceph df" and the "du -sh" outputs (see also this thread: https://lists.ceph.io\ /hyperkitty/list/ceph-users(a)ceph.io/thread/BBED4ADXO3CE4FYLCNUWB4OML6N6CTZU/ ). I have the feeling that the problem is because of a lot of deletions done by some users (I know that some of them in their jobs use to write from time to time a checkpoint file and then delete the previous checkpoint file) Trying to reproduce the problem, I am running a script with 30 parallel threads where each thread: - writes a 40 GB file - sleeps 5 secs - deletes the file produced in the previous iteration After some hours I have been able to reproduce the issue. Right now "ceph df" shows a usage of ~ 11 TB (~ 33 TB considering the replica 3) while "du -sh" shows a usage of about 2.7 TB. We have 2 active (and 1 standby) MDS instances. In one MDS I see: [root@ceph-mds-01 ~]# ceph --admin-daemon /run/ceph/ceph-mds.ceph-mds-01.asok perf dump | jq '.["purge_queue"]' { "pq_executing_ops": 10240, "pq_executing_ops_high_water": 13202, "pq_executing": 1, "pq_executing_high_water": 16, "pq_executed_ops": 74781922, "pq_executed": 924254, "pq_item_in_journal": 26986 } while in the second one: [root@ceph-mds-02 ~]# ceph --admin-daemon /run/ceph/ceph-mds.ceph-mds-02.asok perf dump | jq '.["purge_queue"]' { "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executed_ops": 0, "pq_executed": 0, "pq_item_in_journal": 0 } I am using the default values for filer_max_purge_ops and "mds_max_purge*: [root@ceph-mds-01 ~]# ceph daemon /run/ceph/ceph-mds.ceph-mds-01.asok config show | grep filer_max_purge "filer_max_purge_ops": "10", [root@ceph-mds-01 ~]# ceph daemon /run/ceph/ceph-mds.ceph-mds-01.asok config show | grep mds_max_purge "mds_max_purge_files": "64", "mds_max_purge_ops": "8192", "mds_max_purge_ops_per_pg": "0.500000", mds_cache_memory_limit is set to 32 GiB and doing a: # ceph daemon mds.<mds> perf dump | grep mds_co_bytes I see for the 2 MDS instances, , respectively: 8237145057 3479698 We are running ceph reef (we will soon update to squid) Should I try to increase filer_max_purge_ops ? Thanks a lot Cheers, Massimo

3 3

Ceph containers image version to pull them in a local registry
by wodel youchi 19 Nov '25

19 Nov '25

Hi, When deploying ceph, several images are used beside ceph itself, like prometheus, grafana, ...etc. When using a local registry, how to get the version of those images to pull them and prepare the registry? Regards.

2 1

ceph.conf on every node
by Albert Shih 19 Nov '25

19 Nov '25

Hi, Just like to know if it's a bad idea (like something going to bite me in few year) to have a copy of the /etc/ceph/ceph.conf from the master copy on every node of the cluster (install with cephadm). I «need» the ceph.conf to able to get the fsid on every node. Regards. -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mar. 18 nov. 2025 10:40:25 CET

6 10

Re: Ceph Steering Committee Meeting 2025-11-03
by Neha Ojha 18 Nov '25

18 Nov '25

Thank you for testing 20.2.0! Can you please share which version you upgraded this cluster from? I assume this was a test environment. Thanks, Neha On Mon, Nov 10, 2025 at 2:49 AM Lukasz Borek <lukasz(a)borek.org.pl> wrote: > > Thanks. > > 8 nodes NVME cluster with EC4+2 upgraded. No problem observed. > > Let me know if you want me to grab some logs. > > I will do some testing later this week. > > Thanks. > > NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID > ceph-exporter.fast-cephbackup-mon1 fast-cephbackup-mon1 running (117m) 9m ago 2w 9349k - 20.2.0 524f3da27646 4de9da93f7dc > ceph-exporter.fast-cephbackup-mon2 fast-cephbackup-mon2 running (117m) 9m ago 2w 9118k - 20.2.0 524f3da27646 456692e98ef9 > ceph-exporter.fast-cephbackup-mon3 fast-cephbackup-mon3 running (117m) 9m ago 2w 9172k - 20.2.0 524f3da27646 20c6764da039 > ceph-exporter.fast-cephbackup-osd1 fast-cephbackup-osd1 running (117m) 9m ago 2w 22.3M - 20.2.0 524f3da27646 6e5f442f5dde > ceph-exporter.fast-cephbackup-osd2 fast-cephbackup-osd2 running (117m) 8m ago 2w 22.3M - 20.2.0 524f3da27646 b341d7dd4390 > ceph-exporter.fast-cephbackup-osd3 fast-cephbackup-osd3 running (116m) 8m ago 2w 22.4M - 20.2.0 524f3da27646 1240b6cc66f7 > ceph-exporter.fast-cephbackup-osd4 fast-cephbackup-osd4 running (116m) 6m ago 2w 22.3M - 20.2.0 524f3da27646 df81ed883912 > ceph-exporter.fast-cephbackup-osd5 fast-cephbackup-osd5 running (116m) 6m ago 2w 22.2M - 20.2.0 524f3da27646 1c089585b35d > ceph-exporter.fast-cephbackup-osd6 fast-cephbackup-osd6 running (116m) 4m ago 2w 22.5M - 20.2.0 524f3da27646 3be6d4d8b1da > ceph-exporter.fast-cephbackup-osd7 fast-cephbackup-osd7 running (116m) 3m ago 2w 22.4M - 20.2.0 524f3da27646 cb690c2e1936 > ceph-exporter.fast-cephbackup-osd8 fast-cephbackup-osd8 running (115m) 81s ago 2w 22.5M - 20.2.0 524f3da27646 187336418c0d > crash.fast-cephbackup-mon1 fast-cephbackup-mon1 running (2h) 9m ago 4w 7625k - 20.2.0 524f3da27646 5bc9c4e61b24 > crash.fast-cephbackup-mon2 fast-cephbackup-mon2 running (2h) 9m ago 3w 7629k - 20.2.0 524f3da27646 927fc4633bf8 > crash.fast-cephbackup-mon3 fast-cephbackup-mon3 running (2h) 9m ago 3w 7633k - 20.2.0 524f3da27646 b1d1ad6e5904 > crash.fast-cephbackup-osd1 fast-cephbackup-osd1 running (2h) 9m ago 3w 7671k - 20.2.0 524f3da27646 a07c9ba448b0 > crash.fast-cephbackup-osd2 fast-cephbackup-osd2 running (2h) 8m ago 3w 7646k - 20.2.0 524f3da27646 6ddae0f43fb0 > crash.fast-cephbackup-osd3 fast-cephbackup-osd3 running (2h) 8m ago 3w 7650k - 20.2.0 524f3da27646 41f2c7c9fd33 > crash.fast-cephbackup-osd4 fast-cephbackup-osd4 running (2h) 6m ago 3w 7646k - 20.2.0 524f3da27646 b4953c96bd1b > crash.fast-cephbackup-osd5 fast-cephbackup-osd5 running (2h) 6m ago 3w 7650k - 20.2.0 524f3da27646 ad2a9d34e155 > crash.fast-cephbackup-osd6 fast-cephbackup-osd6 running (2h) 4m ago 3w 7646k - 20.2.0 524f3da27646 89eaa53086c4 > crash.fast-cephbackup-osd7 fast-cephbackup-osd7 running (2h) 3m ago 3w 7646k - 20.2.0 524f3da27646 87fd08e6b274 > crash.fast-cephbackup-osd8 fast-cephbackup-osd8 running (2h) 81s ago 3w 7638k - 20.2.0 524f3da27646 881408f9d76d > mgr.fast-cephbackup-mon1.xfamca fast-cephbackup-mon1 *:8443,9283,8765 running (3h) 9m ago 2w 507M - 20.2.0 524f3da27646 59f0778832bd > mgr.fast-cephbackup-mon2.gzasaj fast-cephbackup-mon2 *:8443,9283,8765 running (3h) 9m ago 3w 865M - 20.2.0 524f3da27646 862a55610d55 > mgr.fast-cephbackup-mon3.trmjnk fast-cephbackup-mon3 *:8443,9283,8765 running (3h) 9m ago 3w 508M - 20.2.0 524f3da27646 7c6c8271e0b2 > mon.fast-cephbackup-mon1 fast-cephbackup-mon1 running (3h) 9m ago 2w 442M 2048M 20.2.0 524f3da27646 b3701a59d934 > mon.fast-cephbackup-mon2 fast-cephbackup-mon2 running (3h) 9m ago 3w 440M 2048M 20.2.0 524f3da27646 c70c96d6f220 > mon.fast-cephbackup-mon3 fast-cephbackup-mon3 running (3h) 9m ago 3w 444M 2048M 20.2.0 524f3da27646 e0465231d5bb > osd.0 fast-cephbackup-osd5 running (2h) 6m ago 3w 14.2G 21.7G 20.2.0 524f3da27646 77d1125f3ded > osd.1 fast-cephbackup-osd3 running (2h) 8m ago 3w 15.6G 21.7G 20.2.0 524f3da27646 6224a7873350 > osd.2 fast-cephbackup-osd6 running (2h) 4m ago 3w 12.6G 21.7G 20.2.0 524f3da27646 173a847e73bf > osd.3 fast-cephbackup-osd7 running (2h) 3m ago 3w 14.8G 21.7G 20.2.0 524f3da27646 515a1b402559 > osd.4 fast-cephbackup-osd2 running (2h) 8m ago 3w 12.6G 21.7G 20.2.0 524f3da27646 555e60b3d272 > osd.5 fast-cephbackup-osd1 running (2h) 9m ago 3w 14.6G 21.7G 20.2.0 524f3da27646 15ec0a8364a1 > osd.6 fast-cephbackup-osd8 running (2h) 81s ago 3w 14.4G 21.7G 20.2.0 524f3da27646 cefe12a100b3 > osd.7 fast-cephbackup-osd4 running (2h) 6m ago 3w 13.7G 21.7G 20.2.0 524f3da27646 5ac5aa3645f7 > osd.8 fast-cephbackup-osd5 running (2h) 6m ago 3w 13.9G 21.7G 20.2.0 524f3da27646 3d9b4fc53385 > osd.9 fast-cephbackup-osd3 running (2h) 8m ago 3w 14.9G 21.7G 20.2.0 524f3da27646 8bd7b4154925 > osd.10 fast-cephbackup-osd6 running (2h) 4m ago 3w 14.0G 21.7G 20.2.0 524f3da27646 037877a08541 > osd.11 fast-cephbackup-osd1 running (2h) 9m ago 3w 14.2G 21.7G 20.2.0 524f3da27646 58659b91ef03 > osd.12 fast-cephbackup-osd7 running (2h) 3m ago 3w 14.3G 21.7G 20.2.0 524f3da27646 c9aec438a8e4 > osd.13 fast-cephbackup-osd8 running (2h) 81s ago 3w 15.1G 21.7G 20.2.0 524f3da27646 d60262f8462d > osd.14 fast-cephbackup-osd2 running (2h) 8m ago 3w 15.8G 21.7G 20.2.0 524f3da27646 9bba8fe67e06 > osd.15 fast-cephbackup-osd4 running (2h) 6m ago 3w 16.2G 21.7G 20.2.0 524f3da27646 847b77855b6f > osd.16 fast-cephbackup-osd5 running (2h) 6m ago 3w 13.3G 21.7G 20.2.0 524f3da27646 736932c6d76a > osd.17 fast-cephbackup-osd3 running (2h) 8m ago 3w 15.9G 21.7G 20.2.0 524f3da27646 4092581a7fe4 > osd.18 fast-cephbackup-osd6 running (2h) 4m ago 3w 15.2G 21.7G 20.2.0 524f3da27646 64c154a002aa > osd.19 fast-cephbackup-osd1 running (2h) 9m ago 3w 15.3G 21.7G 20.2.0 524f3da27646 00e58326a1ef > osd.20 fast-cephbackup-osd8 running (2h) 81s ago 3w 15.6G 21.7G 20.2.0 524f3da27646 22bb3db279d5 > osd.21 fast-cephbackup-osd2 running (2h) 8m ago 3w 15.0G 21.7G 20.2.0 524f3da27646 cc6c38738efd > osd.22 fast-cephbackup-osd4 running (2h) 6m ago 3w 14.5G 21.7G 20.2.0 524f3da27646 2d13e252b00e > osd.23 fast-cephbackup-osd7 running (2h) 3m ago 3w 14.1G 21.7G 20.2.0 524f3da27646 f9e17e6a58fa > osd.24 fast-cephbackup-osd5 running (2h) 6m ago 3w 14.3G 21.7G 20.2.0 524f3da27646 0f8197209d12 > osd.25 fast-cephbackup-osd3 running (2h) 8m ago 3w 14.1G 21.7G 20.2.0 524f3da27646 5589f732c8e7 > osd.26 fast-cephbackup-osd6 running (2h) 4m ago 3w 13.2G 21.7G 20.2.0 524f3da27646 936ef5adcd5b > osd.27 fast-cephbackup-osd5 running (2h) 6m ago 3w 12.3G 21.7G 20.2.0 524f3da27646 0241c249ebab > osd.28 fast-cephbackup-osd3 running (2h) 8m ago 3w 14.4G 21.7G 20.2.0 524f3da27646 0daec562fbee > osd.29 fast-cephbackup-osd6 running (2h) 4m ago 3w 14.3G 21.7G 20.2.0 524f3da27646 1c9dd693d8e0 > osd.30 fast-cephbackup-osd8 running (2h) 81s ago 3w 13.9G 21.7G 20.2.0 524f3da27646 c76825580158 > osd.31 fast-cephbackup-osd7 running (2h) 3m ago 3w 14.2G 21.7G 20.2.0 524f3da27646 4e49578cf239 > osd.32 fast-cephbackup-osd1 running (2h) 9m ago 3w 14.9G 21.7G 20.2.0 524f3da27646 5e49d50a8b93 > osd.33 fast-cephbackup-osd2 running (2h) 8m ago 3w 12.9G 21.7G 20.2.0 524f3da27646 ad90aff39abd > osd.34 fast-cephbackup-osd4 running (2h) 6m ago 3w 15.1G 21.7G 20.2.0 524f3da27646 c5849cdceb23 > osd.35 fast-cephbackup-osd5 running (2h) 6m ago 3w 15.3G 21.7G 20.2.0 524f3da27646 6c870213ae6a > osd.36 fast-cephbackup-osd3 running (2h) 8m ago 3w 11.0G 21.7G 20.2.0 524f3da27646 04ebd6dc1162 > osd.37 fast-cephbackup-osd6 running (2h) 4m ago 3w 13.7G 21.7G 20.2.0 524f3da27646 f700c3738d68 > osd.38 fast-cephbackup-osd1 running (2h) 9m ago 3w 15.5G 21.7G 20.2.0 524f3da27646 1669493ed49c > osd.39 fast-cephbackup-osd8 running (2h) 81s ago 3w 11.9G 21.7G 20.2.0 524f3da27646 bf9c144c445a > osd.40 fast-cephbackup-osd2 running (2h) 8m ago 3w 15.1G 21.7G 20.2.0 524f3da27646 539b78b9c0e8 > osd.41 fast-cephbackup-osd7 running (2h) 3m ago 3w 10.2G 21.7G 20.2.0 524f3da27646 be6466b9a097 > osd.42 fast-cephbackup-osd4 running (2h) 6m ago 3w 15.0G 21.7G 20.2.0 524f3da27646 a5e9904a0bb4 > osd.43 fast-cephbackup-osd5 running (2h) 6m ago 3w 16.0G 21.7G 20.2.0 524f3da27646 e2f62ba83020 > osd.44 fast-cephbackup-osd6 running (2h) 4m ago 3w 15.2G 21.7G 20.2.0 524f3da27646 a8072fa05fa7 > osd.45 fast-cephbackup-osd3 running (2h) 8m ago 3w 14.6G 21.7G 20.2.0 524f3da27646 01d56e9c8a62 > osd.46 fast-cephbackup-osd1 running (2h) 9m ago 3w 11.4G 21.7G 20.2.0 524f3da27646 59cbbb9896ba > osd.47 fast-cephbackup-osd2 running (2h) 8m ago 3w 15.0G 21.7G 20.2.0 524f3da27646 d93301e82e43 > osd.48 fast-cephbackup-osd8 running (2h) 81s ago 3w 14.7G 21.7G 20.2.0 524f3da27646 f61d43751328 > osd.49 fast-cephbackup-osd7 running (2h) 3m ago 3w 14.4G 21.7G 20.2.0 524f3da27646 52447edc6a9b > osd.50 fast-cephbackup-osd4 running (2h) 6m ago 3w 14.9G 21.7G 20.2.0 524f3da27646 49b6112a27d8 > osd.51 fast-cephbackup-osd5 running (2h) 6m ago 3w 14.0G 21.7G 20.2.0 524f3da27646 9229368e0e16 > osd.52 fast-cephbackup-osd6 running (2h) 4m ago 3w 14.5G 21.7G 20.2.0 524f3da27646 eb677be81e9b > osd.53 fast-cephbackup-osd3 running (2h) 8m ago 3w 15.6G 21.7G 20.2.0 524f3da27646 d5de2b9fff4c > osd.54 fast-cephbackup-osd7 running (2h) 3m ago 3w 15.3G 21.7G 20.2.0 524f3da27646 fa076a6e94d0 > osd.55 fast-cephbackup-osd1 running (2h) 9m ago 3w 14.4G 21.7G 20.2.0 524f3da27646 3c1d2089252d > osd.56 fast-cephbackup-osd2 running (2h) 8m ago 3w 14.0G 21.7G 20.2.0 524f3da27646 b29067e714b9 > osd.57 fast-cephbackup-osd8 running (2h) 81s ago 3w 14.1G 21.7G 20.2.0 524f3da27646 a9a5be72ba34 > osd.58 fast-cephbackup-osd4 running (2h) 6m ago 3w 15.4G 21.7G 20.2.0 524f3da27646 abbf11f1e2ea > osd.59 fast-cephbackup-osd7 running (2h) 3m ago 3w 13.6G 21.7G 20.2.0 524f3da27646 ca8df0031018 > osd.60 fast-cephbackup-osd1 running (2h) 9m ago 3w 15.1G 21.7G 20.2.0 524f3da27646 74738c3d930d > osd.61 fast-cephbackup-osd2 running (2h) 8m ago 3w 17.4G 21.7G 20.2.0 524f3da27646 098aa2c69b0f > osd.62 fast-cephbackup-osd8 running (2h) 81s ago 2w 11.6G 21.7G 20.2.0 524f3da27646 113f92904b8f > osd.63 fast-cephbackup-osd4 running (2h) 6m ago 3w 15.2G 21.7G 20.2.0 524f3da27646 70dff6c0798d > rgw.label.fast-cephbackup-osd1.oyqdoe fast-cephbackup-osd1 *:8000 running (2h) 9m ago 3w 648M - 20.2.0 524f3da27646 d83719a402d5 > rgw.label.fast-cephbackup-osd2.lzxiwi fast-cephbackup-osd2 *:8000 running (2h) 8m ago 3w 554M - 20.2.0 524f3da27646 49914023f796 > rgw.label.fast-cephbackup-osd3.bzbawb fast-cephbackup-osd3 *:8000 running (2h) 8m ago 2w 545M - 20.2.0 524f3da27646 81dce830901e > rgw.label.fast-cephbackup-osd4.epcdwr fast-cephbackup-osd4 *:8000 running (2h) 6m ago 3w 595M - 20.2.0 524f3da27646 b7b04e7728cc > rgw.label.fast-cephbackup-osd5.fhzzju fast-cephbackup-osd5 *:8000 running (2h) 6m ago 2w 542M - 20.2.0 524f3da27646 845d6fcf68b6 > rgw.label.fast-cephbackup-osd6.fztzxe fast-cephbackup-osd6 *:8000 running (119m) 4m ago 2w 648M - 20.2.0 524f3da27646 45bb95f840d7 > rgw.label.fast-cephbackup-osd7.asrxjx fast-cephbackup-osd7 *:8000 running (119m) 3m ago 2w 853M - 20.2.0 524f3da27646 0e6bae7f8a14 > rgw.label.fast-cephbackup-osd8.avbbsq fast-cephbackup-osd8 *:8000 running (119m) 81s ago 3w 584M - 20.2.0 524f3da27646 bfd48ec8d147 > > > On Sun, 9 Nov 2025 at 23:31, Yuri Weinstein <yweinste(a)redhat.com> wrote: >> >> Hello >> >> 20.2.0 image is pre-released >> Please don't share it >> https://quay.io/repository/yuriw/ceph?tab=tags >> v20.2.0-20251104 >> >> Thx for helping! >> >> >> On Sun, Nov 9, 2025 at 1:02 PM Neha Ojha <nojha(a)redhat.com> wrote: >>> >>> Thank you, Lukasz! @yuri can you please share the pre-release image? >>> >>> Thanks, >>> Neha >>> >>> On Sun, Nov 9, 2025 at 12:49 PM Lukasz Borek <lukasz(a)borek.org.pl> wrote: >>>> >>>> Hi, >>>> >>>> Happy to help with testing. I have not yet production cluster with EC that can be upgraded. >>>> >>>> Let me know image detail. >>>> >>>> Łukasz Borek >>>> lukasz(a)borek.org.pl >>>> >>>> >>>> W dniu pt., 7 lis 2025 o 21:12 Neha Ojha <nojha(a)redhat.com> napisał(a): >>>>> >>>>> Special thanks to @Stefan Kooman for testing upgrades for 20.2.0! He >>>>> has successfully upgraded 2 clusters including one with RGW >>>>> mutli-site. >>>>> >>>>> Thanks, >>>>> Neha >>>>> >>>>> >>>>> On Thu, Nov 6, 2025 at 12:31 PM Neha Ojha <nojha(a)redhat.com> wrote: >>>>> > >>>>> > Hi everyone, >>>>> > >>>>> > On Mon, Nov 3, 2025 at 9:28 AM Casey Bodley <cbodley(a)redhat.com> wrote: >>>>> > > >>>>> > > we covered two topics today that generated a lot of good discussion, >>>>> > > but neither reached a consensus: >>>>> > > >>>>> > > AI Code contributions >>>>> > > * current lack of clear policy on submissions containing AI-generated >>>>> > > code. do we want one? what are the concerns/risks? how do we enforce? >>>>> > > * requires further discussion >>>>> > > >>>>> > > Tentacle updates? >>>>> > > * merge conflicts in https://github.com/ceph/ceph/pull/66096 >>>>> > > * lrc upgrade? >>>>> > > ** very risky with pending lab move next week. after the move, lrc may >>>>> > > not be available for upgrade until 2026? >>>>> > > ** should this block the release or not? requires further discussion >>>>> > >>>>> > Are there any community members who would be willing to help test >>>>> > Tentacle 20.2.0 upgrades on their test/staging clusters? This will >>>>> > immensely help us build more confidence in the stability of the >>>>> > release and expedite GA'ing long-awaited features like Fast EC! >>>>> > >>>>> > Thanks, >>>>> > Neha >>>>> > >>>>> > >>>>> > > _______________________________________________ >>>>> > > ceph-users mailing list -- ceph-users(a)ceph.io >>>>> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> > > >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > -- > Łukasz Borek > lukasz(a)borek.org.pl

2 3

Ceph kernel page cache behaviour
by Domhnall McGuigan 18 Nov '25

18 Nov '25

Hi all, I've been investigating a problem with tests of our database processing jobs running on cephfs apparently reading several times more data than actually exists in the working set. I haven't had much luck diagnosing the cause so far, but one possibility I'd at least like to rule out is that the kernel cephfs client is discarding pages with cached reads too aggressively and so we're redundantly reading small chunks of data over and over again. Our access patterns are quite unusual -- we read small chunks (perhaps a few hundred kilobytes) from files, then close the file handles before reopening and doing more reads. I found a 10 year old thread indicating that cephfs would outright clear page caches for files in situations like this: https://ceph-users.ceph.narkive.com/o2vJ4rWf/cephfs-and-page-cache That behaviour was patched out long ago, but I'd like to know whether aggressive page cache eviction like this is still known to be an issue with certain access patterns, even if just to eliminate it as a possibility. We're hosting ceph on the same machines as our database which further makes me wonder if some kind of cache pressure could be involved. Regards, Domhnall *********************************************************************************************************************************************************************** This email, its contents and any files attached are a confidential communication and are intended only for the named addressees indicated in the message. If you are not the named addressee or if you have received this email in error, you may not, without the consent of KX, copy, use or rely on any information or attachments in any way. Please notify the sender by return email and delete it from your email system. Unless separately agreed, KX does not accept any responsibility for the accuracy or completeness of the contents of this email or its attachments. Please note that any views, opinion or advice contained in this communication are those of the sending individual and not those of KX and KX shall have no liability whatsoever in relation to this communication (or its content) unless separately agreed. ***********************************************************************************************************************************************************************

2 1

[PSA] MDS running on big-endian machines will break clients operating on fragmented directories
by Patrick Donnelly 17 Nov '25

17 Nov '25

Greetings, For anyone running or planning to run an MDS on a big-endian architecture, please be aware that when directories are fragmented (an automatic process when they become large) that the MDS will send byte-swapped fragment identifiers to clients. This confuses the client and can result in it entering an infinite loop trying to read the contents of a directory. The issue is tracked here: https://tracker.ceph.com/issues/73792 The fix includes code to automatically correct any persisted byte-swapped fragment identifiers. We expect it to be backported to Tentacle and Squid as soon as the fix is in the main branch. Regards, -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Ceph Network Design
by Alexander Leutz 14 Nov '25

14 Nov '25

Hi Ceph Users, I need to understand the Ceph network design and the way it communicates with different partners. Ceph is my storage, also I name the communication as storage network. The proxmox ceph servers has 2 1Gb management ports an 2 10 Gb ports for storage network. I set the mon public and the osd cluster networks to the privat IP range 10.10.x.x: ceph config set mon public_network 10.10.x.x/24 ceph config set osd cluster_network 10.10.x.x/24 After this I only see this as paramter in the GUI screen, under host / ceph / configuration in the section Configuration Database Und [global] I still see a second entry named public_network = 192.168.x.x (this is my management network of the proxmox host). Is this correct? I thought, all storage traffic needs to be isolated in my storage network 10.10.x.x - no IPs in the mgmt network for ceph. Or do I have the wrong understanding of the ceph network design? Best regards, Enjoy your weekend, Beste Grüße, Schönes Wochenende, Alexander Concat AG IT Solutions | Berliner Ring 127 - 129 | 64625 Bensheim | www.concat.de alexander.leutz(a)concat.de | Tel.: +49 6251 7026-143 | Mobil: +49 151 18856009 | Fax: +49 6251 7026-356 Hauptsitz Bensheim a. d. Bergstraße | Vorstand Olaf von Heyer Aufsichtsratsvorsitzender Tim Patronik | Amtsgericht Darmstadt (HRB 25326)

4 5

New user with some questions...
by Matthias Riße 14 Nov '25

14 Nov '25

Hey all! I am evaluating Ceph at work and have some questions to people with more experience with it, to avoid making dumb mistakes... I hope this is the right place to ask. So, to give some context, we have 5 machines with about 400 TiB of raw storage total in the form of 8 HDDs per host. The hardware is not all the same, two nodes have a rather low core count which would be dedicated entirely to Ceph, the other three I would like to use in a HCI fashion to run VMs off of Ceph. I'd like to at least be able to take one node at a time out of rotation for maintenance purposes. I'd be using cephadm for the setup, which in my testing works really well. As for the questions: 1. 3x replication is a tough sell IMO. For better storage utilization I am looking at EC. 2+2 would be an obvious choice, and seems to be rather performant. But what about MSR with higher values, e.g. 5+3 with 2 OSDs per host across 4 hosts? Or taken to more of an extreme, 17+7 with 6 OSDs across 4 hosts? This should also allow me to take one host out for maintenance at a time, but has a better storage utilization. Is this a stupid idea? (16+8 might make more sense, to also be able to sustain a disk failure while in maintenance.) 2. I've read that I can update the crush rule of EC pools after the fact, to change both the failure domain as well as the device class of the pool. What about changing k, m, or the plugin type? My understanding is that this is not supported, but Ceph didn't stop me from doing it and it seems to do /something/ when those values are changed? 3. Right now we are using libvirt with qcow2 images on local storage. I know that with Ceph the commonly recommended way would be to use RBD instead, but we have an existing proprietary tape archive for backup purposes whose official client can to my knowledge only do file-based backups, and around which we already have a system to back up live snapshots of VMs. How bad of an idea would it actually be to use qcow2 on top of (kernel- or fuse-mounted?) CephFS? So far it seems to perform on par with RBD in my testing, but both also seem to fully saturate the single OSDs per host I am testing with anyway. 4. In v20 there seems to be a new ec_optimizations feature for ec pools used in CephFS or RBD. Are those a good idea with this kind of (large) qcow2 image workload on top of CephFS? 5. Speaking of v20, while it is not yet the latest "active" version extrapolating the releases suggests that it could soon become that. Should I be waiting for it / start a new cluster with v20 now already? Thanks for any insights you can give! Kind regards Matthias Riße -- --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellvertretende Vorsitzende), Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers --------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------

4 8