- ceph-users - lists.ceph.io

Squid: successfully drained host can't be removed
by Eugen Block 26 Jul '25

26 Jul '25

Hi *, an unexpected issue occurred today, at least twice, so it seems kind of reproducable. I've been preparing a demo in a (virtual) lab cluster (19.2.2) and wanted to drain multiple hosts. The first time I didn't pay much attention, but the draining seemed stuck (kind of a common issue these days), so I intervened and cleaned up until I got into a healthy state, all good. Then I did my thing, changed the crush tree, added the removed hosts again, cephadm created the OSDs, backfill finished successfully. Now I wanted to reset the cluster again to my starting point, so I issued the drain command again for multiple hosts (each host has 2 OSDs): # for i in {5..8}; do ceph orch host drain host$i; done This time all OSDs were drained successfully (I watched 'ceph orch osd rm status'), so I wanted to remove the hosts, but it failed: # for i in {5..8}; do ceph orch host rm host$i --rm-crush-entry; done Removed host 'host5' Removed host 'host6' Removed host 'host7' Error EINVAL: Not allowed to remove host8 from cluster. The following daemons are running in the host: type id -------------------- --------------- osd 6 Please run 'ceph orch host drain host8' to remove daemons from host But there was nothing to drain anymore, osd.6 was already successfully removed from the crush tree. But on host8 there was still a daemon I had to clean up manually: host8:~ # cephadm rm-daemon --name osd.6 --fsid 543967bc-e586-32b8-bd2c-2d8b8b168f02 --force I compared the cephadm.log files (3 out of 4 to-be-drained hosts were successfully drained) and on host8 the command rm-daemon was never executed (until I did manually). Is this a known issue? It doesn't seem to happen with only on host, at least I didn't notice in the past. Should I create a tracker for this? Thanks, Eugen

2 3

MDS Mount Issue - 19.2.2
by Devender Singh 26 Jul '25

26 Jul '25

Hello all I have cephfs working fine with running ads but still not able to mount, getting error mount error: no mds (Metadata Server) is up Below are details # ceph osd dump | grep min_compat_client require_min_compat_client reef min_compat_client reef # ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.002882", "last_optimize_started": "Tue Jul 15 21:16:41 2025", "mode": "upmap-read", "no_optimization_needed": true, "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", "plans": [] } root@van2-host06n:~# dpkg -l ceph-common Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==============-=============-============-================================================================== ii ceph-common 19.2.2-1jammy amd64 common utilities to mount and interact with a ceph storage cluster root@van2-host06n:~# ceph fs status van2.dev - 0 clients ======== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active van2.dev.van2-host06n.wqttsi Reqs: 0 /s 10 13 12 0 1 active van2.dev.van2-host07n.fngvlt Reqs: 0 /s 10 13 11 0 POOL TYPE USED AVAIL cephfs.van2.dev.meta metadata 230k 53.0T cephfs.van2.dev.data data 0 53.0T STANDBY MDS van2-dev.van2-host07n.jnacmh van2-dev.van2-host06n.wpzshe van2-dev.van2-host08n.slrfej MDS version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable) SYstemd file. =========== # cat cephfs.mount # # this mounts on /cephfs [Unit] Description=mount /cephfs After=network-online.target Requires=network-online.target [Mount] Where=/cephfs What=10.101.1.246:6789,10.101.1.247:6789,10.101.1.248:6789:/ Type=ceph Options=fs=van2.dev,name=admin,noatime,_netdev,x-systemd.automount [Install] WantedBy=multi-user.target root@van2-host06n:~# journalctl -u cephfs.mount Jul 15 19:31:10 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:32:12 van2-host06n.van2.example.com mount[1809096]: mount error: no mds (Metadata Server) is up. The clu> Jul 15 19:32:12 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=> Jul 15 19:32:12 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:32:12 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:34:59 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:35:59 van2-host06n.van2.example.com mount[1810584]: mount error: no mds (Metadata Server) is up. The clu> Jul 15 19:35:59 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=> Jul 15 19:35:59 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:35:59 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:36:16 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:37:17 van2-host06n.van2.example.com mount[1811498]: mount error: no mds (Metadata Server) is up. The clu> Jul 15 19:37:17 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=> Jul 15 19:37:17 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:37:17 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:49:38 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:50:40 van2-host06n.van2.example.com mount[1817486]: mount error: no mds (Metadata Server) is up. The clu> Jul 15 19:50:40 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=> Jul 15 19:50:40 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:50:40 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:51:58 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=killed, status=> Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'signal'. Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Unit process 1817555 (mount.ceph) remains > Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: Unmounted mount /cephfs. Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Found left-over process 1817555 (mount.cep> Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: This usually indicates unclean termination of a previous> Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:52:59 van2-host06n.van2.example.com mount[1817555]: mount error: no mds (Metadata Server) is up. The clu> lines 1-29...skipping... Jul 15 19:31:10 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:32:12 van2-host06n.van2.example.com mount[1809096]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be> Jul 15 19:32:12 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:32:12 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:32:12 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:34:59 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:35:59 van2-host06n.van2.example.com mount[1810584]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be> Jul 15 19:35:59 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:35:59 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:35:59 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:36:16 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:37:17 van2-host06n.van2.example.com mount[1811498]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be> Jul 15 19:37:17 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:37:17 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:37:17 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:49:38 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:50:40 van2-host06n.van2.example.com mount[1817486]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be> Jul 15 19:50:40 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:50:40 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:50:40 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:51:58 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=killed, status=15/TERM Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'signal'. Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Unit process 1817555 (mount.ceph) remains running after unit stopped. Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: Unmounted mount /cephfs. Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Found left-over process 1817555 (mount.ceph) in control group while starting uni> Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: This usually indicates unclean termination of a previous run, or service implementation defici> Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:52:59 van2-host06n.van2.example.com mount[1817555]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be> Jul 15 19:53:26 van2-host06n.van2.example.com mount[1817572]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be> Jul 15 19:53:26 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:53:26 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:53:26 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:56:29 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:57:29 van2-host06n.van2.example.com mount[1817655]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be> Jul 15 19:57:29 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:57:29 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:57:29 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. lines 1-38...skipping... Jul 15 19:31:10 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:32:12 van2-host06n.van2.example.com mount[1809096]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized Jul 15 19:32:12 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:32:12 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:32:12 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:34:59 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:35:59 van2-host06n.van2.example.com mount[1810584]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized Jul 15 19:35:59 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:35:59 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:35:59 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:36:16 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:37:17 van2-host06n.van2.example.com mount[1811498]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized Jul 15 19:37:17 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:37:17 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:37:17 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:49:38 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:50:40 van2-host06n.van2.example.com mount[1817486]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized Jul 15 19:50:40 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:50:40 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:50:40 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:51:58 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=killed, status=15/TERM Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'signal'. Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Unit process 1817555 (mount.ceph) remains running after unit stopped. Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: Unmounted mount /cephfs. Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Found left-over process 1817555 (mount.ceph) in control group while starting unit. Ignoring. Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Jul 15 19:52:24 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:52:59 van2-host06n.van2.example.com mount[1817555]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized Jul 15 19:53:26 van2-host06n.van2.example.com mount[1817572]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized Jul 15 19:53:26 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:53:26 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:53:26 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:56:29 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:57:29 van2-host06n.van2.example.com mount[1817655]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized Jul 15 19:57:29 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:57:29 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:57:29 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 19:58:50 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 19:59:51 van2-host06n.van2.example.com mount[1818287]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized Jul 15 19:59:51 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Mount process exited, code=exited, status=32/n/a Jul 15 19:59:51 van2-host06n.van2.example.com systemd[1]: cephfs.mount: Failed with result 'exit-code'. Jul 15 19:59:51 van2-host06n.van2.example.com systemd[1]: Failed to mount mount /cephfs. Jul 15 20:05:43 van2-host06n.van2.example.com systemd[1]: Mounting mount /cephfs... Jul 15 20:06:44 van2-host06n.van2.example.com mount[1818912]: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized lines 1-45

3 11

2 MDSs behind on trimming on my Ceph Cluster since the upgrade from 18.2.6 (reef) to 19.2.2 (squid)
by Edouard FAZENDA 25 Jul '25

25 Jul '25

Dear all, I have the following issue on my Ceph cluster MDSs behind on trimming on my Ceph Cluster since the upgrade using cephadm from 18.2.6 to 19.2.2. Here some cluster logs : 8/5/25 09:00 AM[WRN]overall HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:50 AM[WRN]overall HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:40 AM[WRN]mds.cephfs.node2.isqjza(mds.0): Behind on trimming (326/128) max_segments: 128, num_segments: 326 8/5/25 08:40 AM[WRN]mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming (326/128) max_segments: 128, num_segments: 326 8/5/25 08:40 AM[WRN][WRN] MDS_TRIM: 2 MDSs behind on trimming 8/5/25 08:40 AM[WRN]Health detail: HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:33 AM[WRN]Health check update: 2 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:33 AM[WRN]Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:30 AM[INF]overall HEALTH_OK 8/5/25 08:22 AM[INF]Cluster is now healthy 8/5/25 08:22 AM[INF]Health check cleared: MDS_TRIM (was: 1 MDSs behind on trimming) 8/5/25 08:22 AM[INF]MDS health message cleared (mds.?): Behind on trimming (525/128) 8/5/25 08:22 AM[WRN]Health check update: 1 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:22 AM[INF]MDS health message cleared (mds.?): Behind on trimming (525/128) 8/5/25 08:20 AM[WRN]overall HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:10 AM[WRN]mds.cephfs.node2.isqjza(mds.0): Behind on trimming (332/128) max_segments: 128, num_segments: 332 8/5/25 08:10 AM[WRN]mds.cephfs.node1.ojmpnk(mds.0): Behind on trimming (332/128) max_segments: 128, num_segments: 332 8/5/25 08:10 AM[WRN][WRN] MDS_TRIM: 2 MDSs behind on trimming 8/5/25 08:10 AM[WRN]Health detail: HEALTH_WARN 2 MDSs behind on trimming 8/5/25 08:03 AM[WRN]Health check update: 2 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:03 AM[WRN]Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 8/5/25 08:00 AM[INF]overall HEALTH_OK #ceph fs status cephfs - 50 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephfs.node1.ojmpnk Reqs: 10 /s 305k 294k 91.8k 6818 0-s standby-replay cephfs.node2.isqjza Evts: 0 /s 551k 243k 90.6k 0 POOL TYPE USED AVAIL cephfs_metadata metadata 2630M 2413G cephfs_data data 12.7T 3620G STANDBY MDS cephfs.node3.vdicdn MDS version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable) # ceph versions { "mon": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "mgr": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 2 }, "osd": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 18 }, "mds": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 3 }, "rgw": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 6 }, "overall": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 32 } } #ceph orch ps --daemon-type mds NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mds.cephfs.node1.ojmpnk rke-sh1-1 running (18h) 4m ago 19M 1709M - 19.2.2 4892a7ef541b 8dd8db30a1de mds.cephfs.node2.isqjza rke-sh1-2 running (18h) 2m ago 3y 1720M - 19.2.2 4892a7ef541b 7b9d5b692764 mds.cephfs.node3.vdicdn rke-sh1-3 running (18h) 108s ago 18M 27.9M - 19.2.2 4892a7ef541b d2de22a15e18 root@node1:~# ceph config show-with-defaults mds.cephfs.rke-sh1-3.vdicdn | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate" mds_cache_memory_limit 4294967296 default mds_cache_trim_decay_rate 1.000000 default mds_cache_trim_threshold 262144 default mds_recall_max_caps 30000 default mds_recall_max_decay_rate 1.500000 default root@node2:~# ceph config show-with-defaults mds.cephfs.rke-sh1-2.isqjza | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate" mds_cache_memory_limit 4294967296 default mds_cache_trim_decay_rate 1.000000 default mds_cache_trim_threshold 262144 default mds_recall_max_caps 30000 default mds_recall_max_decay_rate 1.500000 default root@node3:~# ceph config show-with-defaults mds.cephfs.rke-sh1-1.ojmpnk | egrep "mds_cache_trim_threshold|mds_cache_trim_decay_rate|mds_cache_memory_limit|mds_recall_max_caps|mds_recall_max_decay_rate" mds_cache_memory_limit 4294967296 default mds_cache_trim_decay_rate 1.000000 default mds_cache_trim_threshold 262144 default mds_recall_max_caps 30000 default mds_recall_max_decay_rate 1.500000 default # ceph mds stat cephfs:1 {0=cephfs.node1.ojmpnk=up:active} 1 up:standby-replay 1 up:standby Do you have an idea on what could happen ? Should I increate mds_cache_trim_decay_rate ? I saw the folloing issue : <https://tracker.ceph.com/issues/66948> Bug #66948: mon.a (mon.0) 326 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log - CephFS - Ceph ( <https://github.com/ceph/ceph/pull/60838> squid: mds: trim mdlog when segments exceed threshold and trim was idle by vshankar · Pull Request #60838 · ceph/ceph · GitHub ) maybe related ? Thanks for the help 😊 Best Regards, Edouard Fazenda. <https://www.csti.ch/> Swiss Cloud Provider Edouard Fazenda Technical Support <https://www.csti.ch/> Chemin du Curé-Desclouds, 2 CH-1226 Thonex +41 22 869 04 40 www.csti.ch

4 6

Squid 19.2.2 - mon_target_pg_per_osd change not applied
by Steven Vacaroaia 25 Jul '25

25 Jul '25

Hi, I wanted to increase the number of PG per OSD ....and did so by using ceph config set global mon_target_pg_per_osd 800 Although the OSD config has the new value , I am still unable to create pools that end up creating more than 250PG per OSD I have restarted the OSDs ...and the monitors ... Any ideas or suggestions for properly applying the change would be appreciated Steven [image: image.png]

4 3

Question about Dashboard and SSO (version 19.2.2)
by Taylor, Kevin P. (AOS) 25 Jul '25

25 Jul '25

Hi. I've spent quite a bit working through how to get SSO working with Keycloak using SAML, and I succeeded finally in getting that to go. We have 3 nodes in our test cluster and a round-robin DNS entry pointing to all 3 of them (just calling them node-1, node-2, and node-3 here. The configuration we did for this to work involved creating an SP cert and key, which I had to copy into the mgr containers in order to work: ceph dashboard sso setup saml2 https://node-1:8443 https://keycloak/realms/ceph/protocol/saml/descriptor username https://keycloak/realms/ceph /tmp/sp-cert.txt /tmp/sp-key.txt I'm able to log in fine (get sent to keycloak and directed back) as long as the management server is running on node-1, if I fail it over to node-2, I get prompted to go to keycloak and log in, but then receive this error in the browser: {"status": "415 Unsupported Media Type", "detail": "Expected an entity of content type application/json, text/javascript", "request_id": "dd5a9f68-a86c-462f-b25d-3d881e0fc350"} I know I didn't post all details of everything in this setup, but maybe my description of what's going on is useful. Does anyone have any ideas about this, or have suggestions as to getting proper failover + SSO for the dashboard? Thanks for any suggestions.

1 0

[ceph-user]ceph-ansible pacific || RGW integration with ceph dashboard
by Danish Khan 25 Jul '25

25 Jul '25

Dear Team, I am trying to integrate ceph dashboard with RGW so that I can create bucket from ceph dashboard. I am following below document for it: https://docs.ceph.com/en/pacific/mgr/dashboard/ when I run below command then it says "RGW credentials configured" #ceph dashboard set-rgw-credentials But my ceph dashboard is still not integrated with RGW and ceph dashboard still says that "The Object Gateway Service is not configured". In octopus release I used to run below commands which still works fine: $ ceph dashboard set-rgw-api-host <host> $ ceph dashboard set-rgw-api-port <port> but now these commands don't work in pacific release. Do we have any command in pacific release to specify rgw endpoint to integrate it with ceph dashboards? Regards, Danish Khan

2 1

CEPH performance all Flash lower than local
by Devender Singh 24 Jul '25

24 Jul '25

Hello all I tried doing some fio test on local disk(NVME) and ceph rbd. Why ceph is having low IO whereas it’s also on all NVME. What to tune to reach equal amount of IO? root@node01:~/fio-cdm# python3 fio-cdm ./ tests: 5, size: 1.0GiB, target: /root/fio-cdm 6.3GiB/64.4GiB |Name | Read(MB/s)| Write(MB/s)| |------------|------------|------------| |SEQ1M Q8 T1 | 8441.37| 3588.71| |SEQ1M Q1 T1 | 3074.86| 1172.46| |RND4K Q32T16| 723.65| 733.76| |. IOPS | 176671.80| 179141.74| |. latency us| 2892.49| 2839.37| |RND4K Q1 T1 | 71.05| 57.88| |. IOPS | 17347.13| 14131.57| |. latency us| 56.13| 66.40| ### When vm moved to ceph storage... root@node01:~/fio-cdm# python3 fio-cdm ./ tests: 5, size: 1.0GiB, target: /root/fio-cdm 9.3GiB/64.4GiB |Name | Read(MB/s)| Write(MB/s)| |------------|------------|------------| |SEQ1M Q8 T1 | 1681.40| 889.89| |SEQ1M Q1 T1 | 310.74| 852.11| |RND4K Q32T16| 403.04| 274.23| |. IOPS | 98397.32| 66951.49| |. latency us| 5196.98| 7637.10| |RND4K Q1 T1 | 4.69| 45.18| |. IOPS | 1146.17| 11029.39| |. latency us| 869.47| 87.50| Regards Dev

5 5

Re: squid 19.2.2 - cannot bootstrap - error writing to /tmp/monmap (21) Is a directory
by Steven Vacaroaia 24 Jul '25

24 Jul '25

Thanks Anthony The volume bind command looks like this -v /tmp/ceph-tmp-wvh4u18:/tmp/monmap:z so i guess the /tmp/ceph-tmp-xxxx never gets created so there is nothing to map What would be the solution though ...I tried removing and reinstalling docker Update and upgrade ( Ubuntu 24.04.2 LTS) On Thu, 24 Jul 2025 at 11:09, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > > How did you get that attachment through the list? Curious. > > I think /tmp/monmap is relative to the directory tree that the container > sees: > > -v /tmp/ceph-tmp-wvh4u18:/tmp/monmapz > > Because of the wrapping I’m not sure about the z on the next line, but -v > can be considered like a bind mount of the first path so that container > sees it on the second path. > > Now, as to what happened to /tmp/ceph-tmpxxxx, I can’t say. > > > > On Jul 24, 2025, at 10:53 AM, Steven Vacaroaia <stef97(a)gmail.com> wrote: > > > > Hi, > > > > I had to zap and start from scratch my cluster and now I cannot > bootstrap using cephadm > > > > The error is " monmaptool: error writing to /tmp/monmap: (21) Is a > directory " > > > > There is no /tmp/monmap directory though > > > > I also made sure there is nothing in /var/lib/ceph, no "ceph" process , > nothing in /etc/systemd/system/ceph* or /etc/ceph AND rebooted the server > > > > Any help will be appreciated > > > > Many thanks > > Steven > > > > <image.png> > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 3

squid 19.2.2 - cannot bootstrap - error writing to /tmp/monmap (21) Is a directory
by Steven Vacaroaia 24 Jul '25

24 Jul '25

Hi, I had to zap and start from scratch my cluster and now I cannot bootstrap using cephadm The error is " monmaptool: error writing to /tmp/monmap: (21) Is a directory " There is no /tmp/monmap directory though I also made sure there is nothing in /var/lib/ceph, no "ceph" process , nothing in /etc/systemd/system/ceph* or /etc/ceph AND rebooted the server Any help will be appreciated Many thanks Steven [image: image.png]

1 0

Ceph sf
by Szabo, Istvan (Agoda) 24 Jul '25

24 Jul '25

Hi, Somehow this output always made me confused. Can you confirm my calculation is good here? I have this output: --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED nvme 89 TiB 82 TiB 6.6 TiB 6.6 TiB 7.43 rbd 31 TiB 16 TiB 16 TiB 16 TiB 49.52 ssd 2.3 PiB 1.0 PiB 1.2 PiB 1.2 PiB 54.14 TOTAL 2.4 PiB 1.1 PiB 1.2 PiB 1.2 PiB 52.36 SSD: ec 4:2 rbd: repl 3 I want to know the stored data, which means the user side data. In ceph in EC 4:2 a 100MB object is stored on 150MB. In case of ssd, I want to know the used 1.2PiB means how much real user data? For me 1.2PiB used SSD means ~903TB user data on ec 4:2. In case of rbd, ~6TB. Am I correct? Ty ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1