- ceph-users - lists.ceph.io

Re: space size issue
by Eugen Block 28 Mar '25

28 Mar '25

You probably have to change the existing crush rule to use OSD as failure domain instead of host. But if you have only one host, why not set up some other file storage with (software) raid or something? Zitat von "mihai.ciubancan" <mihai.ciubancan(a)eli-np.ro>: > The problem is that I have only one host, and the host doesn't have > a RAID controller...I have set as Anthone suggested, but nothing > changed.Mihai > -------- Original message --------From: Eugen Block <eblock(a)nde.ag> > Date: 3/28/25 3:47 PM (GMT+02:00) To: Mihai Ciubancan > <mihai.ciubancan(a)eli-np.ro> Cc: ceph-users(a)ceph.io Subject: > [ceph-users] Re: space size issue You have the autoscaler enabled, > but it stuck changing the pg_num. The default replicated_rule > (which you are using) requires as many hosts as your pool size is, > so in your case 2. (If you value your data, don't use replicated > pools with size 2.)You could make it work with only one host (as > Anthony suggested with osd_crush_chooseleaf_type 0), but you don't > any real resiliency. I recommend to reconsider your setup.Zitat von > Mihai Ciubancan <mihai.ciubancan(a)eli-np.ro>:> Hi Eugen,>> Thanks for > your answer. Please find below the output of the command:>> ceph osd > pool ls detail> pool 1 '.mgr' replicated size 2 min_size 1 > crush_rule 0 object_hash > rjenkins pg_num 1 pgp_num 1 > autoscale_mode on last_change 323 flags > hashpspool,nearfull > stripe_width 0 pg_num_max 32 pg_num_min 1 > application mgr > read_balance_score 12.50> pool 2 'cephfs.cephfs.meta' replicated > size 2 min_size 1 crush_rule > 0 object_hash rjenkins pg_num 16 > pgp_num 1 pgp_num_target 16 > autoscale_mode on last_change 295 > lfor 0/0/54 flags > hashpspool,nearfull stripe_width 0 > pg_autoscale_bias 4 pg_num_min 16 > recovery_priority 5 application > cephfs read_balance_score 12.50> pool 3 'cephfs.cephfs.data' > replicated size 2 min_size 1 crush_rule > 0 object_hash rjenkins > pg_num 129 pgp_num 1 pg_num_target 512 > pgp_num_target 512 > autoscale_mode on last_change 326 lfor 0/0/54 > flags > hashpspool,nearfull,bulk stripe_width 0 application cephfs > > read_balance_score 12.50>>> How can I decrese the number of > pg_num?>> Best,> Mihai>>> On 2025-03-28 13:19, Eugen Block wrote:>> > Do you use size 1 for your data? You know that's bad, right? > Please>> share 'ceph osd pool ls detail'.>> Also, it's recommended > to use a power of 2 for pg_num, so you should>> decrease the pg_num > for the pool cephfs.cephfs.data.>>>> Zitat von Mihai Ciubancan > <mihai.ciubancan(a)eli-np.ro>:>>>>> Hi Anthony,>>> Thanks for the > answer:>>>>>> The output of 'ceph osd df' is:>>>>>> ceph osd df > tree>>> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA > OMAP >>> META AVAIL %USE VAR PGS STATUS TYPE > NAME>>> -1 167.64825 - 168 TiB 12 TiB 12 TiB > 100 MiB >>> 27 GiB 156 TiB 7.13 1.00 - root > default>>> -3 167.64825 - 168 TiB 12 TiB 12 > TiB 100 MiB >>> 27 GiB 156 TiB 7.13 1.00 - > host >>> sto-core-hpc01>>> 0 ssd 13.97069 1.00000 14 TiB > 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 > 0 up osd.0>>> 1 ssd 13.97069 1.00000 14 TiB > 12 TiB 12 TiB 6 KiB >>> 26 GiB 2.0 TiB 85.53 12.00 > 129 up osd.1>>> 2 ssd 13.97069 1.00000 14 > TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 > 0 0 up osd.2>>> 3 ssd 13.97069 1.00000 14 > TiB 1.7 GiB 258 MiB 100 MiB >>> 1.3 GiB 14 TiB 0.01 > 0.00 16 up osd.3>>> 4 ssd 13.97069 1.00000 > 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB > 0 0 0 up osd.4>>> 5 ssd 13.97069 > 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 > TiB 0 0 0 up osd.5>>> 6 ssd > 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 > MiB 14 TiB 0 0 0 up osd.6>>> 7 ssd > 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 > MiB 14 TiB 0 0 0 up osd.7>>> 8 ssd > 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB >>> 63 > MiB 14 TiB 0 0 1 up osd.8>>> 9 ssd > 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 > MiB 14 TiB 0 0 0 up osd.9>>> 10 > ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> > 28 MiB 14 TiB 0 0 0 up osd.10>>> 11 > ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB >>> > 63 MiB 14 TiB 0 0 1 up > osd.11>>> TOTAL 168 TiB 12 TiB 12 TiB > 100 MiB >>> 27 GiB 156 TiB 7.13>>>>>> So all the date is on > osd.1>>>>>> But I have checked the balancer and seems active:>>> > ceph balancer status>>> {>>> "active": true,>>> > "last_optimize_duration": "0:00:00.000368",>>> > "last_optimize_started": "Fri Mar 28 10:55:06 2025",>>> "mode": > "upmap",>>> "no_optimization_needed": false,>>> > "optimize_result": "Some objects (0.500000) are degraded; try >>> > again later",>>> "plans": []>>> }>>>>>> But the output of the > commnad 'ceph config dump|grep balancer' >>> gives me > nothing.>>>>>> Best,>>> Mihai>>>>>> On 2025-03-27 23:06, Anthony > D'Atri wrote:>>>> Look at `ceph osd df`. Is the balancer > enabled?>>>>>>>>> On Mar 27, 2025, at 8:50 AM, Mihai Ciubancan > >>>>> <mihai.ciubancan(a)eli-np.ro> wrote:>>>>>>>>>> Hello,>>>>>>>>>> > My name is Mihai, and I have started using CEPH this mount for a > >>>>> HPC cluster.>>>>> When was lunch in the production the > available space shown was >>>>> 80TB now is 16TB and I didn't do > anything, while I'm having 12 >>>>> OSD (SSD of 14TB):>>>>>>>>>> > sudo ceph osd tree>>>>> ID CLASS WEIGHT TYPE > NAME STATUS REWEIGHT PRI-AFF>>>>> -1 > 167.64825 root default>>>>> -3 167.64825 host > sto-core-hpc01>>>>> 0 ssd 13.97069 > osd.0 up 1.00000 1.00000>>>>> 1 ssd > 13.97069 osd.1 up 1.00000 1.00000>>>>> > 2 ssd 13.97069 osd.2 up 1.00000 > 1.00000>>>>> 3 ssd 13.97069 osd.3 up > 1.00000 1.00000>>>>> 4 ssd 13.97069 > osd.4 up 1.00000 1.00000>>>>> 5 ssd > 13.97069 osd.5 up 1.00000 1.00000>>>>> > 6 ssd 13.97069 osd.6 up 1.00000 > 1.00000>>>>> 7 ssd 13.97069 osd.7 up > 1.00000 1.00000>>>>> 8 ssd 13.97069 > osd.8 up 1.00000 1.00000>>>>> 9 ssd > 13.97069 osd.9 up 1.00000 1.00000>>>>> > 10 ssd 13.97069 osd.10 up 1.00000 > 1.00000>>>>> 11 ssd 13.97069 osd.11 up > 1.00000 1.00000>>>>>>>>>> sudo ceph df detail>>>>> --- RAW STORAGE > --->>>>> CLASS SIZE AVAIL USED RAW USED %RAW USED>>>>> > ssd 168 TiB 156 TiB 12 TiB 12 TiB 7.12>>>>> TOTAL 168 > TiB 156 TiB 12 TiB 12 TiB 7.12>>>>>>>>>> --- POOLS > --->>>>> POOL ID PGS STORED (DATA) (OMAP) > OBJECTS >>>>> USED (DATA) (OMAP) %USED MAX AVAIL QUOTA > OBJECTS QUOTA >>>>> BYTES DIRTY USED COMPR UNDER COMPR>>>>> > .mgr 1 1 705 KiB 705 KiB 0 B 2 > >>>>> 1.4 MiB 1.4 MiB 0 B 0 8.1 TiB > N/A >>>>> N/A N/A 0 B 0 B>>>>> > cephfs.cephfs.meta 2 16 270 MiB 270 MiB 0 B 85.96k > >>>>> 270 MiB 270 MiB 0 B 0 16 TiB > N/A >>>>> N/A N/A 0 B 0 B>>>>> > cephfs.cephfs.data 3 129 12 TiB 12 TiB 0 B 3.73M > >>>>> 12 TiB 12 TiB 0 B 42.49 16 TiB > N/A >>>>> N/A N/A 0 B 0 B>>>>>>>>>> > While on the client side I have this:>>>>>>>>>> $ df -h>>>>> > 10.18.31.1:6789:/ 21T 13T 8.1T 61% > /data>>>>>>>>>> I don't know where it's gone all the space that was > at the beginning.>>>>> Someone has any hint?>>>>>>>>>> Best > regards,>>>>> Mihai>>>>> > _______________________________________________>>>>> ceph-users > mailing list -- ceph-users(a)ceph.io>>>>> To unsubscribe send an email > to ceph-users-leave(a)ceph.io>>> > _______________________________________________>>> ceph-users > mailing list -- ceph-users(a)ceph.io>>> To unsubscribe send an email > to ceph-users-leave(a)ceph.io>>>>>> > _______________________________________________>> ceph-users mailing > list -- ceph-users(a)ceph.io>> To unsubscribe send an email to > ceph-users-leave@ceph.io_______________________________________________ceph-users mailing list -- ceph-users(a)ceph.ioTo unsubscribe send an email to > ceph-users-leave(a)ceph.io

1 0

Server not responding to keepalive - cephadm 24.04
by Reid Kelley 28 Mar '25

28 Mar '25

Setting up a new cluster on fresh ubuntu 24.04 hosts using cephadm. The first 5 hosts all added without issue, but the next N hosts all throw the same error when adding through the dashboard or through ceph orch add... All hosts have ssh access, docker and all base requirements confirmed. Command: ceph orch host add foxtrot 10.1.20.16 Error: Error EINVAL: check-host failed: Unable to write foxtrot:/var/lib/ceph/260974b7-fbb6-11ef-911a-107c6152e887/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11: Server not responding to keepalive There is a long stack trace that references asyncssh and the /lib/python3.9/site-packages/asyncssh/sftp.py file. Happy to share but not sure best way in the mailing list. Thanks for any help and advice. -Reid -- Drawbridge Digital is proud to be a certified Veteran-Owned Small Business. The information contained in this e-mail message and any attachments may be privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that any review, dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this email and delete the message and any attachments from your computer.

3 3

Debian Builds for aarch64
by Whitehouse, Dan 28 Mar '25

28 Mar '25

Hi, We are using Debian (bookworm) and have an Ampere based server which we are using for an Openstack compute node alongside some x86_64 machines. If I navigate to the repository here: https://download.ceph.com/debian-squid/pool/main/c/ceph/ I can see that there appear to be x86_64 binaries for Debian and Ubuntu. However, it looks to me like only Ubuntu binaries have been provided for arm64. Can we expect Debian bookworm binaries to be provided too (like they seem to have been for x86_64) or is aarch64 treated differently in this regard? Thanks, Dan Whitehouse

1 0

Production cluster in bad shape after several OSD crashes
by Michel Jouvin 28 Mar '25

28 Mar '25

Hi, We have a production cluster made of 3 mon+mgr, 18 OSD servers and ~500 OSDs and configured with ~50 pools, 1/2 EC (9+6) and 1/2 replica 3. It also has 2 CephFS filesystems with 1 MDS each. 2 days ago, in a period spanning 16 hours, 13 OSD crashed with an OOM. The OSD were first restarted but it was decided to reboot the server with a crashed OSD and "by mistake" (it was at least useless), the OSD of the rebooted server were set noout,norebalance before the reboot. The flags were removed after the reboot. After all of this, 'ceph -s' started to report a lot of misplaced PG and recovery started. All the PGs but one were successfully reactivated. One stayed in the activating+remapped state (located in a pool used for tests). 'ceph health' (I don't put the details here to avoid a too long mail but I can shared them) says: HEALTH_WARN 1 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow metadata IOs; Reduced data availability: 1 pg inactive; 13 daemons have recently crashed and reports about one of the filesystem being degraded despite the only PG reported inactive is not part of a pool related to the FS. The recovery was slow until we realized we should change the mclock profile to high_recovery_ops. Then it completed in a few hours. Unfortunately the degraded filesystem remains degraded without an obvious reason... and the inactive page is still in the activating+remapped state. We have not been able to identify a relevant error in the logs up to now (but we may have missed something...). So far we have avoided restarting too many things until we have a better understanding of what happened and what is the current state. We only restarted the mgr which was using a lot of CPU and the MDS for the degraded FS, without any improvement. We are looking on advices about where to start... It seems we have (at least) 2 independent problems: - A PG that cannot be reactivated with a remap operation that doesn't proceed: would stopping osd.17 help (so that osd.460 is reused)? [root@ijc-mon1 ~]# ceph pg dump_stuck PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 32.7ef activating+remapped [100,154,17] 100 [100,154,460] 100 - 1 degraded filesystem: where to look for a reason? Thanks in advance for any help? Cheers, Michel

3 10

Ceph orch placement - anti affinity
by Kasper Rasmussen 28 Mar '25

28 Mar '25

Let’s say I have 2 cephfs, and three hosts I want to use as MDS hosts. I use ceph orch apply mds to spin up the MDS daemons. Is there a way to ensure that I don’t get two active MDS running on the same host? I mean when using the ceph orch apply mds command, I can specify —placement, but it only seems like I can define the hosts, but not let’s say a primary host or something similar. Anyone with knowledge of how to do this or is it simply not possible?

2 2

Ceph orch placement anti affinity
by Kasper Rasmussen 27 Mar '25

27 Mar '25

Let’s say I have 2 cephfs, and three hosts I want to use as MDS hosts. I use ceph orch apply mds to spin up the MDS daemons. Is there a way to ensure that I don’t get two active MDS running on the same host? I mean when using the ceph orch apply mds command, I can specify —placement, but it only seems like I can define the hosts, but not let’s say a primary host or something similar. Anyone with knowledge of how to do this or is it simply not possible?

2 1

Reef: highly-available NFS with keepalive_only
by Eugen Block 26 Mar '25

26 Mar '25

Hi, I'm re-evaluating NFS again, testing on a virtual cluster with 18.2.4. For now, I don't need haproxy so I use "keepalive_only: true" as described in the docs [0]. I first create the ingress service, wait for it to start, then create the nfs cluster. I've added the specs at the bottom. I can mount the export with the virtual ip. Then I just shut down the VM where the nfs service was running, the orchestrator successfully starts a nfs daemon elsewhere, but the keepalive daemon is not failed over. So mounting or accessing the export is impossible, of course. And after I power up the offline host again, nothing is "repaired", keepalive and nfs run on different servers until I intervene manually. This doesn't seem to work as expected, is this a known issue (couldn't find anything on tracker)? I have my doubts, but maybe it works better with haproxy? Or am I missing something in my configuration? I haven't tried with a newer release yet. I'd appreciate any comments. Thanks, Eugen ---snip--- service_type: ingress service_id: nfs.ebl-nfs-cephfs service_name: ingress.nfs.ebl-nfs-cephfs placement: count: 1 hosts: - ceph01 - ceph02 - ceph03 spec: backend_service: nfs.ebl-nfs-cephfs first_virtual_router_id: 50 keepalive_only: true monitor_port: 9049 virtual_ip: 192.168.168.114/24 service_type: nfs service_id: ebl-nfs-cephfs service_name: nfs.ebl-nfs-cephfs placement: count: 1 hosts: - ceph01 - ceph02 - ceph03 spec: port: 2049 virtual_ip: 192.168.168.114 ---snip--- [0] https://docs.ceph.com/en/reef/cephadm/services/nfs/#nfs-with-virtual-ip-but…

5 13

Question about cluster expansion
by Alan Murrell 26 Mar '25

26 Mar '25

Hello, We have a 5-node cluster that each have the following drives: * 4 x 16TB HDD * 4 x 2TB NVME * 1 x 1TB NVME (for the WAL/DB for the HDDs) The nodes don't have any more room to add more NVMEs, but they do have room to add four more HDDs. I know adding more HDDs are able to make the cluster faster due to the additional IOPs. So my question is this: Is it better to: * Add the additional drives/IOPs by adding an additional node * Add the additional drives by adding the the HDDs to the existing nodes Or does it not really matter? I would prefer to add the drives to the existing nodes (ultimately maxing them out), but just wondering if that affects performance as much as expanding by adding additional nodes. Thanks! :-) Sent from my mobile device. Please excuse brevity and ttpos.

3 5

OSD failed: still recovering
by Alan Murrell 25 Mar '25

25 Mar '25

Hello, We had a drive (OSD) failed innour 5 node cluster three days ago (late afternoon of Mar 20). The PGs have sorted themselves out, but the cluster has has been recovering with backfill since then. Every time I run a 'ceph -s' it shows a little over 5% misplaced objects with several jobs of backfill_wait and some scrubbing. What is sort of weird is that if I run the 'ceph -s' as few times in a row, I can see the the percentage of misplaced objects go down a bit but then if I leave it for a while and run 'ceph -s' again, it is still just over 5% misplaced objects but has typically slightly increased. For example, it might be 5.364% when I check it, and then after checking it several times in a row it might go down to 5.276% but then if I check it again after a few hours, it might be something like 5.478% (so still in the 5% range but slightly increased from last check) The cluster is on 10Gbit, and I have increased the max_backfills to 4 while the recovery runs, but it just doesn't seem to be making much progress. I know the failed drive needs to be replaced, but I think it is recommended to wait until the cluster is finished recovering? Your thoughts/advice (as usual) are greatly appreciated. Sent from my mobile device. Please excuse brevity and ttpos.

5 6

OSD_UNREACHABLE After Upgrade to 17.2.8 – Issue with Public Network Detection
by Илья Безруков 25 Mar '25

25 Mar '25

Hello everyone, After upgrading our Ceph cluster from 17.2.7 to 17.2.8 using `cephadm`, all OSDs are reported as unreachable with the following error: ``` HEALTH_ERR 32 osds(s) are not reachable [ERR] OSD_UNREACHABLE: 32 osds(s) are not reachable osd.0's public address is not in '172.20.180.1/32,172.20.180.0/24' subnet osd.1's public address is not in '172.20.180.1/32,172.20.180.0/24' subnet ... osd.31's public address is not in '172.20.180.1/32,172.20.180.0/24' subnet ``` However, all OSDs actually have IP addresses within the `172.20.180.0/24` subnet. The cluster remains functional (CephFS is accessible), and muting the warning with `ceph health mute OSD_UNREACHABLE --sticky` allows normal operation, but the underlying issue persists. ### **Environment Details** - **Ceph Version:** 17.2.8 (f817ceb7f187defb1d021d6328fa833eb8e943b3) Quincy (stable) - **Container Image:** ` quay.io/ceph/ceph@sha256:a0f373aaaf5a5ca5c4379c09da24c771b8266a09dc9e2181f90eacf423d7326f` - **OS:** CentOS Stream 9 - **Deployment Method:** cephadm ### **Network Configuration** The network settings appear correct: ``` ceph config get osd public_network 172.20.180.0/24 ceph config get osd cluster_network 172.20.180.0/24 ceph config get mon public_network 172.20.180.0/24 ceph config get mgr public_network 172.20.180.0/24 ``` OSD metadata confirms that the assigned addresses are in the correct subnet: ``` "front_addr": "[v2: 172.20.180.126:6800/87977614,v1:172.20.180.126:6801/87977614]" ``` ### **Current Cluster Status** - All OSDs are **up** and **in**, and `ceph osd status` reports them as operational. - `ceph orch ps` confirms that OSD services are running. - `ceph mon stat` shows all monitors in quorum. - Cluster services (CephFS, RBD) are working as expected despite the warning. ### **Questions** 1. Has anyone else encountered this issue after upgrading to 17.2.8? 2. Is this a known regression? (This seems similar to issue #67517.) 3. Would upgrading to Ceph 18.x (Reef) resolve the problem? 4. Is there any solution other than muting the health warning? Any insights or recommendations would be greatly appreciated! -- Best regards, Ilya

3 4