Hi Konstantin, Thank you for the reply. I tried setting ulimit to 32768 when I saw 25726 number in lsof output and then after 2 disks deletion again it got an error and checked lsof and which is above 35000. I'm not sure how to handle it. I rebooted the monitor node, but the open files kept growing. root@ceph-mon01 ~# lsof | wc -l 49296 root@ceph-mon01 ~# Thanks, Pardh On Thu, Apr 18, 2024 at 11:36 PM Konstantin Shalygin <k0ste@k0ste.ru> wrote:

Hi,

Your shell seems reached the default file discriptors limit (1024 mostly) and your cluster maybe more than 1000 OSD

Try to set command `ulimit -n 10240` before rbd rm task

k Sent from my iPhone

...

On 18 Apr 2024, at 23:50, Pardhiv Karri <meher4india@gmail.com> wrote:

Hi,

Trying to delete images in a Ceph pool is causing errors in one of the clusters. I rebooted all the monitor nodes sequentially to see if the error went away, but it still persists. What is the best way to fix this? The Ceph cluster is in an OK state, with no rebalancing or scrubbing happening (I did set the noscrub and deep-noscrub flags) and also no load on the cluster, very few IO.

root@ceph-mon01 ~# rbd rm 000dca3d-4f2b-4033-b8f5-95458e0c3444_disk_delete -p compute Removing image: 31% complete...2024-04-18 20:42:52.525135 7f6de0c79700 -1 NetHandler create_socket couldn't create socket (24) Too many open files Removing image: 32% complete...2024-04-18 20:42:52.539882 7f6de9c7b700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.541508 7f6de947a700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.546613 7f6de0c79700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.558133 7f6de9c7b700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.573819 7f6de947a700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.589733 7f6de0c79700 -1 NetHandler create_socket couldn't create socket (24) Too many open files Removing image: 33% complete...2024-04-18 20:42:52.643489 7f6de9c7b700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.727262 7f6de0c79700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.737135 7f6de9c7b700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.743292 7f6de947a700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.746167 7f6de0c79700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.757404 7f6de9c7b700 -1 NetHandler create_socket couldn't create socket (24) Too many open files Removing image: 34% complete...2024-04-18 20:42:52.773182 7f6de947a700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.773222 7f6de947a700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.789847 7f6de0c79700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.844201 7f6de9c7b700 -1 NetHandler create_socket couldn't create socket (24) Too many open files

^C root@ceph-mon01 ~#

Thanks, Pardh _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- *Pardhiv Karri* "Rise and Rise again until LAMBS become LIONS"

With cephadm you're able to set these values cluster wide. See the host-management section of the docs. https://docs.ceph.com/en/reef/cephadm/host-management/#os-tuning-profiles On Fri, 19 Apr 2024 at 12:40, Konstantin Shalygin <k0ste@k0ste.ru> wrote:

Hi,

...

On 19 Apr 2024, at 10:39, Pardhiv Karri <meher4india@gmail.com> wrote:

Thank you for the reply. I tried setting ulimit to 32768 when I saw 25726 number in lsof output and then after 2 disks deletion again it got an error and checked lsof and which is above 35000. I'm not sure how to handle it. I rebooted the monitor node, but the open files kept growing.

root@ceph-mon01 ~# lsof | wc -l 49296 root@ceph-mon01 ~#

This means that is not a Ceph problem. Is a problem in this system at all

k _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io