19 Apr
2024
19 Apr
'24
1:19 a.m.
Hi,
Trying to delete images in a Ceph pool is causing errors in one of
the clusters. I rebooted all the monitor nodes sequentially to see if the
error went away, but it still persists. What is the best way to fix this?
The Ceph cluster is in an OK state, with no rebalancing or scrubbing
happening (I did set the noscrub and deep-noscrub flags) and also no load
on the cluster, very few IO.
root@ceph-mon01 ~# rbd rm 000dca3d-4f2b-4033-b8f5-95458e0c3444_disk_delete
-p compute
Removing image: 31% complete...2024-04-18 20:42:52.525135 7f6de0c79700 -1
NetHandler create_socket couldn't create socket (24) Too many open files
Removing image: 32% complete...2024-04-18 20:42:52.539882 7f6de9c7b700 -1
NetHandler create_socket couldn't create socket (24) Too many open files
2024-04-18 20:42:52.541508 7f6de947a700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.546613 7f6de0c79700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.558133 7f6de9c7b700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.573819 7f6de947a700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.589733 7f6de0c79700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
Removing image: 33% complete...2024-04-18 20:42:52.643489 7f6de9c7b700 -1
NetHandler create_socket couldn't create socket (24) Too many open files
2024-04-18 20:42:52.727262 7f6de0c79700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.737135 7f6de9c7b700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.743292 7f6de947a700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.746167 7f6de0c79700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.757404 7f6de9c7b700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
Removing image: 34% complete...2024-04-18 20:42:52.773182 7f6de947a700 -1
NetHandler create_socket couldn't create socket (24) Too many open files
2024-04-18 20:42:52.773222 7f6de947a700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.789847 7f6de0c79700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.844201 7f6de9c7b700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
^C
root@ceph-mon01 ~#
Thanks,
Pardh
19 Apr
19 Apr
11:06 a.m.
Hi,
Your shell seems reached the default file discriptors limit (1024 mostly) and your cluster
maybe more than 1000 OSD
Try to set command `ulimit -n 10240` before rbd rm task
k
Sent from my iPhone
On 18 Apr 2024, at 23:50, Pardhiv Karri
<meher4india(a)gmail.com> wrote:
Hi,
Trying to delete images in a Ceph pool is causing errors in one of
the clusters. I rebooted all the monitor nodes sequentially to see if the
error went away, but it still persists. What is the best way to fix this?
The Ceph cluster is in an OK state, with no rebalancing or scrubbing
happening (I did set the noscrub and deep-noscrub flags) and also no load
on the cluster, very few IO.
root@ceph-mon01 ~# rbd rm 000dca3d-4f2b-4033-b8f5-95458e0c3444_disk_delete
-p compute
Removing image: 31% complete...2024-04-18 20:42:52.525135 7f6de0c79700 -1
NetHandler create_socket couldn't create socket (24) Too many open files
Removing image: 32% complete...2024-04-18 20:42:52.539882 7f6de9c7b700 -1
NetHandler create_socket couldn't create socket (24) Too many open files
2024-04-18 20:42:52.541508 7f6de947a700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.546613 7f6de0c79700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.558133 7f6de9c7b700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.573819 7f6de947a700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.589733 7f6de0c79700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
Removing image: 33% complete...2024-04-18 20:42:52.643489 7f6de9c7b700 -1
NetHandler create_socket couldn't create socket (24) Too many open files
2024-04-18 20:42:52.727262 7f6de0c79700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.737135 7f6de9c7b700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.743292 7f6de947a700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.746167 7f6de0c79700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.757404 7f6de9c7b700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
Removing image: 34% complete...2024-04-18 20:42:52.773182 7f6de947a700 -1
NetHandler create_socket couldn't create socket (24) Too many open files
2024-04-18 20:42:52.773222 7f6de947a700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.789847 7f6de0c79700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.844201 7f6de9c7b700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
^C
root@ceph-mon01 ~#
Thanks,
Pardh
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
12:09 p.m.
Hi Konstantin,
Thank you for the reply. I tried setting ulimit to 32768 when I saw 25726
number in lsof output and then after 2 disks deletion again it got an error
and checked lsof and which is above 35000. I'm not sure how to handle it.
I rebooted the monitor node, but the open files kept growing.
root@ceph-mon01 ~# lsof | wc -l
49296
root@ceph-mon01 ~#
Thanks,
Pardh
On Thu, Apr 18, 2024 at 11:36 PM Konstantin Shalygin <k0ste(a)k0ste.ru> wrote:
Hi,
Your shell seems reached the default file discriptors limit (1024 mostly)
and your cluster maybe more than 1000 OSD
Try to set command `ulimit -n 10240` before rbd rm task
k
Sent from my iPhone
--
*Pardhiv Karri*
"Rise and Rise again until LAMBS become LIONS"
On 18 Apr 2024, at 23:50, Pardhiv Karri
<meher4india(a)gmail.com> wrote:
Hi,
Trying to delete images in a Ceph pool is causing errors in one of
the clusters. I rebooted all the monitor nodes sequentially to see if the
error went away, but it still persists. What is the best way to fix this?
The Ceph cluster is in an OK state, with no rebalancing or scrubbing
happening (I did set the noscrub and deep-noscrub flags) and also no load
on the cluster, very few IO.
root@ceph-mon01 ~# rbd rm
000dca3d-4f2b-4033-b8f5-95458e0c3444_disk_delete
-p compute
Removing image: 31% complete...2024-04-18 20:42:52.525135 7f6de0c79700 -1
NetHandler create_socket couldn't create socket (24) Too many open files
Removing image: 32% complete...2024-04-18 20:42:52.539882 7f6de9c7b700 -1
NetHandler create_socket couldn't create socket (24) Too many open files
2024-04-18 20:42:52.541508 7f6de947a700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.546613 7f6de0c79700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.558133 7f6de9c7b700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.573819 7f6de947a700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.589733 7f6de0c79700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
Removing image: 33% complete...2024-04-18 20:42:52.643489 7f6de9c7b700 -1
NetHandler create_socket couldn't create socket (24) Too many open files
2024-04-18 20:42:52.727262 7f6de0c79700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.737135 7f6de9c7b700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.743292 7f6de947a700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.746167 7f6de0c79700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.757404 7f6de9c7b700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
Removing image: 34% complete...2024-04-18 20:42:52.773182 7f6de947a700 -1
NetHandler create_socket couldn't create socket (24) Too many open files
2024-04-18 20:42:52.773222 7f6de947a700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.789847 7f6de0c79700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
2024-04-18 20:42:52.844201 7f6de9c7b700 -1 NetHandler create_socket
couldn't create socket (24) Too many open files
^C
root@ceph-mon01 ~#
Thanks,
Pardh
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:10 p.m.
Hi,
On 19 Apr 2024, at 10:39, Pardhiv Karri
<meher4india(a)gmail.com> wrote:
Thank you for the reply. I tried setting ulimit to 32768 when I saw 25726
number in lsof output and then after 2 disks deletion again it got an error
and checked lsof and which is above 35000. I'm not sure how to handle it.
I rebooted the monitor node, but the open files kept growing.
root@ceph-mon01 ~# lsof | wc -l
49296
root@ceph-mon01 ~#
This means that is not a Ceph problem. Is a problem in this system at all
k
6:35 p.m.
With cephadm you're able to set these values cluster wide.
See the host-management section of the docs.
https://docs.ceph.com/en/reef/cephadm/host-management/#os-tuning-profiles
On Fri, 19 Apr 2024 at 12:40, Konstantin Shalygin <k0ste(a)k0ste.ru> wrote:
Hi,
On 19 Apr 2024, at 10:39, Pardhiv Karri
<meher4india(a)gmail.com> wrote:
Thank you for the reply. I tried setting ulimit to 32768 when I saw 25726
number in lsof output and then after 2 disks deletion again it got an
error
and checked lsof and which is above 35000.
I'm not sure how to handle
it.
I rebooted the monitor node, but the open files
kept growing.
root@ceph-mon01 ~# lsof | wc -l
49296
root@ceph-mon01 ~#
This means that is not a Ceph problem. Is a problem in this system at all
k
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
37
days inactive
38
days old
4 comments
3 participants
participants (3)
-
Konstantin Shalygin -
P Wagner-Beccard -
Pardhiv Karri