Removing pool in nautilus is incredibly slow
Hello, I am running ceph nautilus 14.2.8 I had to remove 2 pools (old cephfs data and metadata pool with 1024 pgs). The removal of the pools seems to take a incredible time to free the space (the data pool I deleted was more than 100 TB and in 36h I got back only 10TB). In the meantime, the cluster is extremely slow (a rbd extract takes ~1h30 mn for a 32 GB image and writing 10MB in cephfs takes half a minute !!) which makes the cluster almost unusable. It seems that the removal of deleted pg is done by deep-scrubs according tohttps://medium.com/opsops/a-very-slow-pool-removal-7089e4ac8301 Also it has been reported that this could be a regression in nautilushttps://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/W4M5XQRDBLXF... But I couldn't find a fix or a way to speedup (or slow down) the process and get back the cluster to a decent reactivity. Is there a way ? Thanks F.
Hi Francois, Have you already looked at the option "osd_delete_sleep"? It will not speed up the process but I will give you some control over your cluster performance. Something like: ceph tell osd.\* injectargs '--osd_delete_sleep1' kind regards, Wout 42on On 25-06-2020 09:57, Francois Legrand wrote:
Does someone have an idea ? F. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for the hint. I tryed but it doesn't seems to change anything... Moreover, as the osds seems quite loaded I had regularly some osd marked down which triggered some new peering and thus more load !!! I set the osd no down flag, but I still have some osd reported (wrongly) as down (and back up in the minute) which generate peering and remapping. I don't really understand the action of no down parameter ! Is there a way to tell ceph not to peer immediately after an osd is reported down (let say wait for 60s) ? I am thinking about restarting all osd (or maybe the whole cluster) to get osd_op_queue_cut_off changed to high and osd_op_thread_timeout to something higher than 15 (but I don't think it will really improve the situation). F. Le 25/06/2020 à 14:26, Wout van Heeswijk a écrit :
Hi Francois,
Have you already looked at the option "osd_delete_sleep"? It will not speed up the process but I will give you some control over your cluster performance.
Something like:
ceph tell osd.\* injectargs '--osd_delete_sleep1' kind regards,
Wout 42on On 25-06-2020 09:57, Francois Legrand wrote:
Does someone have an idea ? F. _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
I'm not sure if your OSDs have their rocksDB on faster devices, if not it sounds a lot like rocksdb fragmentation [1] leading to a very high load on the OSDs and occasionally crashing OSDs. If you don't plan to delete so much data at once on a regular basis you could sit this one out, but one solution is to re-create the OSDs with rocksDB/WAL on faster devices. [1] https://www.mail-archive.com/ceph-users@ceph.io/msg03160.html Zitat von Francois Legrand <fleg@lpnhe.in2p3.fr>:
Thanks for the hint. I tryed but it doesn't seems to change anything... Moreover, as the osds seems quite loaded I had regularly some osd marked down which triggered some new peering and thus more load !!! I set the osd no down flag, but I still have some osd reported (wrongly) as down (and back up in the minute) which generate peering and remapping. I don't really understand the action of no down parameter ! Is there a way to tell ceph not to peer immediately after an osd is reported down (let say wait for 60s) ? I am thinking about restarting all osd (or maybe the whole cluster) to get osd_op_queue_cut_off changed to high and osd_op_thread_timeout to something higher than 15 (but I don't think it will really improve the situation). F.
Le 25/06/2020 à 14:26, Wout van Heeswijk a écrit :
Hi Francois,
Have you already looked at the option "osd_delete_sleep"? It will not speed up the process but I will give you some control over your cluster performance.
Something like:
ceph tell osd.\* injectargs '--osd_delete_sleep1' kind regards,
Wout 42on On 25-06-2020 09:57, Francois Legrand wrote:
Does someone have an idea ? F. _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I actually don't think this is the problem. I removed a 120TB file system EC-data pool in mimic without any special flags and magic. The OSDs of the data pool are HDD with everything collocated. I had absolutely no problem, the data was removed after 2-3 days and nobody even noticed. This is a standard operation and should just work without OPS queues running full, heartbeat losses and manual compaction or the like. Looking at all the different reports that came in on this list over the past 1-2 years about performance issues starting with nautilus, it really sounds to me that a serious regression happened. Maybe the messenger introduction? Maybe the prioritizing problem that Robert LeBlanc reported in https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/W4M5XQRDBLXF... ? I guess anyone who started with nautilus doesn't know the good old times of being able to do admin work without a completely normal cluster collapsing for no reason. Others do. I find it a bit strange that there is such a long silence on this topic. There are numerous reports of people having issues with PG changes or rebalancing. Benign operations that should just work. With our mimic cluster I have absolutely no problems migrating pools in one go to a completely new set of disks. I have no problems doubling the number of disks and at the same time doubling the number of PGs in a pooI and let the rebalancing loose in one single go. No need for slowly increasing weights. No need for slow changes of PG counts. In such cases, I casually push the recovery options up close to max available bandwidth and nobody even notices a performance drop. And all this with WAL/DB and data collocated on the same disk and with rather low RAM available, I can only afford 2GB per HDD OSD. Anyone on nautilus or higher who has the same experience? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eugen Block <eblock@nde.ag> Sent: 25 June 2020 16:42:57 To: ceph-users@ceph.io Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow I'm not sure if your OSDs have their rocksDB on faster devices, if not it sounds a lot like rocksdb fragmentation [1] leading to a very high load on the OSDs and occasionally crashing OSDs. If you don't plan to delete so much data at once on a regular basis you could sit this one out, but one solution is to re-create the OSDs with rocksDB/WAL on faster devices. [1] https://www.mail-archive.com/ceph-users@ceph.io/msg03160.html Zitat von Francois Legrand <fleg@lpnhe.in2p3.fr>:
Thanks for the hint. I tryed but it doesn't seems to change anything... Moreover, as the osds seems quite loaded I had regularly some osd marked down which triggered some new peering and thus more load !!! I set the osd no down flag, but I still have some osd reported (wrongly) as down (and back up in the minute) which generate peering and remapping. I don't really understand the action of no down parameter ! Is there a way to tell ceph not to peer immediately after an osd is reported down (let say wait for 60s) ? I am thinking about restarting all osd (or maybe the whole cluster) to get osd_op_queue_cut_off changed to high and osd_op_thread_timeout to something higher than 15 (but I don't think it will really improve the situation). F.
Le 25/06/2020 à 14:26, Wout van Heeswijk a écrit :
Hi Francois,
Have you already looked at the option "osd_delete_sleep"? It will not speed up the process but I will give you some control over your cluster performance.
Something like:
ceph tell osd.\* injectargs '--osd_delete_sleep1' kind regards,
Wout 42on On 25-06-2020 09:57, Francois Legrand wrote:
Does someone have an idea ? F. _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frank, "With our mimic cluster I have absolutely no problems migrating pools in one go to a completely new set of disks. I have no problems doubling the number of disks and at the same time doubling the number of PGs in a pooI and let the rebalancing loose in one single go. No need for slowly increasing weights. No need for slow changes of PG counts. In such cases,..." This is also my experience. I have 2 clusters running on Nautilus 14.2.8, one upgraded 2 weeks ago from mimic. I do NOT see any performance drop from the client side. But recovering is extremely slow, after replacing a defect OSD. When I need to replace an OSD, I destroy them, turn on the noout flag, turn off the server, replace the disk, and turn the server on. All within 30min. In Mimic I had only some misplaced objects and it recovered within an hour. In Nautilis, when I do exactly the same, I get beside misplaced objects, also degraded PGs and undersized PGs, and the recovery takes almost a day. I still need to investigate this (tips are welcome ;) ) But what is standing out, is the load on the manager. Grtz, Jiri On Thu, 25 Jun 2020 at 17:18, Frank Schilder <frans@dtu.dk> wrote:
I actually don't think this is the problem. I removed a 120TB file system EC-data pool in mimic without any special flags and magic. The OSDs of the data pool are HDD with everything collocated. I had absolutely no problem, the data was removed after 2-3 days and nobody even noticed. This is a standard operation and should just work without OPS queues running full, heartbeat losses and manual compaction or the like.
Looking at all the different reports that came in on this list over the past 1-2 years about performance issues starting with nautilus, it really sounds to me that a serious regression happened. Maybe the messenger introduction? Maybe the prioritizing problem that Robert LeBlanc reported in https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/W4M5XQRDBLXF... ?
I guess anyone who started with nautilus doesn't know the good old times of being able to do admin work without a completely normal cluster collapsing for no reason. Others do. I find it a bit strange that there is such a long silence on this topic. There are numerous reports of people having issues with PG changes or rebalancing. Benign operations that should just work.
With our mimic cluster I have absolutely no problems migrating pools in one go to a completely new set of disks. I have no problems doubling the number of disks and at the same time doubling the number of PGs in a pooI and let the rebalancing loose in one single go. No need for slowly increasing weights. No need for slow changes of PG counts. In such cases, I casually push the recovery options up close to max available bandwidth and nobody even notices a performance drop. And all this with WAL/DB and data collocated on the same disk and with rather low RAM available, I can only afford 2GB per HDD OSD.
Anyone on nautilus or higher who has the same experience?
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Eugen Block <eblock@nde.ag> Sent: 25 June 2020 16:42:57 To: ceph-users@ceph.io Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow
I'm not sure if your OSDs have their rocksDB on faster devices, if not it sounds a lot like rocksdb fragmentation [1] leading to a very high load on the OSDs and occasionally crashing OSDs. If you don't plan to delete so much data at once on a regular basis you could sit this one out, but one solution is to re-create the OSDs with rocksDB/WAL on faster devices.
[1] https://www.mail-archive.com/ceph-users@ceph.io/msg03160.html
Zitat von Francois Legrand <fleg@lpnhe.in2p3.fr>:
Thanks for the hint. I tryed but it doesn't seems to change anything... Moreover, as the osds seems quite loaded I had regularly some osd marked down which triggered some new peering and thus more load !!! I set the osd no down flag, but I still have some osd reported (wrongly) as down (and back up in the minute) which generate peering and remapping. I don't really understand the action of no down parameter ! Is there a way to tell ceph not to peer immediately after an osd is reported down (let say wait for 60s) ? I am thinking about restarting all osd (or maybe the whole cluster) to get osd_op_queue_cut_off changed to high and osd_op_thread_timeout to something higher than 15 (but I don't think it will really improve the situation). F.
Le 25/06/2020 à 14:26, Wout van Heeswijk a écrit :
Hi Francois,
Have you already looked at the option "osd_delete_sleep"? It will not speed up the process but I will give you some control over your cluster performance.
Something like:
ceph tell osd.\* injectargs '--osd_delete_sleep1' kind regards,
Wout 42on On 25-06-2020 09:57, Francois Legrand wrote:
Does someone have an idea ? F. _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Jiri, this doesn't sound too bad. I don't know if the recovery time is to be expected, how does it compare to the same operation on mimic with the same utilization? In any case, single disk recovery is slow and I plan not to do it. Replacements are installed together with upgrades to have all-to-all instead of all-to-one recovery. Assuming that you replaced the disk after all objects were recovered (cluster health ok), then degraded objects and undersized PGs should not occur. What exactly was the sequence of these events: - disk fails - ceph shows health warn - PGs peer - ceph starts recovery - recovery completes - ceph shows health ok - server shut down - disk is exchanged - server boot - ceph health ok - new OSD deployed - degraded objects and undersized PGs show up If the order was exactly as listed here, something serious is wrong. The MGR load problem seems to be a known issue and due to interpreted (slow!) code being executed in a high-frequency call back. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Jiri D. Hoogeveen <wica128@gmail.com> Sent: 25 June 2020 17:44:23 To: Frank Schilder Cc: Eugen Block; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Removing pool in nautilus is incredibly slow Hi Frank, "With our mimic cluster I have absolutely no problems migrating pools in one go to a completely new set of disks. I have no problems doubling the number of disks and at the same time doubling the number of PGs in a pooI and let the rebalancing loose in one single go. No need for slowly increasing weights. No need for slow changes of PG counts. In such cases,..." This is also my experience. I have 2 clusters running on Nautilus 14.2.8, one upgraded 2 weeks ago from mimic. I do NOT see any performance drop from the client side. But recovering is extremely slow, after replacing a defect OSD. When I need to replace an OSD, I destroy them, turn on the noout flag, turn off the server, replace the disk, and turn the server on. All within 30min. In Mimic I had only some misplaced objects and it recovered within an hour. In Nautilis, when I do exactly the same, I get beside misplaced objects, also degraded PGs and undersized PGs, and the recovery takes almost a day. I still need to investigate this (tips are welcome ;) ) But what is standing out, is the load on the manager. Grtz, Jiri On Thu, 25 Jun 2020 at 17:18, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: I actually don't think this is the problem. I removed a 120TB file system EC-data pool in mimic without any special flags and magic. The OSDs of the data pool are HDD with everything collocated. I had absolutely no problem, the data was removed after 2-3 days and nobody even noticed. This is a standard operation and should just work without OPS queues running full, heartbeat losses and manual compaction or the like. Looking at all the different reports that came in on this list over the past 1-2 years about performance issues starting with nautilus, it really sounds to me that a serious regression happened. Maybe the messenger introduction? Maybe the prioritizing problem that Robert LeBlanc reported in https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/W4M5XQRDBLXF... ? I guess anyone who started with nautilus doesn't know the good old times of being able to do admin work without a completely normal cluster collapsing for no reason. Others do. I find it a bit strange that there is such a long silence on this topic. There are numerous reports of people having issues with PG changes or rebalancing. Benign operations that should just work. With our mimic cluster I have absolutely no problems migrating pools in one go to a completely new set of disks. I have no problems doubling the number of disks and at the same time doubling the number of PGs in a pooI and let the rebalancing loose in one single go. No need for slowly increasing weights. No need for slow changes of PG counts. In such cases, I casually push the recovery options up close to max available bandwidth and nobody even notices a performance drop. And all this with WAL/DB and data collocated on the same disk and with rather low RAM available, I can only afford 2GB per HDD OSD. Anyone on nautilus or higher who has the same experience? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eugen Block <eblock@nde.ag<mailto:eblock@nde.ag>> Sent: 25 June 2020 16:42:57 To: ceph-users@ceph.io<mailto:ceph-users@ceph.io> Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow I'm not sure if your OSDs have their rocksDB on faster devices, if not it sounds a lot like rocksdb fragmentation [1] leading to a very high load on the OSDs and occasionally crashing OSDs. If you don't plan to delete so much data at once on a regular basis you could sit this one out, but one solution is to re-create the OSDs with rocksDB/WAL on faster devices. [1] https://www.mail-archive.com/ceph-users@ceph.io/msg03160.html Zitat von Francois Legrand <fleg@lpnhe.in2p3.fr<mailto:fleg@lpnhe.in2p3.fr>>:
Thanks for the hint. I tryed but it doesn't seems to change anything... Moreover, as the osds seems quite loaded I had regularly some osd marked down which triggered some new peering and thus more load !!! I set the osd no down flag, but I still have some osd reported (wrongly) as down (and back up in the minute) which generate peering and remapping. I don't really understand the action of no down parameter ! Is there a way to tell ceph not to peer immediately after an osd is reported down (let say wait for 60s) ? I am thinking about restarting all osd (or maybe the whole cluster) to get osd_op_queue_cut_off changed to high and osd_op_thread_timeout to something higher than 15 (but I don't think it will really improve the situation). F.
Le 25/06/2020 à 14:26, Wout van Heeswijk a écrit :
Hi Francois,
Have you already looked at the option "osd_delete_sleep"? It will not speed up the process but I will give you some control over your cluster performance.
Something like:
ceph tell osd.\* injectargs '--osd_delete_sleep1' kind regards,
Wout 42on On 25-06-2020 09:57, Francois Legrand wrote:
Does someone have an idea ? F. _______________________________________________ ceph-users mailing list --ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email toceph-users-leave@ceph.io<mailto:toceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
I also had this kind of symptoms with nautilus. Replacing a failed disk (from cluster ok) generates degraded objects. Also, we have a proxmox cluster accessing vm images stored in our ceph storage with rbd. Each time I had some operation on the ceph cluster like adding or removing a pool, most of our proxmox vms lost contact with their system disk in ceph and crashed (or remount system storage in read-only mode). At first I thought it was a network problem, but now I am sure that it's related to ceph becoming unresponsive during background operations. For now, proxmox cannot even access ceph storage using rbd (it fails with timeout).
OK, this *does* sound bad. I would consider this a show stopper for upgrade from mimic. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Francois Legrand <fleg@lpnhe.in2p3.fr> Sent: 25 June 2020 19:25:14 To: ceph-users@ceph.io Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow I also had this kind of symptoms with nautilus. Replacing a failed disk (from cluster ok) generates degraded objects. Also, we have a proxmox cluster accessing vm images stored in our ceph storage with rbd. Each time I had some operation on the ceph cluster like adding or removing a pool, most of our proxmox vms lost contact with their system disk in ceph and crashed (or remount system storage in read-only mode). At first I thought it was a network problem, but now I am sure that it's related to ceph becoming unresponsive during background operations. For now, proxmox cannot even access ceph storage using rbd (it fails with timeout). _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
For sure, If I could downgrade to mimic I would probably do it !!! So I understand that you plan not to upgrade ! F. Le 25/06/2020 à 19:28, Frank Schilder a écrit :
OK, this *does* sound bad. I would consider this a show stopper for upgrade from mimic.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Francois Legrand <fleg@lpnhe.in2p3.fr> Sent: 25 June 2020 19:25:14 To: ceph-users@ceph.io Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow
I also had this kind of symptoms with nautilus. Replacing a failed disk (from cluster ok) generates degraded objects. Also, we have a proxmox cluster accessing vm images stored in our ceph storage with rbd. Each time I had some operation on the ceph cluster like adding or removing a pool, most of our proxmox vms lost contact with their system disk in ceph and crashed (or remount system storage in read-only mode). At first I thought it was a network problem, but now I am sure that it's related to ceph becoming unresponsive during background operations. For now, proxmox cannot even access ceph storage using rbd (it fails with timeout). _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We are now using osd_op_queue = wpq. Maybe returning to prio should help ? What are you using on your mimic custer ? F. Le 25/06/2020 à 19:28, Frank Schilder a écrit :
OK, this *does* sound bad. I would consider this a show stopper for upgrade from mimic.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Francois Legrand <fleg@lpnhe.in2p3.fr> Sent: 25 June 2020 19:25:14 To: ceph-users@ceph.io Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow
I also had this kind of symptoms with nautilus. Replacing a failed disk (from cluster ok) generates degraded objects. Also, we have a proxmox cluster accessing vm images stored in our ceph storage with rbd. Each time I had some operation on the ceph cluster like adding or removing a pool, most of our proxmox vms lost contact with their system disk in ceph and crashed (or remount system storage in read-only mode). At first I thought it was a network problem, but now I am sure that it's related to ceph becoming unresponsive during background operations. For now, proxmox cannot even access ceph storage using rbd (it fails with timeout). _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'm using osd_op_queue = wpq osd_op_queue_cut_off = high and these settings are recommended. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Francois Legrand <fleg@lpnhe.in2p3.fr> Sent: 26 June 2020 09:44:00 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Removing pool in nautilus is incredibly slow We are now using osd_op_queue = wpq. Maybe returning to prio should help ? What are you using on your mimic custer ? F. Le 25/06/2020 à 19:28, Frank Schilder a écrit :
OK, this *does* sound bad. I would consider this a show stopper for upgrade from mimic.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Francois Legrand <fleg@lpnhe.in2p3.fr> Sent: 25 June 2020 19:25:14 To: ceph-users@ceph.io Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow
I also had this kind of symptoms with nautilus. Replacing a failed disk (from cluster ok) generates degraded objects. Also, we have a proxmox cluster accessing vm images stored in our ceph storage with rbd. Each time I had some operation on the ceph cluster like adding or removing a pool, most of our proxmox vms lost contact with their system disk in ceph and crashed (or remount system storage in read-only mode). At first I thought it was a network problem, but now I am sure that it's related to ceph becoming unresponsive during background operations. For now, proxmox cannot even access ceph storage using rbd (it fails with timeout). _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks. I will try to change osd_op_queue_cut_off to high and restart everything (and use this downtime to upgrade the servers). F. Le 26/06/2020 à 09:46, Frank Schilder a écrit :
I'm using
osd_op_queue = wpq osd_op_queue_cut_off = high
and these settings are recommended.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Francois Legrand <fleg@lpnhe.in2p3.fr> Sent: 26 June 2020 09:44:00 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Removing pool in nautilus is incredibly slow
We are now using osd_op_queue = wpq. Maybe returning to prio should help ? What are you using on your mimic custer ? F.
Le 25/06/2020 à 19:28, Frank Schilder a écrit :
OK, this *does* sound bad. I would consider this a show stopper for upgrade from mimic.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Francois Legrand <fleg@lpnhe.in2p3.fr> Sent: 25 June 2020 19:25:14 To: ceph-users@ceph.io Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow
I also had this kind of symptoms with nautilus. Replacing a failed disk (from cluster ok) generates degraded objects. Also, we have a proxmox cluster accessing vm images stored in our ceph storage with rbd. Each time I had some operation on the ceph cluster like adding or removing a pool, most of our proxmox vms lost contact with their system disk in ceph and crashed (or remount system storage in read-only mode). At first I thought it was a network problem, but now I am sure that it's related to ceph becoming unresponsive during background operations. For now, proxmox cannot even access ceph storage using rbd (it fails with timeout). _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I changed osd_op_queue_cut_off to high and rebooted all the osds. But the result is more or less the same (storage is still extremely slow, 2h30 to rdb extract a 64GB image !). The only improvement is that it seems that degraded pgs have disapeared (which is at least a good point). It seems that there is a problem in priority of operations. Thus do you think (and also others on the list) that changing the osd_op_queue setting could help (change to prio or mclock_client). What are the risks or secondary effects of trying mclock_client on a production cluster (is it safe) ? F. Le 26/06/2020 à 09:46, Frank Schilder a écrit :
I'm using
osd_op_queue = wpq osd_op_queue_cut_off = high
and these settings are recommended.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Francois Legrand <fleg@lpnhe.in2p3.fr> Sent: 26 June 2020 09:44:00 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Removing pool in nautilus is incredibly slow
We are now using osd_op_queue = wpq. Maybe returning to prio should help ? What are you using on your mimic custer ? F.
Le 25/06/2020 à 19:28, Frank Schilder a écrit :
OK, this *does* sound bad. I would consider this a show stopper for upgrade from mimic.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Francois Legrand <fleg@lpnhe.in2p3.fr> Sent: 25 June 2020 19:25:14 To: ceph-users@ceph.io Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow
I also had this kind of symptoms with nautilus. Replacing a failed disk (from cluster ok) generates degraded objects. Also, we have a proxmox cluster accessing vm images stored in our ceph storage with rbd. Each time I had some operation on the ceph cluster like adding or removing a pool, most of our proxmox vms lost contact with their system disk in ceph and crashed (or remount system storage in read-only mode). At first I thought it was a network problem, but now I am sure that it's related to ceph becoming unresponsive during background operations. For now, proxmox cannot even access ceph storage using rbd (it fails with timeout). _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I never tried "prio" out, but the reports I have seen claim that prio is inferior. However, as far as I know it is safe to change these settings. Unfortunately, you need to restart services to apply the changes. Before you do, check if *all* daemons are using the same setting. Contrary to the naming (osd_*), this setting applies to all daemons. I added it to the global options and, most notably, performance of the MDS was improved a lot. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Francois Legrand <fleg@lpnhe.in2p3.fr> Sent: 26 June 2020 15:03:23 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Removing pool in nautilus is incredibly slow I changed osd_op_queue_cut_off to high and rebooted all the osds. But the result is more or less the same (storage is still extremely slow, 2h30 to rdb extract a 64GB image !). The only improvement is that it seems that degraded pgs have disapeared (which is at least a good point). It seems that there is a problem in priority of operations. Thus do you think (and also others on the list) that changing the osd_op_queue setting could help (change to prio or mclock_client). What are the risks or secondary effects of trying mclock_client on a production cluster (is it safe) ? F. Le 26/06/2020 à 09:46, Frank Schilder a écrit :
I'm using
osd_op_queue = wpq osd_op_queue_cut_off = high
and these settings are recommended.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Francois Legrand <fleg@lpnhe.in2p3.fr> Sent: 26 June 2020 09:44:00 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Removing pool in nautilus is incredibly slow
We are now using osd_op_queue = wpq. Maybe returning to prio should help ? What are you using on your mimic custer ? F.
Le 25/06/2020 à 19:28, Frank Schilder a écrit :
OK, this *does* sound bad. I would consider this a show stopper for upgrade from mimic.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Francois Legrand <fleg@lpnhe.in2p3.fr> Sent: 25 June 2020 19:25:14 To: ceph-users@ceph.io Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow
I also had this kind of symptoms with nautilus. Replacing a failed disk (from cluster ok) generates degraded objects. Also, we have a proxmox cluster accessing vm images stored in our ceph storage with rbd. Each time I had some operation on the ceph cluster like adding or removing a pool, most of our proxmox vms lost contact with their system disk in ceph and crashed (or remount system storage in read-only mode). At first I thought it was a network problem, but now I am sure that it's related to ceph becoming unresponsive during background operations. For now, proxmox cannot even access ceph storage using rbd (it fails with timeout). _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks. I also added osd_op_queue_cut_off to high in global (as you mentioned in a previous thread that osd and mds should use it). F. Le 26/06/2020 à 16:35, Frank Schilder a écrit :
I never tried "prio" out, but the reports I have seen claim that prio is inferior.
However, as far as I know it is safe to change these settings. Unfortunately, you need to restart services to apply the changes.
Before you do, check if *all* daemons are using the same setting. Contrary to the naming (osd_*), this setting applies to all daemons. I added it to the global options and, most notably, performance of the MDS was improved a lot.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Francois Legrand <fleg@lpnhe.in2p3.fr> Sent: 26 June 2020 15:03:23 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Removing pool in nautilus is incredibly slow
I changed osd_op_queue_cut_off to high and rebooted all the osds. But the result is more or less the same (storage is still extremely slow, 2h30 to rdb extract a 64GB image !). The only improvement is that it seems that degraded pgs have disapeared (which is at least a good point). It seems that there is a problem in priority of operations. Thus do you think (and also others on the list) that changing the osd_op_queue setting could help (change to prio or mclock_client). What are the risks or secondary effects of trying mclock_client on a production cluster (is it safe) ? F.
Le 26/06/2020 à 09:46, Frank Schilder a écrit :
I'm using
osd_op_queue = wpq osd_op_queue_cut_off = high
and these settings are recommended.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Francois Legrand <fleg@lpnhe.in2p3.fr> Sent: 26 June 2020 09:44:00 To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Removing pool in nautilus is incredibly slow
We are now using osd_op_queue = wpq. Maybe returning to prio should help ? What are you using on your mimic custer ? F.
Le 25/06/2020 à 19:28, Frank Schilder a écrit :
OK, this *does* sound bad. I would consider this a show stopper for upgrade from mimic.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Francois Legrand <fleg@lpnhe.in2p3.fr> Sent: 25 June 2020 19:25:14 To: ceph-users@ceph.io Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow
I also had this kind of symptoms with nautilus. Replacing a failed disk (from cluster ok) generates degraded objects. Also, we have a proxmox cluster accessing vm images stored in our ceph storage with rbd. Each time I had some operation on the ceph cluster like adding or removing a pool, most of our proxmox vms lost contact with their system disk in ceph and crashed (or remount system storage in read-only mode). At first I thought it was a network problem, but now I am sure that it's related to ceph becoming unresponsive during background operations. For now, proxmox cannot even access ceph storage using rbd (it fails with timeout). _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Do you do things like[1] with the vm's? [1] echo 120 > /sys/block/sda/device/timeout -----Original Message----- From: Francois Legrand [mailto:fleg@lpnhe.in2p3.fr] Sent: donderdag 25 juni 2020 19:25 To: ceph-users@ceph.io Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow I also had this kind of symptoms with nautilus. Replacing a failed disk (from cluster ok) generates degraded objects. Also, we have a proxmox cluster accessing vm images stored in our ceph storage with rbd. Each time I had some operation on the ceph cluster like adding or removing a pool, most of our proxmox vms lost contact with their system disk in ceph and crashed (or remount system storage in read-only mode). At first I thought it was a network problem, but now I am sure that it's related to ceph becoming unresponsive during background operations. For now, proxmox cannot even access ceph storage using rbd (it fails with timeout). _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 26/06/2020 1:44 am, Jiri D. Hoogeveen wrote:
In Mimic I had only some misplaced objects and it recovered within an hour. In Nautilis, when I do exactly the same, I get beside misplaced objects, also degraded PGs and undersized PGs, and the recovery takes almost a day.
Slowness of recovery aside, isn't degraded PG's the correct behavior? You've removed an OSD and a bunch of objects replica counts are now below the size setting. -- Lindsay
I think he means that after disk failure he waits for the cluster to get back to ok (so all data on the lost disk have been reconstructed elsewhere) and then the disk is changed. In that case it's normal to have misplaced objects (because with the new disk some pgs needs to be migrated to populate this new space), but degraded pg does not seems to be the good behaviour !
This depends on which point in the procedure you refer to. He explicitly wrote
Note, we have not deployed the new OSD jet.
meaning he observed misplaced objects before deploying the new disk. This should not happen. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Francois Legrand <fleg@lpnhe.in2p3.fr> Sent: 26 June 2020 09:27:33 To: ceph-users@ceph.io Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow I think he means that after disk failure he waits for the cluster to get back to ok (so all data on the lost disk have been reconstructed elsewhere) and then the disk is changed. In that case it's normal to have misplaced objects (because with the new disk some pgs needs to be migrated to populate this new space), but degraded pg does not seems to be the good behaviour ! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 26/06/2020 5:27 pm, Francois Legrand wrote:
In that case it's normal to have misplaced objects (because with the new disk some pgs needs to be migrated to populate this new space), but degraded pg does not seems to be the good behaviour !
Yes, that would be bad, not sure if thats the procedure being used though. -- Lindsay
participants (7)
-
Eugen Block
-
Francois Legrand
-
Frank Schilder
-
Jiri D. Hoogeveen
-
Lindsay Mathieson
-
Marc Roos
-
Wout van Heeswijk