Hi all, We have a Ceph cluster which has been expanded from 10 to 16 nodes. Each node has between 14 and 16 OSDs of which 2 are NVMe disks. Most disks (except NVMe's) are 16TB large. The expansion of 16 nodes went ok, but we've configured the system to prevent auto balance towards the new disks (weight was set to 0) so we could control the expansion. We started adding 6 disks last week (1 disk on each new node) which didn't give a lot of issues. When the Ceph status indicated the PG degraded was almost finished, we've added 2 disks on each node again. All seemed to go fine, till yesterday morning... IOs towards the system were slowing down. Diving onto the nodes we could see that the OSD daemons are consuming the CPU power, resulting in average CPU loads going near 10 (!). The RGWs nor monitors nor other involved servers are having CPU issues (except for the management server which is fighting with Prometheus), so it's latency seems to be related to the ODS hosts. All of the hosts are interconnected with 25Gbit connections, no bottlenecks are reached on the network either. Important piece of information: We are using erasure coding (6/3), and we do have a lot of small files... The current health detail indicates degraded health redundancy where 1192911/103387889228 objects are degraded. (1 pg degraded, 1 pg undersized). Diving into the historic ops of an OSD we can see that the main latency is found between the event "queued_for_pg" and "reached_pg". (Averaging +/- 3 secs) As the system load is quite high I assume the systems are busy recalculating the code chunks for using the new disks we've added (though not sure), but I was wondering how I can better fine tune the system or pinpoint the exact bottle neck. Latency towards the disks doesn't seem an issue at first sight... We are running Ceph 14.2.11 Who can give me some thoughts on how I can better pinpoint the bottle neck? Thanks Kristof
Hi Kristof, are you seeing high (around 100%) OSDs' disks (main or DB ones) utilization along with slow ops? Thanks, Igor On 10/6/2020 11:09 AM, Kristof Coucke wrote:
Hi all,
We have a Ceph cluster which has been expanded from 10 to 16 nodes. Each node has between 14 and 16 OSDs of which 2 are NVMe disks. Most disks (except NVMe's) are 16TB large.
The expansion of 16 nodes went ok, but we've configured the system to prevent auto balance towards the new disks (weight was set to 0) so we could control the expansion.
We started adding 6 disks last week (1 disk on each new node) which didn't give a lot of issues. When the Ceph status indicated the PG degraded was almost finished, we've added 2 disks on each node again.
All seemed to go fine, till yesterday morning... IOs towards the system were slowing down.
Diving onto the nodes we could see that the OSD daemons are consuming the CPU power, resulting in average CPU loads going near 10 (!).
The RGWs nor monitors nor other involved servers are having CPU issues (except for the management server which is fighting with Prometheus), so it's latency seems to be related to the ODS hosts. All of the hosts are interconnected with 25Gbit connections, no bottlenecks are reached on the network either.
Important piece of information: We are using erasure coding (6/3), and we do have a lot of small files... The current health detail indicates degraded health redundancy where 1192911/103387889228 objects are degraded. (1 pg degraded, 1 pg undersized).
Diving into the historic ops of an OSD we can see that the main latency is found between the event "queued_for_pg" and "reached_pg". (Averaging +/- 3 secs)
As the system load is quite high I assume the systems are busy recalculating the code chunks for using the new disks we've added (though not sure), but I was wondering how I can better fine tune the system or pinpoint the exact bottle neck. Latency towards the disks doesn't seem an issue at first sight...
We are running Ceph 14.2.11
Who can give me some thoughts on how I can better pinpoint the bottle neck?
Thanks
Kristof _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yes, some disks are spiking near 100%... The delay I see with the iostat (r_await) seems to be synchronised with the delays between queued_for_pg and reached_pg events. The NVMe disks are not spiking, just the spinner disks. I know the rocksdb is only partial on the NVMe. The read-ahead is also 128kb (os level) (for spinner disks). As we are dealing with smaller files, this might also lead to a decrease of the performance. I'm still investigating, but I'm wondering if the system is also reading from disk for finding the KV pairs. Op di 6 okt. 2020 om 11:23 schreef Igor Fedotov <ifedotov@suse.de>:
Hi Kristof,
are you seeing high (around 100%) OSDs' disks (main or DB ones) utilization along with slow ops?
Thanks,
Igor
On 10/6/2020 11:09 AM, Kristof Coucke wrote:
Hi all,
We have a Ceph cluster which has been expanded from 10 to 16 nodes. Each node has between 14 and 16 OSDs of which 2 are NVMe disks. Most disks (except NVMe's) are 16TB large.
The expansion of 16 nodes went ok, but we've configured the system to prevent auto balance towards the new disks (weight was set to 0) so we could control the expansion.
We started adding 6 disks last week (1 disk on each new node) which didn't give a lot of issues. When the Ceph status indicated the PG degraded was almost finished, we've added 2 disks on each node again.
All seemed to go fine, till yesterday morning... IOs towards the system were slowing down.
Diving onto the nodes we could see that the OSD daemons are consuming the CPU power, resulting in average CPU loads going near 10 (!).
The RGWs nor monitors nor other involved servers are having CPU issues (except for the management server which is fighting with Prometheus), so it's latency seems to be related to the ODS hosts. All of the hosts are interconnected with 25Gbit connections, no bottlenecks are reached on the network either.
Important piece of information: We are using erasure coding (6/3), and we do have a lot of small files... The current health detail indicates degraded health redundancy where 1192911/103387889228 objects are degraded. (1 pg degraded, 1 pg undersized).
Diving into the historic ops of an OSD we can see that the main latency is found between the event "queued_for_pg" and "reached_pg". (Averaging +/- 3 secs)
As the system load is quite high I assume the systems are busy recalculating the code chunks for using the new disks we've added (though not sure), but I was wondering how I can better fine tune the system or pinpoint the exact bottle neck. Latency towards the disks doesn't seem an issue at first sight...
We are running Ceph 14.2.11
Who can give me some thoughts on how I can better pinpoint the bottle neck?
Thanks
Kristof _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Another strange thing is going on: No client software is using the system any longer, so we would expect that all IOs are related to the recovery (fixing of the degraded PG). However, the disks that are reaching high IO are not a member of the PGs that are being fixed. So, something is heavily using the disk, but I can't find the process immediately. I've read something that there can be old client processes that keep on connecting to an OSD for retrieving data for a specific PG while that PG is no longer available on that disk. Op di 6 okt. 2020 om 11:41 schreef Kristof Coucke <kristof.coucke@gmail.com
:
Yes, some disks are spiking near 100%... The delay I see with the iostat (r_await) seems to be synchronised with the delays between queued_for_pg and reached_pg events. The NVMe disks are not spiking, just the spinner disks.
I know the rocksdb is only partial on the NVMe. The read-ahead is also 128kb (os level) (for spinner disks). As we are dealing with smaller files, this might also lead to a decrease of the performance.
I'm still investigating, but I'm wondering if the system is also reading from disk for finding the KV pairs.
Op di 6 okt. 2020 om 11:23 schreef Igor Fedotov <ifedotov@suse.de>:
Hi Kristof,
are you seeing high (around 100%) OSDs' disks (main or DB ones) utilization along with slow ops?
Thanks,
Igor
Hi all,
We have a Ceph cluster which has been expanded from 10 to 16 nodes. Each node has between 14 and 16 OSDs of which 2 are NVMe disks. Most disks (except NVMe's) are 16TB large.
The expansion of 16 nodes went ok, but we've configured the system to prevent auto balance towards the new disks (weight was set to 0) so we could control the expansion.
We started adding 6 disks last week (1 disk on each new node) which didn't give a lot of issues. When the Ceph status indicated the PG degraded was almost finished, we've added 2 disks on each node again.
All seemed to go fine, till yesterday morning... IOs towards the system were slowing down.
Diving onto the nodes we could see that the OSD daemons are consuming
On 10/6/2020 11:09 AM, Kristof Coucke wrote: the
CPU power, resulting in average CPU loads going near 10 (!).
The RGWs nor monitors nor other involved servers are having CPU issues (except for the management server which is fighting with Prometheus), so it's latency seems to be related to the ODS hosts. All of the hosts are interconnected with 25Gbit connections, no bottlenecks are reached on the network either.
Important piece of information: We are using erasure coding (6/3), and we do have a lot of small files... The current health detail indicates degraded health redundancy where 1192911/103387889228 objects are degraded. (1 pg degraded, 1 pg undersized).
Diving into the historic ops of an OSD we can see that the main latency is found between the event "queued_for_pg" and "reached_pg". (Averaging +/- 3 secs)
As the system load is quite high I assume the systems are busy recalculating the code chunks for using the new disks we've added (though not sure), but I was wondering how I can better fine tune the system or pinpoint the exact bottle neck. Latency towards the disks doesn't seem an issue at first sight...
We are running Ceph 14.2.11
Who can give me some thoughts on how I can better pinpoint the bottle neck?
Thanks
Kristof _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 10/6/2020 1:04 PM, Kristof Coucke wrote:
Another strange thing is going on:
No client software is using the system any longer, so we would expect that all IOs are related to the recovery (fixing of the degraded PG). However, the disks that are reaching high IO are not a member of the PGs that are being fixed.
So, something is heavily using the disk, but I can't find the process immediately. I've read something that there can be old client processes that keep on connecting to an OSD for retrieving data for a specific PG while that PG is no longer available on that disk.
I bet it's rather PG removal happening in background....
Op di 6 okt. 2020 om 11:41 schreef Kristof Coucke <kristof.coucke@gmail.com <mailto:kristof.coucke@gmail.com>>:
Yes, some disks are spiking near 100%... The delay I see with the iostat (r_await) seems to be synchronised with the delays between queued_for_pg and reached_pg events. The NVMe disks are not spiking, just the spinner disks.
I know the rocksdb is only partial on the NVMe. The read-ahead is also 128kb (os level) (for spinner disks). As we are dealing with smaller files, this might also lead to a decrease of the performance.
I'm still investigating, but I'm wondering if the system is also reading from disk for finding the KV pairs.
Op di 6 okt. 2020 om 11:23 schreef Igor Fedotov <ifedotov@suse.de <mailto:ifedotov@suse.de>>:
Hi Kristof,
are you seeing high (around 100%) OSDs' disks (main or DB ones) utilization along with slow ops?
Thanks,
Igor
On 10/6/2020 11:09 AM, Kristof Coucke wrote: > Hi all, > > We have a Ceph cluster which has been expanded from 10 to 16 nodes. > Each node has between 14 and 16 OSDs of which 2 are NVMe disks. > Most disks (except NVMe's) are 16TB large. > > The expansion of 16 nodes went ok, but we've configured the system to > prevent auto balance towards the new disks (weight was set to 0) so we > could control the expansion. > > We started adding 6 disks last week (1 disk on each new node) which didn't > give a lot of issues. > When the Ceph status indicated the PG degraded was almost finished, we've > added 2 disks on each node again. > > All seemed to go fine, till yesterday morning... IOs towards the system > were slowing down. > > Diving onto the nodes we could see that the OSD daemons are consuming the > CPU power, resulting in average CPU loads going near 10 (!). > > The RGWs nor monitors nor other involved servers are having CPU issues > (except for the management server which is fighting with Prometheus), so > it's latency seems to be related to the ODS hosts. > All of the hosts are interconnected with 25Gbit connections, no bottlenecks > are reached on the network either. > > Important piece of information: We are using erasure coding (6/3), and we > do have a lot of small files... > The current health detail indicates degraded health redundancy where > 1192911/103387889228 objects are degraded. (1 pg degraded, 1 pg undersized). > > Diving into the historic ops of an OSD we can see that the main latency is > found between the event "queued_for_pg" and "reached_pg". (Averaging +/- 3 > secs) > > As the system load is quite high I assume the systems are busy > recalculating the code chunks for using the new disks we've added (though > not sure), but I was wondering how I can better fine tune the system or > pinpoint the exact bottle neck. > Latency towards the disks doesn't seem an issue at first sight... > > We are running Ceph 14.2.11 > > Who can give me some thoughts on how I can better pinpoint the bottle neck? > > Thanks > > Kristof > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> > To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Is there a way that I can check if this process is causing performance issues? Can I check somehow if this process is causing the issue? Op di 6 okt. 2020 om 13:05 schreef Igor Fedotov <ifedotov@suse.de>:
On 10/6/2020 1:04 PM, Kristof Coucke wrote:
Another strange thing is going on:
No client software is using the system any longer, so we would expect that all IOs are related to the recovery (fixing of the degraded PG). However, the disks that are reaching high IO are not a member of the PGs that are being fixed.
So, something is heavily using the disk, but I can't find the process immediately. I've read something that there can be old client processes that keep on connecting to an OSD for retrieving data for a specific PG while that PG is no longer available on that disk.
I bet it's rather PG removal happening in background....
Op di 6 okt. 2020 om 11:41 schreef Kristof Coucke < kristof.coucke@gmail.com>:
Yes, some disks are spiking near 100%... The delay I see with the iostat (r_await) seems to be synchronised with the delays between queued_for_pg and reached_pg events. The NVMe disks are not spiking, just the spinner disks.
I know the rocksdb is only partial on the NVMe. The read-ahead is also 128kb (os level) (for spinner disks). As we are dealing with smaller files, this might also lead to a decrease of the performance.
I'm still investigating, but I'm wondering if the system is also reading from disk for finding the KV pairs.
Op di 6 okt. 2020 om 11:23 schreef Igor Fedotov <ifedotov@suse.de>:
Hi Kristof,
are you seeing high (around 100%) OSDs' disks (main or DB ones) utilization along with slow ops?
Thanks,
Igor
Hi all,
We have a Ceph cluster which has been expanded from 10 to 16 nodes. Each node has between 14 and 16 OSDs of which 2 are NVMe disks. Most disks (except NVMe's) are 16TB large.
The expansion of 16 nodes went ok, but we've configured the system to prevent auto balance towards the new disks (weight was set to 0) so we could control the expansion.
We started adding 6 disks last week (1 disk on each new node) which didn't give a lot of issues. When the Ceph status indicated the PG degraded was almost finished, we've added 2 disks on each node again.
All seemed to go fine, till yesterday morning... IOs towards the system were slowing down.
Diving onto the nodes we could see that the OSD daemons are consuming
On 10/6/2020 11:09 AM, Kristof Coucke wrote: the
CPU power, resulting in average CPU loads going near 10 (!).
The RGWs nor monitors nor other involved servers are having CPU issues (except for the management server which is fighting with Prometheus), so it's latency seems to be related to the ODS hosts. All of the hosts are interconnected with 25Gbit connections, no bottlenecks are reached on the network either.
Important piece of information: We are using erasure coding (6/3), and we do have a lot of small files... The current health detail indicates degraded health redundancy where 1192911/103387889228 objects are degraded. (1 pg degraded, 1 pg undersized).
Diving into the historic ops of an OSD we can see that the main latency is found between the event "queued_for_pg" and "reached_pg". (Averaging +/- 3 secs)
As the system load is quite high I assume the systems are busy recalculating the code chunks for using the new disks we've added (though not sure), but I was wondering how I can better fine tune the system or pinpoint the exact bottle neck. Latency towards the disks doesn't seem an issue at first sight...
We are running Ceph 14.2.11
Who can give me some thoughts on how I can better pinpoint the bottle neck?
Thanks
Kristof _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Unfortunately currently available Ceph releases lack any means to monitor KV data removal. The only way is to set debug_bluestore to 20 (for a short period of time, e.g. 1 min) and inspect OSD log for _remove/_do_remove/_omap_clear calls. Plenty of them within the inspected period means ongoing removals. A weak proof of the hypothesis would be having non-zero "numpg_removing" performance counter... On 10/6/2020 2:06 PM, Kristof Coucke wrote:
Is there a way that I can check if this process is causing performance issues? Can I check somehow if this process is causing the issue?
Op di 6 okt. 2020 om 13:05 schreef Igor Fedotov <ifedotov@suse.de <mailto:ifedotov@suse.de>>:
On 10/6/2020 1:04 PM, Kristof Coucke wrote:
Another strange thing is going on:
No client software is using the system any longer, so we would expect that all IOs are related to the recovery (fixing of the degraded PG). However, the disks that are reaching high IO are not a member of the PGs that are being fixed.
So, something is heavily using the disk, but I can't find the process immediately. I've read something that there can be old client processes that keep on connecting to an OSD for retrieving data for a specific PG while that PG is no longer available on that disk.
I bet it's rather PG removal happening in background....
Op di 6 okt. 2020 om 11:41 schreef Kristof Coucke <kristof.coucke@gmail.com <mailto:kristof.coucke@gmail.com>>:
Yes, some disks are spiking near 100%... The delay I see with the iostat (r_await) seems to be synchronised with the delays between queued_for_pg and reached_pg events. The NVMe disks are not spiking, just the spinner disks.
I know the rocksdb is only partial on the NVMe. The read-ahead is also 128kb (os level) (for spinner disks). As we are dealing with smaller files, this might also lead to a decrease of the performance.
I'm still investigating, but I'm wondering if the system is also reading from disk for finding the KV pairs.
Op di 6 okt. 2020 om 11:23 schreef Igor Fedotov <ifedotov@suse.de <mailto:ifedotov@suse.de>>:
Hi Kristof,
are you seeing high (around 100%) OSDs' disks (main or DB ones) utilization along with slow ops?
Thanks,
Igor
On 10/6/2020 11:09 AM, Kristof Coucke wrote: > Hi all, > > We have a Ceph cluster which has been expanded from 10 to 16 nodes. > Each node has between 14 and 16 OSDs of which 2 are NVMe disks. > Most disks (except NVMe's) are 16TB large. > > The expansion of 16 nodes went ok, but we've configured the system to > prevent auto balance towards the new disks (weight was set to 0) so we > could control the expansion. > > We started adding 6 disks last week (1 disk on each new node) which didn't > give a lot of issues. > When the Ceph status indicated the PG degraded was almost finished, we've > added 2 disks on each node again. > > All seemed to go fine, till yesterday morning... IOs towards the system > were slowing down. > > Diving onto the nodes we could see that the OSD daemons are consuming the > CPU power, resulting in average CPU loads going near 10 (!). > > The RGWs nor monitors nor other involved servers are having CPU issues > (except for the management server which is fighting with Prometheus), so > it's latency seems to be related to the ODS hosts. > All of the hosts are interconnected with 25Gbit connections, no bottlenecks > are reached on the network either. > > Important piece of information: We are using erasure coding (6/3), and we > do have a lot of small files... > The current health detail indicates degraded health redundancy where > 1192911/103387889228 objects are degraded. (1 pg degraded, 1 pg undersized). > > Diving into the historic ops of an OSD we can see that the main latency is > found between the event "queued_for_pg" and "reached_pg". (Averaging +/- 3 > secs) > > As the system load is quite high I assume the systems are busy > recalculating the code chunks for using the new disks we've added (though > not sure), but I was wondering how I can better fine tune the system or > pinpoint the exact bottle neck. > Latency towards the disks doesn't seem an issue at first sight... > > We are running Ceph 14.2.11 > > Who can give me some thoughts on how I can better pinpoint the bottle neck? > > Thanks > > Kristof > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> > To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
On 2020-10-06 13:05, Igor Fedotov wrote:
On 10/6/2020 1:04 PM, Kristof Coucke wrote:
Another strange thing is going on:
No client software is using the system any longer, so we would expect that all IOs are related to the recovery (fixing of the degraded PG). However, the disks that are reaching high IO are not a member of the PGs that are being fixed.
So, something is heavily using the disk, but I can't find the process immediately. I've read something that there can be old client processes that keep on connecting to an OSD for retrieving data for a specific PG while that PG is no longer available on that disk.
I bet it's rather PG removal happening in background....
^^ This, and probably the accompanying RocksDB housekeeping that goes with it. As only removing PGs shouldn't be a too big a deal at all. Especially with very small files (and a lot of them) you probably have a lot of OMAP / META data, (ceph osd df will tell you). If that's indeed the case than there is a (way) quicker option to get out of this situation: offline compacting of the OSDs. This process happens orders of magnitude faster than when the OSDs are still online. To check if this hypothesis is true: are the OSD servers under CPU stress where the PGs were located previously (and not the new hosts)? Offline compaction per host: systemctl stop ceph-osd.target for osd in `ls /var/lib/ceph/osd/`; do (ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/$osd compact &);done Gr. Stefan
Ok, I did the compact on 1 osd. The utilization is back to normal, so that's good... Thumbs up to you guys! Though, one thing I want to get out of the way before adapting the other OSDs: When I now get the RocksDb stats, my L1, L2 and L3 are gone: db_statistics { "rocksdb_compaction_statistics": "", "": "", "": "** Compaction Stats [default] **", "": "Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop", "": "----------------------------------------------------------------------------------------------------------------------------------------------------------------------------", "": " L0 1/0 968.45 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 105.1 0.01 0.00 1 0.009 0 0", "": " L4 1557/0 98.10 GB 0.4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0", "": " Sum 1558/0 98.10 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 105.1 0.01 0.00 1 0.009 0 0", "": " Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 105.1 0.01 0.00 1 0.009 0 0", "": "", "": "** Compaction Stats [default] **", "": "Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop", "": "-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------", "": "User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 105.1 0.01 0.00 1 0.009 0 0", "": "Uptime(secs): 0.3 total, 0.3 interval", "": "Flush(GB): cumulative 0.001, interval 0.001", "": "AddFile(GB): cumulative 0.000, interval 0.000", "": "AddFile(Total Files): cumulative 0, interval 0", "": "AddFile(L0 Files): cumulative 0, interval 0", "": "AddFile(Keys): cumulative 0, interval 0", "": "Cumulative compaction: 0.00 GB write, 2.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds", "": "Interval compaction: 0.00 GB write, 2.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds", "": "Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count", "": "", "": "** File Read Latency Histogram By Level [default] **", "": "** Level 0 read latency histogram (micros):", "": "Count: 5 Average: 69.2000 StdDev: 85.92", "": "Min: 0 Median: 1.5000 Max: 201", "": "Percentiles: P50: 1.50 P75: 155.00 P99: 201.00 P99.9: 201.00 P99.99: 201.00", "": "------------------------------------------------------", "": "[ 0, 1 ] 2 40.000% 40.000% ########", "": "( 1, 2 ] 1 20.000% 60.000% ####", "": "( 110, 170 ] 1 20.000% 80.000% ####", "": "( 170, 250 ] 1 20.000% 100.000% ####", "": "", "": "** Level 4 read latency histogram (micros):", "": "Count: 4664 Average: 0.6895 StdDev: 0.82", "": "Min: 0 Median: 0.5258 Max: 27", "": "Percentiles: P50: 0.53 P75: 0.79 P99: 2.61 P99.9: 6.45 P99.99: 13.83", "": "------------------------------------------------------", "": "[ 0, 1 ] 4435 95.090% 95.090% ###################", "": "( 1, 2 ] 149 3.195% 98.285% #", "": "( 2, 3 ] 55 1.179% 99.464% ", "": "( 3, 4 ] 12 0.257% 99.721% ", "": "( 4, 6 ] 8 0.172% 99.893% ", "": "( 6, 10 ] 3 0.064% 99.957% ", "": "( 10, 15 ] 2 0.043% 100.000% ", "": "( 22, 34 ] 1 0.021% 100.021% ", "": "", "": "", "": "** DB Stats **", "": "Uptime(secs): 0.3 total, 0.3 interval", "": "Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s", "": "Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s", "": "Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent", "": "Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s", "": "Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s", "": "Interval stall: 00:00:0.000 H:M:S, 0.0 percent" } We use the NVMe's to store the RocksDb, but with the spillover towards the spinning drives. L4 is intended to be stored on the spinning drives... Will the other levels be created automatically? Op di 6 okt. 2020 om 13:18 schreef Stefan Kooman <stefan@bit.nl>:
On 2020-10-06 13:05, Igor Fedotov wrote:
On 10/6/2020 1:04 PM, Kristof Coucke wrote:
Another strange thing is going on:
No client software is using the system any longer, so we would expect that all IOs are related to the recovery (fixing of the degraded PG). However, the disks that are reaching high IO are not a member of the PGs that are being fixed.
So, something is heavily using the disk, but I can't find the process immediately. I've read something that there can be old client processes that keep on connecting to an OSD for retrieving data for a specific PG while that PG is no longer available on that disk.
I bet it's rather PG removal happening in background....
^^ This, and probably the accompanying RocksDB housekeeping that goes with it. As only removing PGs shouldn't be a too big a deal at all. Especially with very small files (and a lot of them) you probably have a lot of OMAP / META data, (ceph osd df will tell you).
If that's indeed the case than there is a (way) quicker option to get out of this situation: offline compacting of the OSDs. This process happens orders of magnitude faster than when the OSDs are still online.
To check if this hypothesis is true: are the OSD servers under CPU stress where the PGs were located previously (and not the new hosts)?
Offline compaction per host:
systemctl stop ceph-osd.target
for osd in `ls /var/lib/ceph/osd/`; do (ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/$osd compact &);done
Gr. Stefan
On 2020-10-06 14:18, Kristof Coucke wrote:
Ok, I did the compact on 1 osd. The utilization is back to normal, so that's good... Thumbs up to you guys!
We learned the hard way, but happy to spot the issue and share the info.
Though, one thing I want to get out of the way before adapting the other OSDs: When I now get the RocksDb stats, my L1, L2 and L3 are gone:
I guess that they have all been merged now. On a OSD with L0 and L1 I see L0 disappear after a compact. After a restart, recovery, and then dumping stats again it's there again. So yeah, it gets created automatically.
We use the NVMe's to store the RocksDb, but with the spillover towards the spinning drives. L4 is intended to be stored on the spinning drives... Will the other levels be created automatically?
Yeah pretty sure that's how RocksDB works, and tested that (see above). Gr. Stefan
I've seen similar reports after manual compactions as well. But it looks like a presentation bug in RocksDB to me. You can check if all the data is spilled over (as it ought to be for L4) in bluefs section of OSD perf counters dump... On 10/6/2020 3:18 PM, Kristof Coucke wrote:
Ok, I did the compact on 1 osd. The utilization is back to normal, so that's good... Thumbs up to you guys! Though, one thing I want to get out of the way before adapting the other OSDs: When I now get the RocksDb stats, my L1, L2 and L3 are gone:
db_statistics { "rocksdb_compaction_statistics": "", "": "", "": "** Compaction Stats [default] **", "": "Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop", "": "----------------------------------------------------------------------------------------------------------------------------------------------------------------------------", "": " L0 1/0 968.45 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 105.1 0.01 0.00 1 0.009 0 0", "": " L4 1557/0 98.10 GB 0.4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0", "": " Sum 1558/0 98.10 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 105.1 0.01 0.00 1 0.009 0 0", "": " Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 105.1 0.01 0.00 1 0.009 0 0", "": "", "": "** Compaction Stats [default] **", "": "Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop", "": "-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------", "": "User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 105.1 0.01 0.00 1 0.009 0 0", "": "Uptime(secs): 0.3 total, 0.3 interval", "": "Flush(GB): cumulative 0.001, interval 0.001", "": "AddFile(GB): cumulative 0.000, interval 0.000", "": "AddFile(Total Files): cumulative 0, interval 0", "": "AddFile(L0 Files): cumulative 0, interval 0", "": "AddFile(Keys): cumulative 0, interval 0", "": "Cumulative compaction: 0.00 GB write, 2.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds", "": "Interval compaction: 0.00 GB write, 2.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds", "": "Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count", "": "", "": "** File Read Latency Histogram By Level [default] **", "": "** Level 0 read latency histogram (micros):", "": "Count: 5 Average: 69.2000 StdDev: 85.92", "": "Min: 0 Median: 1.5000 Max: 201", "": "Percentiles: P50: 1.50 P75: 155.00 P99: 201.00 P99.9: 201.00 P99.99: 201.00", "": "------------------------------------------------------", "": "[ 0, 1 ] 2 40.000% 40.000% ########", "": "( 1, 2 ] 1 20.000% 60.000% ####", "": "( 110, 170 ] 1 20.000% 80.000% ####", "": "( 170, 250 ] 1 20.000% 100.000% ####", "": "", "": "** Level 4 read latency histogram (micros):", "": "Count: 4664 Average: 0.6895 StdDev: 0.82", "": "Min: 0 Median: 0.5258 Max: 27", "": "Percentiles: P50: 0.53 P75: 0.79 P99: 2.61 P99.9: 6.45 P99.99: 13.83", "": "------------------------------------------------------", "": "[ 0, 1 ] 4435 95.090% 95.090% ###################", "": "( 1, 2 ] 149 3.195% 98.285% #", "": "( 2, 3 ] 55 1.179% 99.464% ", "": "( 3, 4 ] 12 0.257% 99.721% ", "": "( 4, 6 ] 8 0.172% 99.893% ", "": "( 6, 10 ] 3 0.064% 99.957% ", "": "( 10, 15 ] 2 0.043% 100.000% ", "": "( 22, 34 ] 1 0.021% 100.021% ", "": "", "": "", "": "** DB Stats **", "": "Uptime(secs): 0.3 total, 0.3 interval", "": "Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s", "": "Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s", "": "Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent", "": "Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s", "": "Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s", "": "Interval stall: 00:00:0.000 H:M:S, 0.0 percent" }
We use the NVMe's to store the RocksDb, but with the spillover towards the spinning drives. L4 is intended to be stored on the spinning drives... Will the other levels be created automatically?
Op di 6 okt. 2020 om 13:18 schreef Stefan Kooman <stefan@bit.nl <mailto:stefan@bit.nl>>:
On 2020-10-06 13:05, Igor Fedotov wrote: > > On 10/6/2020 1:04 PM, Kristof Coucke wrote: >> Another strange thing is going on: >> >> No client software is using the system any longer, so we would expect >> that all IOs are related to the recovery (fixing of the degraded PG). >> However, the disks that are reaching high IO are not a member of the >> PGs that are being fixed. >> >> So, something is heavily using the disk, but I can't find the process >> immediately. I've read something that there can be old client >> processes that keep on connecting to an OSD for retrieving data for a >> specific PG while that PG is no longer available on that disk. >> >> > I bet it's rather PG removal happening in background....
^^ This, and probably the accompanying RocksDB housekeeping that goes with it. As only removing PGs shouldn't be a too big a deal at all. Especially with very small files (and a lot of them) you probably have a lot of OMAP / META data, (ceph osd df will tell you).
If that's indeed the case than there is a (way) quicker option to get out of this situation: offline compacting of the OSDs. This process happens orders of magnitude faster than when the OSDs are still online.
To check if this hypothesis is true: are the OSD servers under CPU stress where the PGs were located previously (and not the new hosts)?
Offline compaction per host:
systemctl stop ceph-osd.target
for osd in `ls /var/lib/ceph/osd/`; do (ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/$osd compact &);done
Gr. Stefan
Hi Igor and Stefan, Everything seems okay, so we'll now create a script to automate this on all the nodes and we will also review the monitoring possibilities. Thanks for your help, it was a time saver. Does anyone know if this issue is better handled in the newer versions or if this is planned in an upcoming release? My best regards, Kristof Op di 6 okt. 2020 om 14:36 schreef Igor Fedotov <ifedotov@suse.de>:
I've seen similar reports after manual compactions as well. But it looks like a presentation bug in RocksDB to me.
You can check if all the data is spilled over (as it ought to be for L4) in bluefs section of OSD perf counters dump...
On 10/6/2020 3:18 PM, Kristof Coucke wrote:
Ok, I did the compact on 1 osd. The utilization is back to normal, so that's good... Thumbs up to you guys! Though, one thing I want to get out of the way before adapting the other OSDs: When I now get the RocksDb stats, my L1, L2 and L3 are gone:
db_statistics { "rocksdb_compaction_statistics": "", "": "", "": "** Compaction Stats [default] **", "": "Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop", "": "----------------------------------------------------------------------------------------------------------------------------------------------------------------------------", "": " L0 1/0 968.45 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 105.1 0.01 0.00 1 0.009 0 0", "": " L4 1557/0 98.10 GB 0.4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0", "": " Sum 1558/0 98.10 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 105.1 0.01 0.00 1 0.009 0 0", "": " Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 105.1 0.01 0.00 1 0.009 0 0", "": "", "": "** Compaction Stats [default] **", "": "Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop", "": "-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------", "": "User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 105.1 0.01 0.00 1 0.009 0 0", "": "Uptime(secs): 0.3 total, 0.3 interval", "": "Flush(GB): cumulative 0.001, interval 0.001", "": "AddFile(GB): cumulative 0.000, interval 0.000", "": "AddFile(Total Files): cumulative 0, interval 0", "": "AddFile(L0 Files): cumulative 0, interval 0", "": "AddFile(Keys): cumulative 0, interval 0", "": "Cumulative compaction: 0.00 GB write, 2.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds", "": "Interval compaction: 0.00 GB write, 2.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds", "": "Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count", "": "", "": "** File Read Latency Histogram By Level [default] **", "": "** Level 0 read latency histogram (micros):", "": "Count: 5 Average: 69.2000 StdDev: 85.92", "": "Min: 0 Median: 1.5000 Max: 201", "": "Percentiles: P50: 1.50 P75: 155.00 P99: 201.00 P99.9: 201.00 P99.99: 201.00", "": "------------------------------------------------------", "": "[ 0, 1 ] 2 40.000% 40.000% ########", "": "( 1, 2 ] 1 20.000% 60.000% ####", "": "( 110, 170 ] 1 20.000% 80.000% ####", "": "( 170, 250 ] 1 20.000% 100.000% ####", "": "", "": "** Level 4 read latency histogram (micros):", "": "Count: 4664 Average: 0.6895 StdDev: 0.82", "": "Min: 0 Median: 0.5258 Max: 27", "": "Percentiles: P50: 0.53 P75: 0.79 P99: 2.61 P99.9: 6.45 P99.99: 13.83", "": "------------------------------------------------------", "": "[ 0, 1 ] 4435 95.090% 95.090% ###################", "": "( 1, 2 ] 149 3.195% 98.285% #", "": "( 2, 3 ] 55 1.179% 99.464% ", "": "( 3, 4 ] 12 0.257% 99.721% ", "": "( 4, 6 ] 8 0.172% 99.893% ", "": "( 6, 10 ] 3 0.064% 99.957% ", "": "( 10, 15 ] 2 0.043% 100.000% ", "": "( 22, 34 ] 1 0.021% 100.021% ", "": "", "": "", "": "** DB Stats **", "": "Uptime(secs): 0.3 total, 0.3 interval", "": "Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s", "": "Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s", "": "Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent", "": "Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s", "": "Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s", "": "Interval stall: 00:00:0.000 H:M:S, 0.0 percent" }
We use the NVMe's to store the RocksDb, but with the spillover towards the spinning drives. L4 is intended to be stored on the spinning drives... Will the other levels be created automatically?
Op di 6 okt. 2020 om 13:18 schreef Stefan Kooman <stefan@bit.nl>:
On 2020-10-06 13:05, Igor Fedotov wrote:
On 10/6/2020 1:04 PM, Kristof Coucke wrote:
Another strange thing is going on:
No client software is using the system any longer, so we would expect that all IOs are related to the recovery (fixing of the degraded PG). However, the disks that are reaching high IO are not a member of the PGs that are being fixed.
So, something is heavily using the disk, but I can't find the process immediately. I've read something that there can be old client processes that keep on connecting to an OSD for retrieving data for a specific PG while that PG is no longer available on that disk.
I bet it's rather PG removal happening in background....
^^ This, and probably the accompanying RocksDB housekeeping that goes with it. As only removing PGs shouldn't be a too big a deal at all. Especially with very small files (and a lot of them) you probably have a lot of OMAP / META data, (ceph osd df will tell you).
If that's indeed the case than there is a (way) quicker option to get out of this situation: offline compacting of the OSDs. This process happens orders of magnitude faster than when the OSDs are still online.
To check if this hypothesis is true: are the OSD servers under CPU stress where the PGs were located previously (and not the new hosts)?
Offline compaction per host:
systemctl stop ceph-osd.target
for osd in `ls /var/lib/ceph/osd/`; do (ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/$osd compact &);done
Gr. Stefan
I'm working on improving PG removal in master, see: https://github.com/ceph/ceph/pull/37496 Hopefully this will help in case of "cleanup after rebalancing" issue which you presumably had. On 10/6/2020 4:24 PM, Kristof Coucke wrote:
Hi Igor and Stefan,
Everything seems okay, so we'll now create a script to automate this on all the nodes and we will also review the monitoring possibilities. Thanks for your help, it was a time saver.
Does anyone know if this issue is better handled in the newer versions or if this is planned in an upcoming release?
My best regards,
Kristof
Op di 6 okt. 2020 om 14:36 schreef Igor Fedotov <ifedotov@suse.de <mailto:ifedotov@suse.de>>:
I've seen similar reports after manual compactions as well. But it looks like a presentation bug in RocksDB to me.
You can check if all the data is spilled over (as it ought to be for L4) in bluefs section of OSD perf counters dump...
On 10/6/2020 3:18 PM, Kristof Coucke wrote:
Ok, I did the compact on 1 osd. The utilization is back to normal, so that's good... Thumbs up to you guys! Though, one thing I want to get out of the way before adapting the other OSDs: When I now get the RocksDb stats, my L1, L2 and L3 are gone:
db_statistics { "rocksdb_compaction_statistics": "", "": "", "": "** Compaction Stats [default] **", "": "Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop", "": "----------------------------------------------------------------------------------------------------------------------------------------------------------------------------", "": " L0 1/0 968.45 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 105.1 0.01 0.00 1 0.009 0 0", "": " L4 1557/0 98.10 GB 0.4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0", "": " Sum 1558/0 98.10 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 105.1 0.01 0.00 1 0.009 0 0", "": " Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 105.1 0.01 0.00 1 0.009 0 0", "": "", "": "** Compaction Stats [default] **", "": "Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop", "": "-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------", "": "User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 105.1 0.01 0.00 1 0.009 0 0", "": "Uptime(secs): 0.3 total, 0.3 interval", "": "Flush(GB): cumulative 0.001, interval 0.001", "": "AddFile(GB): cumulative 0.000, interval 0.000", "": "AddFile(Total Files): cumulative 0, interval 0", "": "AddFile(L0 Files): cumulative 0, interval 0", "": "AddFile(Keys): cumulative 0, interval 0", "": "Cumulative compaction: 0.00 GB write, 2.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds", "": "Interval compaction: 0.00 GB write, 2.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds", "": "Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count", "": "", "": "** File Read Latency Histogram By Level [default] **", "": "** Level 0 read latency histogram (micros):", "": "Count: 5 Average: 69.2000 StdDev: 85.92", "": "Min: 0 Median: 1.5000 Max: 201", "": "Percentiles: P50: 1.50 P75: 155.00 P99: 201.00 P99.9: 201.00 P99.99: 201.00", "": "------------------------------------------------------", "": "[ 0, 1 ] 2 40.000% 40.000% ########", "": "( 1, 2 ] 1 20.000% 60.000% ####", "": "( 110, 170 ] 1 20.000% 80.000% ####", "": "( 170, 250 ] 1 20.000% 100.000% ####", "": "", "": "** Level 4 read latency histogram (micros):", "": "Count: 4664 Average: 0.6895 StdDev: 0.82", "": "Min: 0 Median: 0.5258 Max: 27", "": "Percentiles: P50: 0.53 P75: 0.79 P99: 2.61 P99.9: 6.45 P99.99: 13.83", "": "------------------------------------------------------", "": "[ 0, 1 ] 4435 95.090% 95.090% ###################", "": "( 1, 2 ] 149 3.195% 98.285% #", "": "( 2, 3 ] 55 1.179% 99.464% ", "": "( 3, 4 ] 12 0.257% 99.721% ", "": "( 4, 6 ] 8 0.172% 99.893% ", "": "( 6, 10 ] 3 0.064% 99.957% ", "": "( 10, 15 ] 2 0.043% 100.000% ", "": "( 22, 34 ] 1 0.021% 100.021% ", "": "", "": "", "": "** DB Stats **", "": "Uptime(secs): 0.3 total, 0.3 interval", "": "Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s", "": "Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s", "": "Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent", "": "Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s", "": "Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s", "": "Interval stall: 00:00:0.000 H:M:S, 0.0 percent" }
We use the NVMe's to store the RocksDb, but with the spillover towards the spinning drives. L4 is intended to be stored on the spinning drives... Will the other levels be created automatically?
Op di 6 okt. 2020 om 13:18 schreef Stefan Kooman <stefan@bit.nl <mailto:stefan@bit.nl>>:
On 2020-10-06 13:05, Igor Fedotov wrote: > > On 10/6/2020 1:04 PM, Kristof Coucke wrote: >> Another strange thing is going on: >> >> No client software is using the system any longer, so we would expect >> that all IOs are related to the recovery (fixing of the degraded PG). >> However, the disks that are reaching high IO are not a member of the >> PGs that are being fixed. >> >> So, something is heavily using the disk, but I can't find the process >> immediately. I've read something that there can be old client >> processes that keep on connecting to an OSD for retrieving data for a >> specific PG while that PG is no longer available on that disk. >> >> > I bet it's rather PG removal happening in background....
^^ This, and probably the accompanying RocksDB housekeeping that goes with it. As only removing PGs shouldn't be a too big a deal at all. Especially with very small files (and a lot of them) you probably have a lot of OMAP / META data, (ceph osd df will tell you).
If that's indeed the case than there is a (way) quicker option to get out of this situation: offline compacting of the OSDs. This process happens orders of magnitude faster than when the OSDs are still online.
To check if this hypothesis is true: are the OSD servers under CPU stress where the PGs were located previously (and not the new hosts)?
Offline compaction per host:
systemctl stop ceph-osd.target
for osd in `ls /var/lib/ceph/osd/`; do (ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/$osd compact &);done
Gr. Stefan
On 2020-10-06 15:27, Igor Fedotov wrote:
I'm working on improving PG removal in master, see: https://github.com/ceph/ceph/pull/37496
Hopefully this will help in case of "cleanup after rebalancing" issue which you presumably had.
That would be great. Does the offline compaction with the ceph-kvstore-tool follows a completely different removal procedure and is that why it's that much faster? @Kristof: https://github.com/ceph/ceph/commit/93e4c56ecc13560e0dad69aaa67afc3ca053fb4c is a commit by Wido that would help to enable compaction at OSD (re)start. Gr. Stefan
On 10/6/2020 4:43 PM, Stefan Kooman wrote:
On 2020-10-06 15:27, Igor Fedotov wrote:
I'm working on improving PG removal in master, see: https://github.com/ceph/ceph/pull/37496
Hopefully this will help in case of "cleanup after rebalancing" issue which you presumably had. That would be great. Does the offline compaction with the ceph-kvstore-tool follows a completely different removal procedure and is that why it's that much faster?
First of all - compaction is primarily about rearranging DB data in an optimal way not removing it. Compaction after massive removals just optimizes DB data layout and eliminates all the leftovers from the previous rewrites/removals that negatively impact DB performance. Offline compaction work in an exclusive access mode and doesn't care about parallel DB operations. Hence it's presumably much more effective comparing to the online one. Not to mention that the latter is rather partial...
@Kristof: https://github.com/ceph/ceph/commit/93e4c56ecc13560e0dad69aaa67afc3ca053fb4c is a commit by Wido that would help to enable compaction at OSD (re)start.
Gr. Stefan
We have similar with this issue last week. We have sluggish disk (10TB SAS in RAID 0 mode) in half of node which affect performance of cluster. These disk has high CPU usage and very high latency. Turns out there is a process *patrol read* from RAID card that running automatically every week. When we stop patrol read, everything is normal again. We also running on Ceph 14.2.11. We don't have this issue with previous Ceph version and never change setting of patrol read. Thanks. On 06/10/20 17.04, Kristof Coucke wrote:
Another strange thing is going on:
No client software is using the system any longer, so we would expect that all IOs are related to the recovery (fixing of the degraded PG). However, the disks that are reaching high IO are not a member of the PGs that are being fixed.
So, something is heavily using the disk, but I can't find the process immediately. I've read something that there can be old client processes that keep on connecting to an OSD for retrieving data for a specific PG while that PG is no longer available on that disk.
Op di 6 okt. 2020 om 11:41 schreef Kristof Coucke <kristof.coucke@gmail.com
: Yes, some disks are spiking near 100%... The delay I see with the iostat (r_await) seems to be synchronised with the delays between queued_for_pg and reached_pg events. The NVMe disks are not spiking, just the spinner disks.
I know the rocksdb is only partial on the NVMe. The read-ahead is also 128kb (os level) (for spinner disks). As we are dealing with smaller files, this might also lead to a decrease of the performance.
I'm still investigating, but I'm wondering if the system is also reading from disk for finding the KV pairs.
Op di 6 okt. 2020 om 11:23 schreef Igor Fedotov <ifedotov@suse.de>:
Hi Kristof,
are you seeing high (around 100%) OSDs' disks (main or DB ones) utilization along with slow ops?
Thanks,
Igor
Hi all,
We have a Ceph cluster which has been expanded from 10 to 16 nodes. Each node has between 14 and 16 OSDs of which 2 are NVMe disks. Most disks (except NVMe's) are 16TB large.
The expansion of 16 nodes went ok, but we've configured the system to prevent auto balance towards the new disks (weight was set to 0) so we could control the expansion.
We started adding 6 disks last week (1 disk on each new node) which didn't give a lot of issues. When the Ceph status indicated the PG degraded was almost finished, we've added 2 disks on each node again.
All seemed to go fine, till yesterday morning... IOs towards the system were slowing down.
Diving onto the nodes we could see that the OSD daemons are consuming
On 10/6/2020 11:09 AM, Kristof Coucke wrote: the
CPU power, resulting in average CPU loads going near 10 (!).
The RGWs nor monitors nor other involved servers are having CPU issues (except for the management server which is fighting with Prometheus), so it's latency seems to be related to the ODS hosts. All of the hosts are interconnected with 25Gbit connections, no bottlenecks are reached on the network either.
Important piece of information: We are using erasure coding (6/3), and we do have a lot of small files... The current health detail indicates degraded health redundancy where 1192911/103387889228 objects are degraded. (1 pg degraded, 1 pg undersized). Diving into the historic ops of an OSD we can see that the main latency is found between the event "queued_for_pg" and "reached_pg". (Averaging +/- 3 secs)
As the system load is quite high I assume the systems are busy recalculating the code chunks for using the new disks we've added (though not sure), but I was wondering how I can better fine tune the system or pinpoint the exact bottle neck. Latency towards the disks doesn't seem an issue at first sight...
We are running Ceph 14.2.11
Who can give me some thoughts on how I can better pinpoint the bottle neck? Thanks
Kristof _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I presume that this might be caused by massive KV data removal which was initiated after(or during) data rebalance. We've seen multiple complains about RocksDB's performance negatively affected by pool/pg removal. And I expect data rebalance might suffer from the same... You might want to run manual DB compaction using ceph-kvstore-tool for every affected OSD to try to work around the issue. This would rather help just temporarily if data removal is still ongoing though. On 10/6/2020 12:41 PM, Kristof Coucke wrote:
Yes, some disks are spiking near 100%... The delay I see with the iostat (r_await) seems to be synchronised with the delays between queued_for_pg and reached_pg events. The NVMe disks are not spiking, just the spinner disks.
I know the rocksdb is only partial on the NVMe. The read-ahead is also 128kb (os level) (for spinner disks). As we are dealing with smaller files, this might also lead to a decrease of the performance.
Can you share the amount of DB data spilled over to spinners? You can learn this from "bluefs" section in performance counters dump...
I'm still investigating, but I'm wondering if the system is also reading from disk for finding the KV pairs.
Op di 6 okt. 2020 om 11:23 schreef Igor Fedotov <ifedotov@suse.de <mailto:ifedotov@suse.de>>:
Hi Kristof,
are you seeing high (around 100%) OSDs' disks (main or DB ones) utilization along with slow ops?
Thanks,
Igor
On 10/6/2020 11:09 AM, Kristof Coucke wrote: > Hi all, > > We have a Ceph cluster which has been expanded from 10 to 16 nodes. > Each node has between 14 and 16 OSDs of which 2 are NVMe disks. > Most disks (except NVMe's) are 16TB large. > > The expansion of 16 nodes went ok, but we've configured the system to > prevent auto balance towards the new disks (weight was set to 0) so we > could control the expansion. > > We started adding 6 disks last week (1 disk on each new node) which didn't > give a lot of issues. > When the Ceph status indicated the PG degraded was almost finished, we've > added 2 disks on each node again. > > All seemed to go fine, till yesterday morning... IOs towards the system > were slowing down. > > Diving onto the nodes we could see that the OSD daemons are consuming the > CPU power, resulting in average CPU loads going near 10 (!). > > The RGWs nor monitors nor other involved servers are having CPU issues > (except for the management server which is fighting with Prometheus), so > it's latency seems to be related to the ODS hosts. > All of the hosts are interconnected with 25Gbit connections, no bottlenecks > are reached on the network either. > > Important piece of information: We are using erasure coding (6/3), and we > do have a lot of small files... > The current health detail indicates degraded health redundancy where > 1192911/103387889228 objects are degraded. (1 pg degraded, 1 pg undersized). > > Diving into the historic ops of an OSD we can see that the main latency is > found between the event "queued_for_pg" and "reached_pg". (Averaging +/- 3 > secs) > > As the system load is quite high I assume the systems are busy > recalculating the code chunks for using the new disks we've added (though > not sure), but I was wondering how I can better fine tune the system or > pinpoint the exact bottle neck. > Latency towards the disks doesn't seem an issue at first sight... > > We are running Ceph 14.2.11 > > Who can give me some thoughts on how I can better pinpoint the bottle neck? > > Thanks > > Kristof > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> > To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
participants (4)
-
Danni Setiawan
-
Igor Fedotov
-
Kristof Coucke
-
Stefan Kooman