How to detect condition for offline compaction of RocksDB?
Hi, We have a big Ceph cluster (RGW case) with a lot of big buckets 10-500M objects with 31-1024 shards and a lot of io generated by many clients. Index pool placed on enterprise SSDs. We have about 120 SSDs (replication 3) and about 90Gb of OMAP data on each drive. About 75 PGs on each SSD for now. I think 75 is not enough for this amount of data, but I’m not sure that it is critical in our case. The problem: For the last few weeks, we can see big degradation of some SSD OSDs. We can see a lot of Ops (150-256 in perf dump) for a long time and we can see high avg ops time like 1-3 seconds (this metric is based on dump_historic_ops_by_duration and our avg calculation). These numbers are very unusual for our deployment. And it makes a big impact on our customers. For now, we make Offline compaction of ‘degraded’ OSDs and after compaction, we see that all PGs of this OSD are returned on this OSD (because recover is disabled during compaction) and we see that OSD works perfect for some time, all our metrics are dropped down for few weeks or only few days… And, I think the problem is in the rocksdb database which grows by levels and slows down requests. And I have few question about compaction: 1. Why can't automatic compaction manage this on its own? 2. How can I see RocksDB levels usage or some program metric which can be used as a condition for manual compacting? Because our metrics like request latency, OSD Ops count, OSD avg slow ops time are not 100% relative to rocksdb internal state. We try to use perf dump, bluestore.kv_xxx_lat metrics, but I think they are absolutely useless. Because we can see higher values after compaction and restart OSD then these metrics were before compaction, lol. We try to see OSD logs with debug_rocksdb=4, but we can’t understand this output compare it between good and bad OSDs: Uptime(secs): 3610809.5 total, 600.0 interval Flush(GB): cumulative 50.519, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 475.34 GB write, 0.13 MB/s write, 453.07 GB read, 0.13 MB/s read, 3346.3 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [p-0] ** ** Compaction Stats [p-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 52.0 52.0 0.0 1.0 0.0 155.2 343.25 176.27 746 0.460 0 0 L1 4/0 204.00 MB 0.8 96.7 52.0 44.7 96.2 51.4 0.0 1.8 164.6 163.6 601.87 316.34 237 2.540 275M 1237K L2 34/0 2.09 GB 1.0 176.8 43.6 133.2 175.3 42.1 7.6 4.0 147.6 146.3 1226.72 590.20 535 2.293 519M 2672K L3 79/0 4.76 GB 0.3 163.6 38.9 124.7 140.7 16.1 8.8 3.6 152.6 131.3 1097.53 443.60 432 2.541 401M 79M L4 316/0 19.96 GB 0.1 0.9 0.4 0.5 0.8 0.3 19.7 2.0 151.1 131.1 5.94 2.53 4 1.485 2349K 385K Sum 433/0 27.01 GB 0.0 438.0 134.9 303.0 465.0 161.9 36.1 8.9 136.9 145.4 3275.31 1528.94 1954 1.676 1198M 83M Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 and this log is not usable as ‘metric’. Our Ceph version is: 16.2.13 and we use default bluestore_rocksdb_options.
Hello Aleksandr, What you're probably experiencing is tombstone accumulation, a known issue for Ceph's use of rocksdb.
1. Why can't automatic compaction manage this on its own?
rocksdb compaction is normally triggered by level fullness and not tombstone counts. However, there is a feature in rocksdb that can cause a file to be compacted if there are many tombstones found in it when iterated that can help immensely with tombstone accumulation problems, available as of 16.2.14. You can find a summary of how to enable it and tweak it here: https://www.spinics.net/lists/ceph-users/msg78514.html
2. How can I see RocksDB levels usage or some program metric which can be used as a condition for manual compacting?
There is no tombstone counter that I'm aware of, which is really what you need in order to trigger compaction when appropriate. Josh On Tue, Jul 16, 2024 at 9:12 AM Rudenko Aleksandr <ARudenko@croc.ru> wrote:
Hi,
We have a big Ceph cluster (RGW case) with a lot of big buckets 10-500M objects with 31-1024 shards and a lot of io generated by many clients. Index pool placed on enterprise SSDs. We have about 120 SSDs (replication 3) and about 90Gb of OMAP data on each drive. About 75 PGs on each SSD for now. I think 75 is not enough for this amount of data, but I’m not sure that it is critical in our case.
The problem: For the last few weeks, we can see big degradation of some SSD OSDs. We can see a lot of Ops (150-256 in perf dump) for a long time and we can see high avg ops time like 1-3 seconds (this metric is based on dump_historic_ops_by_duration and our avg calculation). These numbers are very unusual for our deployment. And it makes a big impact on our customers.
For now, we make Offline compaction of ‘degraded’ OSDs and after compaction, we see that all PGs of this OSD are returned on this OSD (because recover is disabled during compaction) and we see that OSD works perfect for some time, all our metrics are dropped down for few weeks or only few days…
And, I think the problem is in the rocksdb database which grows by levels and slows down requests.
And I have few question about compaction:
1. Why can't automatic compaction manage this on its own? 2. How can I see RocksDB levels usage or some program metric which can be used as a condition for manual compacting? Because our metrics like request latency, OSD Ops count, OSD avg slow ops time are not 100% relative to rocksdb internal state.
We try to use perf dump, bluestore.kv_xxx_lat metrics, but I think they are absolutely useless. Because we can see higher values after compaction and restart OSD then these metrics were before compaction, lol.
We try to see OSD logs with debug_rocksdb=4, but we can’t understand this output compare it between good and bad OSDs:
Uptime(secs): 3610809.5 total, 600.0 interval Flush(GB): cumulative 50.519, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 475.34 GB write, 0.13 MB/s write, 453.07 GB read, 0.13 MB/s read, 3346.3 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
** File Read Latency Histogram By Level [p-0] **
** Compaction Stats [p-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 52.0 52.0 0.0 1.0 0.0 155.2 343.25 176.27 746 0.460 0 0 L1 4/0 204.00 MB 0.8 96.7 52.0 44.7 96.2 51.4 0.0 1.8 164.6 163.6 601.87 316.34 237 2.540 275M 1237K L2 34/0 2.09 GB 1.0 176.8 43.6 133.2 175.3 42.1 7.6 4.0 147.6 146.3 1226.72 590.20 535 2.293 519M 2672K L3 79/0 4.76 GB 0.3 163.6 38.9 124.7 140.7 16.1 8.8 3.6 152.6 131.3 1097.53 443.60 432 2.541 401M 79M L4 316/0 19.96 GB 0.1 0.9 0.4 0.5 0.8 0.3 19.7 2.0 151.1 131.1 5.94 2.53 4 1.485 2349K 385K Sum 433/0 27.01 GB 0.0 438.0 134.9 303.0 465.0 161.9 36.1 8.9 136.9 145.4 3275.31 1528.94 1954 1.676 1198M 83M Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
and this log is not usable as ‘metric’.
Our Ceph version is: 16.2.13 and we use default bluestore_rocksdb_options. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Josh, Thank you for your reply! It was helpful for me, now I understand that I can't measure rocksdb degradation using program metric ( In our version (16.2.13) we have this code (with new option rocksdb_cf_compact_on_deletion). We will try using it. As I understand, tombstones in the case of RGW it's only deletions of objects, right? Do you knowother cases when tombstones are generated in RGW scenario? We have another option in our version: rocksdb_delete_range_threshold Do you think it can be helpful? I think our problem is raised due to massive deletion generated by the lifecycle ruleof big bucket. On 16.07.2024, 19:25, "Joshua Baergen" <jbaergen@digitalocean.com <mailto:jbaergen@digitalocean.com>> wrote: Hello Aleksandr, What you're probably experiencing is tombstone accumulation, a known issue for Ceph's use of rocksdb.
1. Why can't automatic compaction manage this on its own?
rocksdb compaction is normally triggered by level fullness and not tombstone counts. However, there is a feature in rocksdb that can cause a file to be compacted if there are many tombstones found in it when iterated that can help immensely with tombstone accumulation problems, available as of 16.2.14. You can find a summary of how to enable it and tweak it here: https://www.spinics.net/lists/ceph-users/msg78514.html <https://www.spinics.net/lists/ceph-users/msg78514.html>
2. How can I see RocksDB levels usage or some program metric which can be used as a condition for manual compacting?
There is no tombstone counter that I'm aware of, which is really what you need in order to trigger compaction when appropriate. Josh On Tue, Jul 16, 2024 at 9:12 AM Rudenko Aleksandr <ARudenko@croc.ru <mailto:ARudenko@croc.ru>> wrote:
Hi,
We have a big Ceph cluster (RGW case) with a lot of big buckets 10-500M objects with 31-1024 shards and a lot of io generated by many clients. Index pool placed on enterprise SSDs. We have about 120 SSDs (replication 3) and about 90Gb of OMAP data on each drive. About 75 PGs on each SSD for now. I think 75 is not enough for this amount of data, but I’m not sure that it is critical in our case.
The problem: For the last few weeks, we can see big degradation of some SSD OSDs. We can see a lot of Ops (150-256 in perf dump) for a long time and we can see high avg ops time like 1-3 seconds (this metric is based on dump_historic_ops_by_duration and our avg calculation). These numbers are very unusual for our deployment. And it makes a big impact on our customers.
For now, we make Offline compaction of ‘degraded’ OSDs and after compaction, we see that all PGs of this OSD are returned on this OSD (because recover is disabled during compaction) and we see that OSD works perfect for some time, all our metrics are dropped down for few weeks or only few days…
And, I think the problem is in the rocksdb database which grows by levels and slows down requests.
And I have few question about compaction:
1. Why can't automatic compaction manage this on its own? 2. How can I see RocksDB levels usage or some program metric which can be used as a condition for manual compacting? Because our metrics like request latency, OSD Ops count, OSD avg slow ops time are not 100% relative to rocksdb internal state.
We try to use perf dump, bluestore.kv_xxx_lat metrics, but I think they are absolutely useless. Because we can see higher values after compaction and restart OSD then these metrics were before compaction, lol.
We try to see OSD logs with debug_rocksdb=4, but we can’t understand this output compare it between good and bad OSDs:
Uptime(secs): 3610809.5 total, 600.0 interval Flush(GB): cumulative 50.519, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 475.34 GB write, 0.13 MB/s write, 453.07 GB read, 0.13 MB/s read, 3346.3 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
** File Read Latency Histogram By Level [p-0] **
** Compaction Stats [p-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 52.0 52.0 0.0 1.0 0.0 155.2 343.25 176.27 746 0.460 0 0 L1 4/0 204.00 MB 0.8 96.7 52.0 44.7 96.2 51.4 0.0 1.8 164.6 163.6 601.87 316.34 237 2.540 275M 1237K L2 34/0 2.09 GB 1.0 176.8 43.6 133.2 175.3 42.1 7.6 4.0 147.6 146.3 1226.72 590.20 535 2.293 519M 2672K L3 79/0 4.76 GB 0.3 163.6 38.9 124.7 140.7 16.1 8.8 3.6 152.6 131.3 1097.53 443.60 432 2.541 401M 79M L4 316/0 19.96 GB 0.1 0.9 0.4 0.5 0.8 0.3 19.7 2.0 151.1 131.1 5.94 2.53 4 1.485 2349K 385K Sum 433/0 27.01 GB 0.0 438.0 134.9 303.0 465.0 161.9 36.1 8.9 136.9 145.4 3275.31 1528.94 1954 1.676 1198M 83M Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
and this log is not usable as ‘metric’.
Our Ceph version is: 16.2.13 and we use default bluestore_rocksdb_options. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Hey Aleksandr, rocksdb_delete_range_threshold has had some downsides in the past (I don't have a reference handy) so I don't recommend changing it.
As I understand, tombstones in the case of RGW it's only deletions of objects, right?
It can also happen due to bucket reshards, as this will delete the old shards after completing the reshard. Josh On Wed, Jul 17, 2024 at 3:23 AM Rudenko Aleksandr <ARudenko@croc.ru> wrote:
Hi Josh,
Thank you for your reply!
It was helpful for me, now I understand that I can't measure rocksdb degradation using program metric (
In our version (16.2.13) we have this code (with new option rocksdb_cf_compact_on_deletion). We will try using it. As I understand, tombstones in the case of RGW it's only deletions of objects, right? Do you knowother cases when tombstones are generated in RGW scenario?
We have another option in our version: rocksdb_delete_range_threshold
Do you think it can be helpful?
I think our problem is raised due to massive deletion generated by the lifecycle ruleof big bucket. On 16.07.2024, 19:25, "Joshua Baergen" <jbaergen@digitalocean.com <mailto:jbaergen@digitalocean.com>> wrote:
Hello Aleksandr,
What you're probably experiencing is tombstone accumulation, a known issue for Ceph's use of rocksdb.
1. Why can't automatic compaction manage this on its own?
rocksdb compaction is normally triggered by level fullness and not tombstone counts. However, there is a feature in rocksdb that can cause a file to be compacted if there are many tombstones found in it when iterated that can help immensely with tombstone accumulation problems, available as of 16.2.14. You can find a summary of how to enable it and tweak it here: https://www.spinics.net/lists/ceph-users/msg78514.html <https://www.spinics.net/lists/ceph-users/msg78514.html>
2. How can I see RocksDB levels usage or some program metric which can be used as a condition for manual compacting?
There is no tombstone counter that I'm aware of, which is really what you need in order to trigger compaction when appropriate.
Josh
On Tue, Jul 16, 2024 at 9:12 AM Rudenko Aleksandr <ARudenko@croc.ru <mailto:ARudenko@croc.ru>> wrote:
Hi,
We have a big Ceph cluster (RGW case) with a lot of big buckets 10-500M objects with 31-1024 shards and a lot of io generated by many clients. Index pool placed on enterprise SSDs. We have about 120 SSDs (replication 3) and about 90Gb of OMAP data on each drive. About 75 PGs on each SSD for now. I think 75 is not enough for this amount of data, but I’m not sure that it is critical in our case.
The problem: For the last few weeks, we can see big degradation of some SSD OSDs. We can see a lot of Ops (150-256 in perf dump) for a long time and we can see high avg ops time like 1-3 seconds (this metric is based on dump_historic_ops_by_duration and our avg calculation). These numbers are very unusual for our deployment. And it makes a big impact on our customers.
For now, we make Offline compaction of ‘degraded’ OSDs and after compaction, we see that all PGs of this OSD are returned on this OSD (because recover is disabled during compaction) and we see that OSD works perfect for some time, all our metrics are dropped down for few weeks or only few days…
And, I think the problem is in the rocksdb database which grows by levels and slows down requests.
And I have few question about compaction:
1. Why can't automatic compaction manage this on its own? 2. How can I see RocksDB levels usage or some program metric which can be used as a condition for manual compacting? Because our metrics like request latency, OSD Ops count, OSD avg slow ops time are not 100% relative to rocksdb internal state.
We try to use perf dump, bluestore.kv_xxx_lat metrics, but I think they are absolutely useless. Because we can see higher values after compaction and restart OSD then these metrics were before compaction, lol.
We try to see OSD logs with debug_rocksdb=4, but we can’t understand this output compare it between good and bad OSDs:
Uptime(secs): 3610809.5 total, 600.0 interval Flush(GB): cumulative 50.519, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 475.34 GB write, 0.13 MB/s write, 453.07 GB read, 0.13 MB/s read, 3346.3 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
** File Read Latency Histogram By Level [p-0] **
** Compaction Stats [p-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 52.0 52.0 0.0 1.0 0.0 155.2 343.25 176.27 746 0.460 0 0 L1 4/0 204.00 MB 0.8 96.7 52.0 44.7 96.2 51.4 0.0 1.8 164.6 163.6 601.87 316.34 237 2.540 275M 1237K L2 34/0 2.09 GB 1.0 176.8 43.6 133.2 175.3 42.1 7.6 4.0 147.6 146.3 1226.72 590.20 535 2.293 519M 2672K L3 79/0 4.76 GB 0.3 163.6 38.9 124.7 140.7 16.1 8.8 3.6 152.6 131.3 1097.53 443.60 432 2.541 401M 79M L4 316/0 19.96 GB 0.1 0.9 0.4 0.5 0.8 0.3 19.7 2.0 151.1 131.1 5.94 2.53 4 1.485 2349K 385K Sum 433/0 27.01 GB 0.0 438.0 134.9 303.0 465.0 161.9 36.1 8.9 136.9 145.4 3275.31 1528.94 1954 1.676 1198M 83M Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
and this log is not usable as ‘metric’.
Our Ceph version is: 16.2.13 and we use default bluestore_rocksdb_options. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Hi Josh, thanks! I have one more question. I try to reproduce our OSD degradation due to massive lifecycle deletion and next step I will try to fix rocksdb_cf_compact_on_deletion. But I don't understand one thing. Okay, default auto-compaction can't detect tombstones which are growing, but regular compaction based on file size takes place as I can see. For example, we have a degraded OSD (256 Ops for a few hours). And I can see that some compaction take place: grep -E "Compaction start" /var/log/ceph/ceph-osd.75.log 2024-07-18T14:15:06.759+0300 7f6e67047700 4 rocksdb: [compaction/compaction_job.cc:1680] [default] Compaction start summary: Base version 1410 Base level 0, inputs: [366238(51MB) 366236(52MB) 366234(51MB) 366232(47MB)], [366199(67MB) 366200(67MB) 366201(67MB) 366202(11MB)] 2024-07-18T14:15:09.076+0300 7f6e67047700 4 rocksdb: [compaction/compaction_job.cc:1680] [default] Compaction start summary: Base version 1411 Base level 1, inputs: [366244(66MB)], [366204(66MB) 366205(67MB) 366206(66MB) 366207(66MB) 366208(32MB) 366209(67MB) 366210(66MB)] 2024-07-18T14:15:12.054+0300 7f6e67047700 4 rocksdb: [compaction/compaction_job.cc:1680] [default] Compaction start summary: Base version 1412 Base level 1, inputs: [366240(67MB)], [366154(55MB) 366138(67MB) 366139(67MB) 366140(67MB)] grep -E "compaction_started|compaction_finished" /var/log/ceph/ceph-osd.75.log 2024-07-18T14:15:21.094+0300 7f6e67047700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1721301321095580, "job": 2284, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L3": [366275], "files_L4": [366230, 364501], "score": 1.00751, "input_data_size": 89528358} 2024-07-18T14:15:21.727+0300 7f6e67047700 4 rocksdb: (Original Log Time 2024/07/18-14:15:21.728545) EVENT_LOG_v1 {"time_micros": 1721301321728537, "job": 2284, "event": "compaction_finished", "compaction_time_micros": 627038, "compaction_time_cpu_micros": 440211, "output_level": 4, "num_output_files": 1, "total_output_size": 66142074, "num_input_records": 1642765, "num_output_records": 1057434, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 33, 173, 1004, 0, 0]} And my question is: we have regular compaction that does some work. Why It doesn't help with tombstones? Why only offline compaction can help in our case? On 17.07.2024, 16:14, "Joshua Baergen" <jbaergen@digitalocean.com <mailto:jbaergen@digitalocean.com>> wrote: Hey Aleksandr, rocksdb_delete_range_threshold has had some downsides in the past (I don't have a reference handy) so I don't recommend changing it.
As I understand, tombstones in the case of RGW it's only deletions of objects, right?
It can also happen due to bucket reshards, as this will delete the old shards after completing the reshard. Josh On Wed, Jul 17, 2024 at 3:23 AM Rudenko Aleksandr <ARudenko@croc.ru <mailto:ARudenko@croc.ru>> wrote:
Hi Josh,
Thank you for your reply!
It was helpful for me, now I understand that I can't measure rocksdb degradation using program metric (
In our version (16.2.13) we have this code (with new option rocksdb_cf_compact_on_deletion). We will try using it. As I understand, tombstones in the case of RGW it's only deletions of objects, right? Do you knowother cases when tombstones are generated in RGW scenario?
We have another option in our version: rocksdb_delete_range_threshold
Do you think it can be helpful?
I think our problem is raised due to massive deletion generated by the lifecycle ruleof big bucket. On 16.07.2024, 19:25, "Joshua Baergen" <jbaergen@digitalocean.com <mailto:jbaergen@digitalocean.com> <mailto:jbaergen@digitalocean.com <mailto:jbaergen@digitalocean.com>>> wrote:
Hello Aleksandr,
What you're probably experiencing is tombstone accumulation, a known issue for Ceph's use of rocksdb.
1. Why can't automatic compaction manage this on its own?
rocksdb compaction is normally triggered by level fullness and not tombstone counts. However, there is a feature in rocksdb that can cause a file to be compacted if there are many tombstones found in it when iterated that can help immensely with tombstone accumulation problems, available as of 16.2.14. You can find a summary of how to enable it and tweak it here: https://www.spinics.net/lists/ceph-users/msg78514.html <https://www.spinics.net/lists/ceph-users/msg78514.html> <https://www.spinics.net/lists/ceph-users/msg78514.html> <https://www.spinics.net/lists/ceph-users/msg78514.html>>
2. How can I see RocksDB levels usage or some program metric which can be used as a condition for manual compacting?
There is no tombstone counter that I'm aware of, which is really what you need in order to trigger compaction when appropriate.
Josh
On Tue, Jul 16, 2024 at 9:12 AM Rudenko Aleksandr <ARudenko@croc.ru <mailto:ARudenko@croc.ru> <mailto:ARudenko@croc.ru <mailto:ARudenko@croc.ru>>> wrote:
Hi,
We have a big Ceph cluster (RGW case) with a lot of big buckets 10-500M objects with 31-1024 shards and a lot of io generated by many clients. Index pool placed on enterprise SSDs. We have about 120 SSDs (replication 3) and about 90Gb of OMAP data on each drive. About 75 PGs on each SSD for now. I think 75 is not enough for this amount of data, but I’m not sure that it is critical in our case.
The problem: For the last few weeks, we can see big degradation of some SSD OSDs. We can see a lot of Ops (150-256 in perf dump) for a long time and we can see high avg ops time like 1-3 seconds (this metric is based on dump_historic_ops_by_duration and our avg calculation). These numbers are very unusual for our deployment. And it makes a big impact on our customers.
For now, we make Offline compaction of ‘degraded’ OSDs and after compaction, we see that all PGs of this OSD are returned on this OSD (because recover is disabled during compaction) and we see that OSD works perfect for some time, all our metrics are dropped down for few weeks or only few days…
And, I think the problem is in the rocksdb database which grows by levels and slows down requests.
And I have few question about compaction:
1. Why can't automatic compaction manage this on its own? 2. How can I see RocksDB levels usage or some program metric which can be used as a condition for manual compacting? Because our metrics like request latency, OSD Ops count, OSD avg slow ops time are not 100% relative to rocksdb internal state.
We try to use perf dump, bluestore.kv_xxx_lat metrics, but I think they are absolutely useless. Because we can see higher values after compaction and restart OSD then these metrics were before compaction, lol.
We try to see OSD logs with debug_rocksdb=4, but we can’t understand this output compare it between good and bad OSDs:
Uptime(secs): 3610809.5 total, 600.0 interval Flush(GB): cumulative 50.519, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 475.34 GB write, 0.13 MB/s write, 453.07 GB read, 0.13 MB/s read, 3346.3 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
** File Read Latency Histogram By Level [p-0] **
** Compaction Stats [p-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 52.0 52.0 0.0 1.0 0.0 155.2 343.25 176.27 746 0.460 0 0 L1 4/0 204.00 MB 0.8 96.7 52.0 44.7 96.2 51.4 0.0 1.8 164.6 163.6 601.87 316.34 237 2.540 275M 1237K L2 34/0 2.09 GB 1.0 176.8 43.6 133.2 175.3 42.1 7.6 4.0 147.6 146.3 1226.72 590.20 535 2.293 519M 2672K L3 79/0 4.76 GB 0.3 163.6 38.9 124.7 140.7 16.1 8.8 3.6 152.6 131.3 1097.53 443.60 432 2.541 401M 79M L4 316/0 19.96 GB 0.1 0.9 0.4 0.5 0.8 0.3 19.7 2.0 151.1 131.1 5.94 2.53 4 1.485 2349K 385K Sum 433/0 27.01 GB 0.0 438.0 134.9 303.0 465.0 161.9 36.1 8.9 136.9 145.4 3275.31 1528.94 1954 1.676 1198M 83M Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
and this log is not usable as ‘metric’.
Our Ceph version is: 16.2.13 and we use default bluestore_rocksdb_options. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> <mailto:ceph-users@ceph.io <mailto:ceph-users@ceph.io>> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io> <mailto:ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>>
And my question is: we have regular compaction that does some work. Why It doesn't help with tombstones? Why only offline compaction can help in our case?
Regular compaction will take care of any tombstones in the files that end up being compacted, and compaction, when triggered, may even favor files that have more tombstones in them. However, to my original point, compaction decisions themselves have nothing to do with tombstones. I suggest you read up a bit on rocksdb architecture and log-structured merge trees if you want the details, but the basics are: A tombstone exists until it gets compacted to the bottom-most level. You have 4 levels (plus level 0), so levels 1-3 can have any number of tombstones in them, and there's nothing that guarantees that a given tombstone will get compacted to level 4 with any sort of deadline. (Well, sort of - newer rocksdb, available in Nautilus+ IIRC, has a "TTL" concept set at 30 days by default that ensures no file gets older than 30 days. You can tweak this down to, say, a day or hours, and this effectively introduces an upper bound on most tombstones' lifetime, but also increases compaction traffic substantially. rocksdb_cf_compact_on_deletion is a more intelligent way to achieve a similar effect.) Josh
Josh, thanks! I will read more about LSM in RocksDB, thanks! Can I ask last one question) We have a lot of "old" SSD OSDs in the index pool which were deployed before Pacific. In the Pacific we have RocksDB column families. It will be helpful in the case of many tombstones to do resharding of our old OSDs? Do you think It can help without rocksdb_cf_compact_on_deletion? Or, maybe It can help much more with rocksdb_cf_compact_on_deletion? On 18.07.2024, 18:01, "Joshua Baergen" <jbaergen@digitalocean.com <mailto:jbaergen@digitalocean.com>> wrote:
And my question is: we have regular compaction that does some work. Why It doesn't help with tombstones? Why only offline compaction can help in our case?
Regular compaction will take care of any tombstones in the files that end up being compacted, and compaction, when triggered, may even favor files that have more tombstones in them. However, to my original point, compaction decisions themselves have nothing to do with tombstones. I suggest you read up a bit on rocksdb architecture and log-structured merge trees if you want the details, but the basics are: A tombstone exists until it gets compacted to the bottom-most level. You have 4 levels (plus level 0), so levels 1-3 can have any number of tombstones in them, and there's nothing that guarantees that a given tombstone will get compacted to level 4 with any sort of deadline. (Well, sort of - newer rocksdb, available in Nautilus+ IIRC, has a "TTL" concept set at 30 days by default that ensures no file gets older than 30 days. You can tweak this down to, say, a day or hours, and this effectively introduces an upper bound on most tombstones' lifetime, but also increases compaction traffic substantially. rocksdb_cf_compact_on_deletion is a more intelligent way to achieve a similar effect.) Josh
Hey Aleksandr,
In the Pacific we have RocksDB column families. It will be helpful in the case of many tombstones to do resharding of our old OSDs? Do you think It can help without rocksdb_cf_compact_on_deletion? Or, maybe It can help much more with rocksdb_cf_compact_on_deletion?
Ah, I'm glad you brought this up - there was a bug with the initial rocksdb_cf_compact_on_deletion implementation where it wouldn't take effect for the default column family; this has been corrected only recently, and the bug is still present in Pacific. You'll need to reshard in order for the setting to work. Resharding can also help with reducing compaction load a bit, but I don't think it'll help much with tombstones. Josh
Oh, Josh, thanks! It's very helpful! On 18.07.2024, 20:51, "Joshua Baergen" <jbaergen@digitalocean.com <mailto:jbaergen@digitalocean.com>> wrote: Hey Aleksandr,
In the Pacific we have RocksDB column families. It will be helpful in the case of many tombstones to do resharding of our old OSDs? Do you think It can help without rocksdb_cf_compact_on_deletion? Or, maybe It can help much more with rocksdb_cf_compact_on_deletion?
Ah, I'm glad you brought this up - there was a bug with the initial rocksdb_cf_compact_on_deletion implementation where it wouldn't take effect for the default column family; this has been corrected only recently, and the bug is still present in Pacific. You'll need to reshard in order for the setting to work. Resharding can also help with reducing compaction load a bit, but I don't think it'll help much with tombstones. Josh
Hi Josh, Thank you for sharing this information. Can I ask what symptoms made you interested in tombstones? For the past few months, we've been observing successive waves of a large number of OSDs in overspilling. When the phenomenon occurs, we automatically compact the OSDs (on the fly, one after another, as they appear in overspilling). This calms down for a while, then the problem reappears a few weeks later (March 13 for first time, May 8, June 7). It seems that regular compaction is unable to keep up with or prevent overspilling, despite compression being enabled. Cluster is running Pacific v16.2.10 (RHCS 5.3z6) with resharded and compressed RocksDB (compression was enabled after the first occurrence of this phenomenon, so it's not related to compression I would say). Do you think this phenomenon could be related to tombstones? And that enabling rocksdb_cf_compact_on_deletion would help to avoid this situation? Also, could ttl_option be an alternative to rocksdb_cf_compact_on_deletion that wasn't backported to v16.2.10 (RHCS 5.3z7)? Regards, Frédéric. ----- Le 16 Juil 24, à 18:25, Joshua Baergen jbaergen@digitalocean.com a écrit :
Hello Aleksandr,
What you're probably experiencing is tombstone accumulation, a known issue for Ceph's use of rocksdb.
1. Why can't automatic compaction manage this on its own?
rocksdb compaction is normally triggered by level fullness and not tombstone counts. However, there is a feature in rocksdb that can cause a file to be compacted if there are many tombstones found in it when iterated that can help immensely with tombstone accumulation problems, available as of 16.2.14. You can find a summary of how to enable it and tweak it here: https://www.spinics.net/lists/ceph-users/msg78514.html
2. How can I see RocksDB levels usage or some program metric which can be used as a condition for manual compacting?
There is no tombstone counter that I'm aware of, which is really what you need in order to trigger compaction when appropriate.
Josh
On Tue, Jul 16, 2024 at 9:12 AM Rudenko Aleksandr <ARudenko@croc.ru> wrote:
Hi,
We have a big Ceph cluster (RGW case) with a lot of big buckets 10-500M objects with 31-1024 shards and a lot of io generated by many clients. Index pool placed on enterprise SSDs. We have about 120 SSDs (replication 3) and about 90Gb of OMAP data on each drive. About 75 PGs on each SSD for now. I think 75 is not enough for this amount of data, but I’m not sure that it is critical in our case.
The problem: For the last few weeks, we can see big degradation of some SSD OSDs. We can see a lot of Ops (150-256 in perf dump) for a long time and we can see high avg ops time like 1-3 seconds (this metric is based on dump_historic_ops_by_duration and our avg calculation). These numbers are very unusual for our deployment. And it makes a big impact on our customers.
For now, we make Offline compaction of ‘degraded’ OSDs and after compaction, we see that all PGs of this OSD are returned on this OSD (because recover is disabled during compaction) and we see that OSD works perfect for some time, all our metrics are dropped down for few weeks or only few days…
And, I think the problem is in the rocksdb database which grows by levels and slows down requests.
And I have few question about compaction:
1. Why can't automatic compaction manage this on its own? 2. How can I see RocksDB levels usage or some program metric which can be used as a condition for manual compacting? Because our metrics like request latency, OSD Ops count, OSD avg slow ops time are not 100% relative to rocksdb internal state.
We try to use perf dump, bluestore.kv_xxx_lat metrics, but I think they are absolutely useless. Because we can see higher values after compaction and restart OSD then these metrics were before compaction, lol.
We try to see OSD logs with debug_rocksdb=4, but we can’t understand this output compare it between good and bad OSDs:
Uptime(secs): 3610809.5 total, 600.0 interval Flush(GB): cumulative 50.519, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 475.34 GB write, 0.13 MB/s write, 453.07 GB read, 0.13 MB/s read, 3346.3 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
** File Read Latency Histogram By Level [p-0] **
** Compaction Stats [p-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 52.0 52.0 0.0 1.0 0.0 155.2 343.25 176.27 746 0.460 0 0 L1 4/0 204.00 MB 0.8 96.7 52.0 44.7 96.2 51.4 0.0 1.8 164.6 163.6 601.87 316.34 237 2.540 275M 1237K L2 34/0 2.09 GB 1.0 176.8 43.6 133.2 175.3 42.1 7.6 4.0 147.6 146.3 1226.72 590.20 535 2.293 519M 2672K L3 79/0 4.76 GB 0.3 163.6 38.9 124.7 140.7 16.1 8.8 3.6 152.6 131.3 1097.53 443.60 432 2.541 401M 79M L4 316/0 19.96 GB 0.1 0.9 0.4 0.5 0.8 0.3 19.7 2.0 151.1 131.1 5.94 2.53 4 1.485 2349K 385K Sum 433/0 27.01 GB 0.0 438.0 134.9 303.0 465.0 161.9 36.1 8.9 136.9 145.4 3275.31 1528.94 1954 1.676 1198M 83M Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
and this log is not usable as ‘metric’.
Our Ceph version is: 16.2.13 and we use default bluestore_rocksdb_options. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hey Frédéric,
Can I ask what symptoms made you interested in tombstones?
Mostly poor index performance due to slow rocksdb iterators (the cause being excessive tombstone accumulation).
Do you think this phenomenon could be related to tombstones? And that enabling rocksdb_cf_compact_on_deletion would help to avoid this situation?
I don't have experience with rocksdb spill, unfortunately. If compaction is helping with spill, then more frequent compaction sounds like it could help, though I don't know whether this setting in particular will help.
Also, could ttl_option be an alternative to rocksdb_cf_compact_on_deletion that wasn't backported to v16.2.10 (RHCS 5.3z7)?
Yes, we've successfully used ttl to mitigate tombstone problems before the introduction of rocksdb_cf_compact_on_deletion, and it may be more relevant to your situation. You can add ttl=<seconds> to your rocksdb options and restart OSDs to give it a try. Josh
Hi Rudenko, There's been this bug [1] in the past preventing BlueFS alert from popping up on ceph -s due to some code refactoring. You might just be facing over spilling without noticing. I'm saying this because you're running v16.2.13 and this bug was fixed in v16.2.14 (by [3], based on Pacific release notes [2]). Have you checked slow_used_bytes on 'ceph tell osd.x perf dump bluefs'? Also what's your db_total_bytes? Regards, Frédéric. PS: In Quincy, the bug was fixed in 17.2.7 and in Reef, it should be fixed already since PR was merged even though I can't find it in the release notes. [1] https://tracker.ceph.com/issues/58440 [2] https://docs.ceph.com/en/latest/releases/pacific/#v16-2-14-pacific [3] https://github.com/ceph/ceph/pull/50932 ----- Le 16 Juil 24, à 17:11, Rudenko Aleksandr ARudenko@croc.ru a écrit :
Hi,
We have a big Ceph cluster (RGW case) with a lot of big buckets 10-500M objects with 31-1024 shards and a lot of io generated by many clients. Index pool placed on enterprise SSDs. We have about 120 SSDs (replication 3) and about 90Gb of OMAP data on each drive. About 75 PGs on each SSD for now. I think 75 is not enough for this amount of data, but I’m not sure that it is critical in our case.
The problem: For the last few weeks, we can see big degradation of some SSD OSDs. We can see a lot of Ops (150-256 in perf dump) for a long time and we can see high avg ops time like 1-3 seconds (this metric is based on dump_historic_ops_by_duration and our avg calculation). These numbers are very unusual for our deployment. And it makes a big impact on our customers.
For now, we make Offline compaction of ‘degraded’ OSDs and after compaction, we see that all PGs of this OSD are returned on this OSD (because recover is disabled during compaction) and we see that OSD works perfect for some time, all our metrics are dropped down for few weeks or only few days…
And, I think the problem is in the rocksdb database which grows by levels and slows down requests.
And I have few question about compaction:
1. Why can't automatic compaction manage this on its own? 2. How can I see RocksDB levels usage or some program metric which can be used as a condition for manual compacting? Because our metrics like request latency, OSD Ops count, OSD avg slow ops time are not 100% relative to rocksdb internal state.
We try to use perf dump, bluestore.kv_xxx_lat metrics, but I think they are absolutely useless. Because we can see higher values after compaction and restart OSD then these metrics were before compaction, lol.
We try to see OSD logs with debug_rocksdb=4, but we can’t understand this output compare it between good and bad OSDs:
Uptime(secs): 3610809.5 total, 600.0 interval Flush(GB): cumulative 50.519, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 475.34 GB write, 0.13 MB/s write, 453.07 GB read, 0.13 MB/s read, 3346.3 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
** File Read Latency Histogram By Level [p-0] **
** Compaction Stats [p-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 52.0 52.0 0.0 1.0 0.0 155.2 343.25 176.27 746 0.460 0 0 L1 4/0 204.00 MB 0.8 96.7 52.0 44.7 96.2 51.4 0.0 1.8 164.6 163.6 601.87 316.34 237 2.540 275M 1237K L2 34/0 2.09 GB 1.0 176.8 43.6 133.2 175.3 42.1 7.6 4.0 147.6 146.3 1226.72 590.20 535 2.293 519M 2672K L3 79/0 4.76 GB 0.3 163.6 38.9 124.7 140.7 16.1 8.8 3.6 152.6 131.3 1097.53 443.60 432 2.541 401M 79M L4 316/0 19.96 GB 0.1 0.9 0.4 0.5 0.8 0.3 19.7 2.0 151.1 131.1 5.94 2.53 4 1.485 2349K 385K Sum 433/0 27.01 GB 0.0 438.0 134.9 303.0 465.0 161.9 36.1 8.9 136.9 145.4 3275.31 1528.94 1954 1.676 1198M 83M Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
and this log is not usable as ‘metric’.
Our Ceph version is: 16.2.13 and we use default bluestore_rocksdb_options. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frédéric, Thank you for your reply! We can't face spillover because we use dedicated ssd OSDs (no slow device) which are mapped to index pool in our RGW deployment: ceph tell osd.60 perf dump bluefs | grep slow "slow_total_bytes": 0, "slow_used_bytes": 0, "bytes_written_slow": 0, "max_bytes_slow": 0, On 17.07.2024, 09:45, "Frédéric Nass" <frederic.nass@univ-lorraine.fr <mailto:frederic.nass@univ-lorraine.fr>> wrote: Hi Rudenko, There's been this bug [1] in the past preventing BlueFS alert from popping up on ceph -s due to some code refactoring. You might just be facing over spilling without noticing. I'm saying this because you're running v16.2.13 and this bug was fixed in v16.2.14 (by [3], based on Pacific release notes [2]). Have you checked slow_used_bytes on 'ceph tell osd.x perf dump bluefs'? Also what's your db_total_bytes? Regards, Frédéric. PS: In Quincy, the bug was fixed in 17.2.7 and in Reef, it should be fixed already since PR was merged even though I can't find it in the release notes. [1] https://tracker.ceph.com/issues/58440 <https://tracker.ceph.com/issues/58440> [2] https://docs.ceph.com/en/latest/releases/pacific/#v16-2-14-pacific <https://docs.ceph.com/en/latest/releases/pacific/#v16-2-14-pacific> [3] https://github.com/ceph/ceph/pull/50932 <https://github.com/ceph/ceph/pull/50932> ----- Le 16 Juil 24, à 17:11, Rudenko Aleksandr ARudenko@croc.ru <mailto:ARudenko@croc.ru> a écrit :
Hi,
We have a big Ceph cluster (RGW case) with a lot of big buckets 10-500M objects with 31-1024 shards and a lot of io generated by many clients. Index pool placed on enterprise SSDs. We have about 120 SSDs (replication 3) and about 90Gb of OMAP data on each drive. About 75 PGs on each SSD for now. I think 75 is not enough for this amount of data, but I’m not sure that it is critical in our case.
The problem: For the last few weeks, we can see big degradation of some SSD OSDs. We can see a lot of Ops (150-256 in perf dump) for a long time and we can see high avg ops time like 1-3 seconds (this metric is based on dump_historic_ops_by_duration and our avg calculation). These numbers are very unusual for our deployment. And it makes a big impact on our customers.
For now, we make Offline compaction of ‘degraded’ OSDs and after compaction, we see that all PGs of this OSD are returned on this OSD (because recover is disabled during compaction) and we see that OSD works perfect for some time, all our metrics are dropped down for few weeks or only few days…
And, I think the problem is in the rocksdb database which grows by levels and slows down requests.
And I have few question about compaction:
1. Why can't automatic compaction manage this on its own? 2. How can I see RocksDB levels usage or some program metric which can be used as a condition for manual compacting? Because our metrics like request latency, OSD Ops count, OSD avg slow ops time are not 100% relative to rocksdb internal state.
We try to use perf dump, bluestore.kv_xxx_lat metrics, but I think they are absolutely useless. Because we can see higher values after compaction and restart OSD then these metrics were before compaction, lol.
We try to see OSD logs with debug_rocksdb=4, but we can’t understand this output compare it between good and bad OSDs:
Uptime(secs): 3610809.5 total, 600.0 interval Flush(GB): cumulative 50.519, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 475.34 GB write, 0.13 MB/s write, 453.07 GB read, 0.13 MB/s read, 3346.3 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
** File Read Latency Histogram By Level [p-0] **
** Compaction Stats [p-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 52.0 52.0 0.0 1.0 0.0 155.2 343.25 176.27 746 0.460 0 0 L1 4/0 204.00 MB 0.8 96.7 52.0 44.7 96.2 51.4 0.0 1.8 164.6 163.6 601.87 316.34 237 2.540 275M 1237K L2 34/0 2.09 GB 1.0 176.8 43.6 133.2 175.3 42.1 7.6 4.0 147.6 146.3 1226.72 590.20 535 2.293 519M 2672K L3 79/0 4.76 GB 0.3 163.6 38.9 124.7 140.7 16.1 8.8 3.6 152.6 131.3 1097.53 443.60 432 2.541 401M 79M L4 316/0 19.96 GB 0.1 0.9 0.4 0.5 0.8 0.3 19.7 2.0 151.1 131.1 5.94 2.53 4 1.485 2349K 385K Sum 433/0 27.01 GB 0.0 438.0 134.9 303.0 465.0 161.9 36.1 8.9 136.9 145.4 3275.31 1528.94 1954 1.676 1198M 83M Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
and this log is not usable as ‘metric’.
Our Ceph version is: 16.2.13 and we use default bluestore_rocksdb_options. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
participants (3)
-
Frédéric Nass
-
Joshua Baergen
-
Rudenko Aleksandr