Igor, Travis, Thanks for your attention to this issue. We extended the timeout for the liveness probe yesterday, and also extended the time after which a down OSD deployment is deleted by the operator. Once all the OSD deployments were recreated by the operator, we observed two OSD restarts - which is a much lower rate than earlier. Igor, we are still working on piecing together logs (from our log store) before the OSD restarts and will send them shortly. Thanks. Sudhin On Thu, Sep 21, 2023 at 3:12 PM Travis Nielsen <tnielsen(a)redhat.com> wrote:

Hi Igor, Please let where can I upload the OSD logs. Thanks. Sudhin

Hi Igor, I have copied three OSD logs to https://drive.google.com/file/d/1aQxibFJR6Dzvr3RbuqnpPhaSMhPSL--F/view?usp=… Hopefully they include some meaningful information. Thank you. Sudhin

There are some pretty strange compaction behavior happening in these logs.  For instance, in osd0, we see a O-1 CF L1 compaction that's taking ~204 seconds: 2023-09-21T20:03:59.378+0000 7f16a286c700  4 rocksdb: (Original Log Time 2023/09/21-20:03:59.381808) EVENT_LOG_v1 {"time_micros": 1695326639381779, "job": 17834, "event": "compaction_finished", "compaction_time_micros": 204158593, "compaction_time_cpu_micros": 180777275, "output_level": 1, "num_output_files": 36, "total_output_size": 1006338775, "num_input_records": 2547576, "num_output_records": 2479913, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 36, 205, 393, 0, 0, 0]} That's compacting at only 4.7MB/s. Yet a little later on, we see an O-1 L3 compaction taking around 20 seconds despite compacting more data: 2023-09-21T20:04:40.466+0000 7f16a286c700  4 rocksdb: (Original Log Time 2023/09/21-20:04:40.472527) EVENT_LOG_v1 {"time_micros": 1695326680472501, "job": 17848, "event": "compaction_finished", "compaction_time_micros": 20166590, "compaction_time_cpu_micros": 16105672, "output_level": 3, "num_output_files": 24, "total_output_size": 1597043099, "num_input_records": 3342658, "num_output_records": 3247386, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 35, 206, 393, 0, 0, 0]} With a compaction rate of around 75.5MB/s. For OSD1, the compaction behavior is even worse!  An example O-0 L1 compaction took ~969 seconds! 2023-09-22T11:24:14.206+0000 7fc4e4807700  4 rocksdb: (Original Log Time 2023/09/22-11:24:14.211454) EVENT_LOG_v1 {"time_micros": 1695381854211381, "job": 28548, "event": "compaction_finished", "compaction_time_micros": 968781445, "compaction_time_cpu_micros": 844222174, "output_level": 1, "num_output_files": 56, "total_output_size": 1006413354, "num_input_records": 2649457, "num_output_records": 2579105, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 56, 297, 359, 0, 0, 0]} With a compaction rate of around 0.99MB/s. OSD2's worst case compaction rate however was around 19MB/s, but was often significantly faster (~100-200MB/s) I'm leaning toward hybrid allocator issues or hardware issues here, because these compaction rates are all over the place and much slower than expected on flash hardware. Mark On 9/28/23 06:22, Igor Fedotov wrote: -- Best Regards, Mark Nelson Head of Research and Development Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com | e: mark.nelson(a)clyso.com We are hiring: https://www.clyso.com/jobs/