Hi Jan, w.r.t. osd.0 - if this is the only occurrence then I'd propose simply redeploy the OSD. This looks like some BlueStore metadata inconsistency which could occur long before the upgrade. Likely the upgrade just revealed the issue.  And honestly I can hardly imagine how to investigate it at this point. Let's see how further upgrades go and come back to this question if more similar issues pop up. Meanwhile I'd recommend to run fsck for every OSD prior to upgrade to get clear understanding if metadata is consistent or not. This way - if occurred once again - we can prove/disprove my statement about the issue being unrelated to upgrades above. Thanks, Igor On 17/01/2024 15:07, Jan Marek wrote: > Hi Igor, > > many thanks for advice! > > I've tried to start osd.1 and it started already, now it's > resynchronizing data. > > I will start daemons one-by-one. > > What do you mean about osd.0, which have a problem with > bluestore fsck? Is there a way to repair it? > > Sincerely > Jan > > > Dne Út, led 16, 2024 at 08:15:03 CET napsal(a) Igor Fedotov: >> Hi Jan, >> >> I've just fired an upstream ticket for your case, see >> https://tracker.ceph.com/issues/64053 for more details. >> >> >> You might want to tune (or preferably just remove) your custom >> bluestore_cache_.*_ratio settings to fix the issue. >> >> This is reproducible and fixable in my lab this way. >> >> Hope this helps. >> >> >> Thanks, >> >> Igor >> >> >> On 15/01/2024 12:54, Jan Marek wrote: >>> Hi Igor, >>> >>> I've tried to start ceph-sod daemon as you advice me and I'm >>> sending log osd.1.start.log >>> >>> About memory: According to 'top' podman ceph daemon don't reach >>> 2% of whole server memory (64GB)... >>> >>> I have switch on autotune of memory... >>> >>> My ceph config dump - see attached dump.txt >>> >>> Sincerely >>> Jan Marek >>> >>> Dne Čt, led 11, 2024 at 04:02:02 CET napsal(a) Igor Fedotov: >>>> Hi Jan, >>>> >>>> unfortunately this wasn't very helpful. Moreover the log looks a bit messy - >>>> looks like a mixture of outputs from multiple running instances or >>>> something. I'm not an expert in using containerized setups though. >>>> >>>> Could you please simplify things by running ceph-osd process manually like >>>> you did for ceph-objectstore-tool. And enforce log output to a file. Command >>>> line should look somewhat the following: >>>> >>>> ceph-osd -i 0 --log-to-file --log-file <some-file> --debug-bluestore 5/20 >>>> --debug-prioritycache 10 >>>> >>>> Please don't forget to run repair prior to that. >>>> >>>> >>>> Also you haven't answered my questions about custom [memory] settings and >>>> RAM usage during OSD startup. It would be nice to hear some feedback. >>>> >>>> >>>> Thanks, >>>> >>>> Igor >>>> >>>> On 11/01/2024 16:47, Jan Marek wrote: >>>>> Hi Igor, >>>>> >>>>> I've tried to start osd.1 with debug_prioritycache and >>>>> debug_bluestore 5/20, see attached file... >>>>> >>>>> Sincerely >>>>> Jan >>>>> >>>>> Dne St, led 10, 2024 at 01:03:07 CET napsal(a) Igor Fedotov: >>>>>> Hi Jan, >>>>>> >>>>>> indeed this looks like some memory allocation problem - may be OSD's RAM >>>>>> usage threshold reached or something? >>>>>> >>>>>> Curious if you have any custom OSD settings or may be any memory caps for >>>>>> Ceph containers? >>>>>> >>>>>> Could you please set debug_bluestore to 5/20 and debug_prioritycache to 10 >>>>>> and try to start OSD once again. Please monitor process RAM usage along the >>>>>> process and share the resulting log. >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Igor >>>>>> >>>>>> On 10/01/2024 11:20, Jan Marek wrote: >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io