Re: Stuck in upgrade process to reef
Hi Jan, I've just fired an upstream ticket for your case, see https://tracker.ceph.com/issues/64053 for more details. You might want to tune (or preferably just remove) your custom bluestore_cache_.*_ratio settings to fix the issue. This is reproducible and fixable in my lab this way. Hope this helps. Thanks, Igor On 15/01/2024 12:54, Jan Marek wrote:
Hi Igor,
I've tried to start ceph-sod daemon as you advice me and I'm sending log osd.1.start.log
About memory: According to 'top' podman ceph daemon don't reach 2% of whole server memory (64GB)...
I have switch on autotune of memory...
My ceph config dump - see attached dump.txt
Sincerely Jan Marek
Dne Čt, led 11, 2024 at 04:02:02 CET napsal(a) Igor Fedotov:
Hi Jan,
unfortunately this wasn't very helpful. Moreover the log looks a bit messy - looks like a mixture of outputs from multiple running instances or something. I'm not an expert in using containerized setups though.
Could you please simplify things by running ceph-osd process manually like you did for ceph-objectstore-tool. And enforce log output to a file. Command line should look somewhat the following:
ceph-osd -i 0 --log-to-file --log-file <some-file> --debug-bluestore 5/20 --debug-prioritycache 10
Please don't forget to run repair prior to that.
Also you haven't answered my questions about custom [memory] settings and RAM usage during OSD startup. It would be nice to hear some feedback.
Thanks,
Igor
On 11/01/2024 16:47, Jan Marek wrote:
Hi Igor,
I've tried to start osd.1 with debug_prioritycache and debug_bluestore 5/20, see attached file...
Sincerely Jan
Dne St, led 10, 2024 at 01:03:07 CET napsal(a) Igor Fedotov:
Hi Jan,
indeed this looks like some memory allocation problem - may be OSD's RAM usage threshold reached or something?
Curious if you have any custom OSD settings or may be any memory caps for Ceph containers?
Could you please set debug_bluestore to 5/20 and debug_prioritycache to 10 and try to start OSD once again. Please monitor process RAM usage along the process and share the resulting log.
Thanks,
Igor
On 10/01/2024 11:20, Jan Marek wrote:
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Igor, many thanks for advice! I've tried to start osd.1 and it started already, now it's resynchronizing data. I will start daemons one-by-one. What do you mean about osd.0, which have a problem with bluestore fsck? Is there a way to repair it? Sincerely Jan Dne Út, led 16, 2024 at 08:15:03 CET napsal(a) Igor Fedotov:
Hi Jan,
I've just fired an upstream ticket for your case, see https://tracker.ceph.com/issues/64053 for more details.
You might want to tune (or preferably just remove) your custom bluestore_cache_.*_ratio settings to fix the issue.
This is reproducible and fixable in my lab this way.
Hope this helps.
Thanks,
Igor
On 15/01/2024 12:54, Jan Marek wrote:
Hi Igor,
I've tried to start ceph-sod daemon as you advice me and I'm sending log osd.1.start.log
About memory: According to 'top' podman ceph daemon don't reach 2% of whole server memory (64GB)...
I have switch on autotune of memory...
My ceph config dump - see attached dump.txt
Sincerely Jan Marek
Dne Čt, led 11, 2024 at 04:02:02 CET napsal(a) Igor Fedotov:
Hi Jan,
unfortunately this wasn't very helpful. Moreover the log looks a bit messy - looks like a mixture of outputs from multiple running instances or something. I'm not an expert in using containerized setups though.
Could you please simplify things by running ceph-osd process manually like you did for ceph-objectstore-tool. And enforce log output to a file. Command line should look somewhat the following:
ceph-osd -i 0 --log-to-file --log-file <some-file> --debug-bluestore 5/20 --debug-prioritycache 10
Please don't forget to run repair prior to that.
Also you haven't answered my questions about custom [memory] settings and RAM usage during OSD startup. It would be nice to hear some feedback.
Thanks,
Igor
On 11/01/2024 16:47, Jan Marek wrote:
Hi Igor,
I've tried to start osd.1 with debug_prioritycache and debug_bluestore 5/20, see attached file...
Sincerely Jan
Dne St, led 10, 2024 at 01:03:07 CET napsal(a) Igor Fedotov:
Hi Jan,
indeed this looks like some memory allocation problem - may be OSD's RAM usage threshold reached or something?
Curious if you have any custom OSD settings or may be any memory caps for Ceph containers?
Could you please set debug_bluestore to 5/20 and debug_prioritycache to 10 and try to start OSD once again. Please monitor process RAM usage along the process and share the resulting log.
Thanks,
Igor
On 10/01/2024 11:20, Jan Marek wrote:
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html
Hi Jan, w.r.t. osd.0 - if this is the only occurrence then I'd propose simply redeploy the OSD. This looks like some BlueStore metadata inconsistency which could occur long before the upgrade. Likely the upgrade just revealed the issue. And honestly I can hardly imagine how to investigate it at this point. Let's see how further upgrades go and come back to this question if more similar issues pop up. Meanwhile I'd recommend to run fsck for every OSD prior to upgrade to get clear understanding if metadata is consistent or not. This way - if occurred once again - we can prove/disprove my statement about the issue being unrelated to upgrades above. Thanks, Igor On 17/01/2024 15:07, Jan Marek wrote:
Hi Igor,
many thanks for advice!
I've tried to start osd.1 and it started already, now it's resynchronizing data.
I will start daemons one-by-one.
What do you mean about osd.0, which have a problem with bluestore fsck? Is there a way to repair it?
Sincerely Jan
Dne Út, led 16, 2024 at 08:15:03 CET napsal(a) Igor Fedotov:
Hi Jan,
I've just fired an upstream ticket for your case, see https://tracker.ceph.com/issues/64053 for more details.
You might want to tune (or preferably just remove) your custom bluestore_cache_.*_ratio settings to fix the issue.
This is reproducible and fixable in my lab this way.
Hope this helps.
Thanks,
Igor
On 15/01/2024 12:54, Jan Marek wrote:
Hi Igor,
I've tried to start ceph-sod daemon as you advice me and I'm sending log osd.1.start.log
About memory: According to 'top' podman ceph daemon don't reach 2% of whole server memory (64GB)...
I have switch on autotune of memory...
My ceph config dump - see attached dump.txt
Sincerely Jan Marek
Dne Čt, led 11, 2024 at 04:02:02 CET napsal(a) Igor Fedotov:
Hi Jan,
unfortunately this wasn't very helpful. Moreover the log looks a bit messy - looks like a mixture of outputs from multiple running instances or something. I'm not an expert in using containerized setups though.
Could you please simplify things by running ceph-osd process manually like you did for ceph-objectstore-tool. And enforce log output to a file. Command line should look somewhat the following:
ceph-osd -i 0 --log-to-file --log-file <some-file> --debug-bluestore 5/20 --debug-prioritycache 10
Please don't forget to run repair prior to that.
Also you haven't answered my questions about custom [memory] settings and RAM usage during OSD startup. It would be nice to hear some feedback.
Thanks,
Igor
On 11/01/2024 16:47, Jan Marek wrote:
Hi Igor,
I've tried to start osd.1 with debug_prioritycache and debug_bluestore 5/20, see attached file...
Sincerely Jan
Dne St, led 10, 2024 at 01:03:07 CET napsal(a) Igor Fedotov:
Hi Jan,
indeed this looks like some memory allocation problem - may be OSD's RAM usage threshold reached or something?
Curious if you have any custom OSD settings or may be any memory caps for Ceph containers?
Could you please set debug_bluestore to 5/20 and debug_prioritycache to 10 and try to start OSD once again. Please monitor process RAM usage along the process and share the resulting log.
Thanks,
Igor
On 10/01/2024 11:20, Jan Marek wrote:
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Igor Fedotov
-
Jan Marek