13 Sep
2023
13 Sep
'23
1:40 p.m.
Hi Martin,
it looks like you're using custom osd settings. Namely:
- bluestore_allocator set to bitmap (which is fine)
- bluestore_min_alloc_size set to 128K
The latter is apparently out-of-sync with bluefs_shared_alloc_size (set
to 64K by default). Which causes the assertion at some point due to
unexpected allocation request alignment.
Generally one should have bluefs_shared_alloc_size equal or higher than
(and aligned to) bluestore_min_alloc_size.
I'm curious why have you raised bluestore_min_alloc_size though? I
recall no cases when we've heard of any benefit from that. Particularly
for SSD devices...
I'd recommend to set it back to default.
Thanks,
Igor
On 12/09/2023 19:44, Konold, Martin wrote:
> Hi Igor,
>
> I recreated the log with full debugging enabled.
>
> https://www.konsec.com/download/full-debug-20-ceph-osd.43.log.gz
>
> and another without the debug settings
>
> https://www.konsec.com/download/failed-ceph-osd.43.log.gz
>
> I hope you can draw some conclusions from it and I am looking forward
> to your response.
>
> Regards
> ppa. Martin Konold
>
> --
> Martin Konold - Prokurist, CTO
> KONSEC GmbH - make things real
> Amtsgericht Stuttgart, HRB 23690
> Geschäftsführer: Andreas Mack
> Im Köller 3, 70794 Filderstadt, Germany
>
> On 2023-09-11 22:08, Igor Fedotov wrote:
>> Hi Martin,
>>
>> could you please share the full existing log and also set
>> debug_bluestore and debug_bluefs to 20 and collect new osd startup
>> log.
>>
>>
>> Thanks,
>>
>> Igor
>>
>> On 11/09/2023 20:53, Konold, Martin wrote:
>>> Hi,
>>>
>>> I want to create a new OSD on a 4TB Samsung MZ1L23T8HBLA-00A07
>>> enterprise nvme device in a hyper-converged proxmox 8 environment.
>>>
>>> Creating the OSD works but it cannot be initialized and therefore
>>> not started.
>>>
>>> In the log I see an entry about a failed assert.
>>>
>>> ./src/os/bluestore/fastbmap_allocator_impl.cc: 405: FAILED
>>> ceph_assert((aligned_extent.length % l0_granularity) == 0)
>>>
>>> Is this the culprit?
>>>
>>> In addition at the end of the logfile there is a failed mount and a
>>> failed osd init mentioned.
>>>
>>> 2023-09-11T16:30:04.708+0200 7f99aa28f3c0 -1 bluefs
>>> _check_allocations OP_FILE_UPDATE_INC invalid extent 1:
>>> 0x140000~10000: duplicate reference, ino 30
>>> 2023-09-11T16:30:04.708+0200 7f99aa28f3c0 -1 bluefs mount failed to
>>> replay log: (14) Bad address
>>> 2023-09-11T16:30:04.708+0200 7f99aa28f3c0 20 bluefs _stop_alloc
>>> 2023-09-11T16:30:04.708+0200 7f99aa28f3c0 -1
>>> bluestore(/var/lib/ceph/osd/ceph-43) _open_bluefs failed bluefs
>>> mount: (14) Bad address
>>> 2023-09-11T16:30:04.708+0200 7f99aa28f3c0 10 bluefs
>>> maybe_verify_layout no memorized_layout in bluefs superblock
>>> 2023-09-11T16:30:04.708+0200 7f99aa28f3c0 -1
>>> bluestore(/var/lib/ceph/osd/ceph-43) _open_db failed to prepare db
>>> environment:
>>> 2023-09-11T16:30:04.708+0200 7f99aa28f3c0 1 bdev(0x5565c261fc00
>>> /var/lib/ceph/osd/ceph-43/block) close
>>> 2023-09-11T16:30:04.940+0200 7f99aa28f3c0 -1 osd.43 0 OSD:init:
>>> unable to mount object store
>>> 2023-09-11T16:30:04.940+0200 7f99aa28f3c0 -1 ** ERROR: osd init
>>> failed: (5) Input/output error
>>>
>>> I verified that the hardware of the new nvme is working fine.
>>>