I have no idea why, but it worked.
As the fsck went well, I just re did the bluefs-bdev-new-db and now the
OSD is back up, with a block.db device.
Thanks a lot
Am Mo., 17. Mai 2021 um 15:28 Uhr schrieb Igor Fedotov <ifedotov(a)suse.de>de>:
If you haven't had successful OSD.68 starts
with standalone DB I think
it's safe to revert previous DB adding and just retry it.
At first suggest to run bluefs-bdev-new-db command only and then do fsck
again. If it's OK - proceed with bluefs migrate followed by another
fsck. And then finalize with adding lvm tags and OSD activation.
Thanks,
Igor
On 5/17/2021 3:47 PM, Boris Behrens wrote:
The FSCK looks good:
[root@s3db10 export-bluefs2]# ceph-bluestore-tool --path
/var/lib/ceph/osd/ceph-68 fsck
fsck success
Am Mo., 17. Mai 2021 um 14:39 Uhr schrieb Boris Behrens <bb(a)kervyn.de>de>:
> Here is the new output. I kept both for now.
>
> [root@s3db10 export-bluefs2]# ls *
> db:
> 018215.sst 018444.sst 018839.sst 019074.sst 019210.sst 019381.sst
> 019560.sst 019755.sst 019849.sst 019888.sst 019958.sst
019995.sst
> 020007.sst 020042.sst 020067.sst
020098.sst 020115.sst
> 018216.sst 018445.sst 018840.sst 019075.sst 019211.sst 019382.sst
> 019670.sst 019756.sst 019877.sst 019889.sst 019959.sst
019996.sst
> 020008.sst 020043.sst 020068.sst
020104.sst CURRENT
> 018273.sst 018446.sst 018876.sst 019076.sst 019256.sst 019383.sst
> 019671.sst 019757.sst 019878.sst 019890.sst 019960.sst
019997.sst
> 020030.sst 020055.sst 020069.sst
020105.sst IDENTITY
> 018300.sst 018447.sst 018877.sst 019081.sst 019257.sst 019395.sst
> 019672.sst 019762.sst 019879.sst 019918.sst 019961.sst
019998.sst
> 020031.sst 020056.sst 020070.sst
020106.sst LOCK
> 018301.sst 018448.sst 018904.sst 019082.sst 019344.sst 019396.sst
> 019673.sst 019763.sst 019880.sst 019919.sst 019962.sst
019999.sst
> 020032.sst 020057.sst 020071.sst
020107.sst MANIFEST-020084
> 018326.sst 018449.sst 018950.sst 019083.sst 019345.sst 019400.sst
> 019674.sst 019764.sst 019881.sst 019920.sst 019963.sst
020000.sst
> 020035.sst 020058.sst 020072.sst
020108.sst OPTIONS-020084
> 018327.sst 018540.sst 018952.sst 019126.sst 019346.sst 019470.sst
> 019675.sst 019765.sst 019882.sst 019921.sst 019964.sst
020001.sst
> 020036.sst 020059.sst 020073.sst
020109.sst OPTIONS-020087
> 018328.sst 018541.sst 018953.sst 019127.sst 019370.sst 019471.sst
> 019676.sst 019766.sst 019883.sst 019922.sst 019965.sst
020002.sst
> 020037.sst 020060.sst 020074.sst
020110.sst
> 018329.sst 018590.sst 018954.sst 019128.sst 019371.sst 019472.sst
> 019677.sst 019845.sst 019884.sst 019923.sst 019989.sst
020003.sst
> 020038.sst 020061.sst 020075.sst
020111.sst
> 018406.sst 018591.sst 018995.sst 019174.sst 019372.sst 019473.sst
> 019678.sst 019846.sst 019885.sst 019950.sst 019992.sst
020004.sst
> 020039.sst 020062.sst 020094.sst
020112.sst
> 018407.sst 018727.sst 018996.sst 019175.sst 019373.sst 019474.sst
> 019753.sst 019847.sst 019886.sst 019955.sst 019993.sst
020005.sst
> 020040.sst 020063.sst 020095.sst
020113.sst
> 018443.sst 018728.sst 019073.sst 019176.sst 019380.sst 019475.sst
> 019754.sst 019848.sst 019887.sst 019956.sst 019994.sst
020006.sst
> 020041.sst 020064.sst 020096.sst
020114.sst
>
> db.slow:
>
> db.wal:
> 020085.log 020088.log
> [root@s3db10 export-bluefs2]# du -hs
> 12G .
> [root@s3db10 export-bluefs2]# cat db/CURRENT
> MANIFEST-020084
>
> Am Mo., 17. Mai 2021 um 14:28 Uhr schrieb Igor Fedotov <
ifedotov(a)suse.de>gt;:
>
>> On 5/17/2021 2:53 PM, Boris Behrens wrote:
>>> Like this?
>> Yeah.
>>
>> so DB dir structure is more or less O but db/CURRENT looks corrupted.
It
>> should contain something like:
MANIFEST-020081
>>
>> Could you please remove (or even just rename) block.db symlink and do
>> the export again? Preferably to preserve the results for the first
export.
>>
>> if export reveals proper CURRENT content - you might want to run fsck
on
>> the OSD...
>>
>>> [root@s3db10 export-bluefs]# ls *
>>> db:
>>> 018215.sst 018444.sst 018839.sst 019074.sst 019174.sst
019372.sst
>>> 019470.sst 019675.sst 019765.sst
019882.sst 019918.sst
019961.sst
>>> 019997.sst 020022.sst 020042.sst
020061.sst 020073.sst
>>> 018216.sst 018445.sst 018840.sst 019075.sst 019175.sst
019373.sst
>>> 019471.sst 019676.sst 019766.sst
019883.sst 019919.sst
019962.sst
>>> 019998.sst 020023.sst 020043.sst
020062.sst 020074.sst
>>> 018273.sst 018446.sst 018876.sst 019076.sst 019176.sst
019380.sst
>>> 019472.sst 019677.sst 019845.sst
019884.sst 019920.sst
019963.sst
>>> 019999.sst 020030.sst 020049.sst
020063.sst 020075.sst
>>> 018300.sst 018447.sst 018877.sst 019077.sst 019210.sst
019381.sst
>>> 019473.sst 019678.sst 019846.sst
019885.sst 019921.sst
019964.sst
>>> 020000.sst 020031.sst 020051.sst
020064.sst 020077.sst
>>> 018301.sst 018448.sst 018904.sst 019081.sst 019211.sst
019382.sst
>>> 019474.sst 019753.sst 019847.sst
019886.sst 019922.sst
019965.sst
>>> 020001.sst 020032.sst 020052.sst
020065.sst 020080.sst
>>> 018326.sst 018449.sst 018950.sst 019082.sst 019256.sst
019383.sst
>>> 019475.sst 019754.sst 019848.sst
019887.sst 019923.sst
019986.sst
>>> 020002.sst 020035.sst 020053.sst
020066.sst CURRENT
>>> 018327.sst 018540.sst 018952.sst 019083.sst 019257.sst
019395.sst
>>> 019560.sst 019755.sst 019849.sst
019888.sst 019950.sst
019989.sst
>>> 020003.sst 020036.sst 020055.sst
020067.sst IDENTITY
>>> 018328.sst 018541.sst 018953.sst 019124.sst 019344.sst
019396.sst
>>> 019670.sst 019756.sst 019877.sst
019889.sst 019955.sst
019992.sst
>>> 020004.sst 020037.sst 020056.sst
020068.sst LOCK
>>> 018329.sst 018590.sst 018954.sst 019125.sst 019345.sst
019400.sst
>>> 019671.sst 019757.sst 019878.sst
019890.sst 019956.sst
019993.sst
>>> 020005.sst 020038.sst 020057.sst
020069.sst MANIFEST-020081
>>> 018406.sst 018591.sst 018995.sst 019126.sst 019346.sst
019467.sst
>>> 019672.sst 019762.sst 019879.sst
019915.sst 019958.sst
019994.sst
>>> 020006.sst 020039.sst 020058.sst
020070.sst OPTIONS-020081
>>> 018407.sst 018727.sst 018996.sst 019127.sst 019370.sst
019468.sst
>>> 019673.sst 019763.sst 019880.sst
019916.sst 019959.sst
019995.sst
>>> 020007.sst 020040.sst 020059.sst
020071.sst OPTIONS-020084
>>> 018443.sst 018728.sst 019073.sst 019128.sst 019371.sst
019469.sst
>>> 019674.sst 019764.sst 019881.sst
019917.sst 019960.sst
019996.sst
>>> 020008.sst 020041.sst 020060.sst
020072.sst
>>>
>>> db.slow:
>>>
>>> db.wal:
>>> 020082.log
>>> [root@s3db10 export-bluefs]# du -hs
>>> 12G .
>>> [root@s3db10 export-bluefs]# cat db/CURRENT
>>> �g�U
>>> uN�[�+p[root@s3db10 export-bluefs]#
>>>
>>> Am Mo., 17. Mai 2021 um 13:45 Uhr schrieb Igor Fedotov <
>> ifedotov(a)suse.de>gt;:
>>>> You might want to check file structure at new DB using
>> bluestore-tools's
>>>> bluefs-export command:
>>>>
>>>> ceph-bluestore-tool --path <osd-path> --command bluefs-export
--out
>>>> <target-dir>
>>>>
>>>> <target-dir> needs to have enough free space enough to fit DB
data.
>>>>
>>>> Once exported - does <target-dir> contain valid BlueFS directory
>>>> structure - multiple .sst files, CURRENT and IDENTITY files etc?
>>>>
>>>> If so then please check and share the content of
>> <target-dir>/db/CURRENT
>>>> file.
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Igor
>>>>
>>>> On 5/17/2021 1:32 PM, Boris Behrens wrote:
>>>>> Hi Igor,
>>>>> I posted it on pastebin:
https://pastebin.com/Ze9EuCMD
>>>>>
>>>>> Cheers
>>>>> Boris
>>>>>
>>>>> Am Mo., 17. Mai 2021 um 12:22 Uhr schrieb Igor Fedotov <
>> ifedotov(a)suse.de
>>>>> :
>>>>>
>>>>>> Hi Boris,
>>>>>>
>>>>>> could you please share full OSD startup log and file listing for
>>>>>> '/var/lib/ceph/osd/ceph-68'?
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Igor
>>>>>>
>>>>>> On 5/17/2021 1:09 PM, Boris Behrens wrote:
>>>>>>> Hi,
>>>>>>> sorry for replying to this old thread:
>>>>>>>
>>>>>>> I tried to add a block.db to an OSD but now the OSD can not
start
>> with
>>>>>> the
>>>>>>> error:
>>>>>>> Mai 17 09:50:38 s3db10.fra2.gridscale.it ceph-osd[26038]:
-7>
>>>> 2021-05-17
>>>>>>> 09:50:38.362 7fc48ec94a80 -1 rocksdb: Corruption: CURRENT
file
does
>> not
>>>>>> end
>>>>>>> with newline
>>>>>>> Mai 17 09:50:38 s3db10.fra2.gridscale.it ceph-osd[26038]:
-6>
>>>> 2021-05-17
>>>>>>> 09:50:38.362 7fc48ec94a80 -1
bluestore(/var/lib/ceph/osd/ceph-68)
>>>>>> _open_db
>>>>>>> erroring opening db:
>>>>>>> Mai 17 09:50:38 s3db10.fra2.gridscale.it ceph-osd[26038]:
-1>
>>>> 2021-05-17
>>>>>>> 09:50:38.866 7fc48ec94a80 -1
>>>>>>>
>>
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.21/rpm/el7/BUILD/ceph-14.2.21/src/os/bluestore/BlueStore.cc:
>>>>>>> In function 'int
BlueStore::_upgrade_super()' thread 7fc48ec94a80
>> time
>>>>>>> 2021-05-17 09:50:38.865204
>>>>>>> Mai 17 09:50:38 s3db10.fra2.gridscale.it ceph-osd[26038]:
>>>>>>>
>>
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.21/rpm/el7/BUILD/ceph-14.2.21/src/os/bluestore/BlueStore.cc:
>>>>>>> 10647: FAILED
ceph_assert(ondisk_format > 0)
>>>>>>>
>>>>>>> I tried to run an fsck/repair on the disk:
>>>>>>> [root@s3db10 osd]# ceph-bluestore-tool --path ceph-68
repair
>>>>>>> 2021-05-17 10:05:25.695 7f714dea3ec0 -1 rocksdb: Corruption:
CURRENT
>>>> file
>>>>>>> does not end with newline
>>>>>>> 2021-05-17 10:05:25.695 7f714dea3ec0 -1 bluestore(ceph-68)
_open_db
>>>>>>> erroring opening db:
>>>>>>> error from fsck: (5) Input/output error
>>>>>>> [root@s3db10 osd]# ceph-bluestore-tool --path ceph-68 fsck
>>>>>>> 2021-05-17 10:05:35.012 7fb8f22e6ec0 -1 rocksdb: Corruption:
CURRENT
>>>> file
>>>>>>> does not end with newline
>>>>>>> 2021-05-17 10:05:35.012 7fb8f22e6ec0 -1 bluestore(ceph-68)
_open_db
>>>>>>> erroring opening db:
>>>>>>> error from fsck: (5) Input/output error
>>>>>>>
>>>>>>> These are the steps I did to add the disk:
>>>>>>> $ CEPH_ARGS="--bluestore-block-db-size 53687091200
>>>>>>> --bluestore_block_db_create=true" ceph-bluestore-tool
>>>> bluefs-bdev-new-db
>>>>>>> --path /var/lib/ceph/osd/ceph-68 --dev-target /dev/sdj1
>>>>>>> $ chown -h ceph:ceph /var/lib/ceph/osd/ceph-68/block.db
>>>>>>> $ lvchange --addtag ceph.db_device=/dev/sdj1
>>>>>>>
>>
/dev/ceph-3bbfd168-2a54-4593-a037-80d0d7e97afd/osd-block-aaeaea54-eb6a-480c-b2fd-d938e336c0f6
>>>>>>> $ lvchange --addtag
>> ceph.db_uuid=463dd37c-fd49-4ccb-849f-c5827d3d9df2
>>
/dev/ceph-3bbfd168-2a54-4593-a037-80d0d7e97afd/osd-block-aaeaea54-eb6a-480c-b2fd-d938e336c0f6
>>>>>>> $ ceph-volume lvm
activate --all
>>>>>>>
>>>>>>> The UUIDs
>>>>>>> later I tried this:
>>>>>>> $ ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-68
--devs-source
>>>>>>>
/var/lib/ceph/osd/ceph-68/block --dev-target
>>>>>>> /var/lib/ceph/osd/ceph-68/block.db bluefs-bdev-migrate
>>>>>>>
>>>>>>> Any ideas how I can get the rocksdb fixed?
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>>>>
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.