16 Mar
2020
16 Mar
'20
5:15 a.m.
Hi,
We have a production cluster that just suffered an issue with multiple
of our NVMe OSDs. Multiple of them died (>12) with errors that they no
longer had space with a 'ENOSPC from bluestore, misconfigured cluster'
error over 4 nodes. These are all simple one device bluestore osds.
ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus
(stable)
This is an example[0] of one of the logs. In this case each of 8 NVMe
OSDs on a node have 106GB of space allocated to each bluestore NVMe OSD.
The ceph-bluestore-tool bluefs-bdev-sizes output only lists 22GiB for
osd 681. I extended the space of bluestore on a few of the OSDs via LVM
and then the bluefs-bdev-expand command. This worked for a few and not
for others.
Some of the ones that it did work for recovered for a bit then
re-entered the error state. Trying to extend the allocation didn't work
after that. When they failed again I ran the fsck which reported that
it found 1 error and then running repair I got a rather long stack trace[1].
# ceph-bluestore-tool --log-level 30 --command bluefs-bdev-sizes --path
/var/lib/ceph/osd/ceph-681
inferring bluefs devices from bluestore path
slot 1 /var/lib/ceph/osd/ceph-681/block -> /dev/dm-33
1 : device size 0x1a80000000 : own
0x[2480000~10000,24a0000~10000,2520000~60000,25f0000~c0000,2720000~50000,28a0000~110000,2a20000~230000,2cc0000~260000,2f30000~220000,31c0000~6b0000,38a0000~10000,3990000~3e0000,3d80000~530000,42d0000~590000,48d0000~400000,4d00000~7d0000,54f0000~c50000,6150000~10000,6190000~150000,6350000~c0000,6480000~160000,6640000~1e0000,6870000~c0000,6a00000~30000,6a40000~240000,6dd0000~310000,7210000~b0000,73a0000~b0000,76a0000~180000,7830000~80000,78e0000~240000,7b70000~90000,7c50000~b0000,7ef0000~140000,8040000~30000,8180000~250000,8440000~50000,84b0000~110000,8610000~c0000,9e20000~20000,9e50000~b0000,9f10000~60000,9f80000~30000,dd80000~180000,df70000~6a0000,e620000~5ae0000,15200000~3510000,187f0000~bf0000,19490000~1070000,1ab70000~4c0000,1b400000~7d0000,1bbe0000~c20000,1cd10000~340000,1d3a0000~860000,1dd00000~2e00000,20c00000~3f00000,24d00000~700000,25600000~700000,26100000~200000,26400000~300000,26b00000~600000,27400000~400000,27ba0000~6e0000,28500000~1d00000,2a400000~700000,2ac00000~100000,2b100000~300000,2b470000~120000,2b700000~500000,2c000000~200000,2c400000~400000,2ca00000~100000,2cf00000~300000,2d340000~39b0000,30d00000~1f00000,32e00000~4bf0000,380a0000~3c0000,38500000~c0000,38bd0000~400000,390b0000~340000,39400000~100000,39900000~1000000,3ac00000~5d00000,40b90000~400000,41280000~db50000,4ee00000~700000,4f900000~4500000,54390000~100000,54e00000~18400000,6d800000~20d0000,6f8f0000~1a10000,71400000~4500000,76100000~300000,766e0000~6860000,7dd00000~c00000,7eac0000~a0000,7ef90000~f190000,8e1f0000~80000,8e410000~60000,8e480000~20000,8e4b0000~20000,8e5c0000~50000,8e7e0000~50000,8f160000~60000,8f240000~a0000,90000000~15e90000,a6200000~c3a0000,b25d0000~630000,b3000000~c00000,b3ee0000~90000,b4200000~d00000,b5a70000~160000,b63f0000~2a0000,b6720000~2820000,bab00000~400000,bbf60000~10ad0000,ccb90000~2300000,cf000000~2b00000,d1ca0000~10000,d1e00000~1400000,d3230000~1df0000,d5200000~1a00000,d6d00000~800000,d75e0000~6f0000,d7f00000~d00000,d9100000~400000,d9900000~d00000,da800000~600000,daf10000~400000,db700000~1600000,dd280000~20000,dd670000~390000,dda30000~400000,de190000~70000,de2a0000~370000,de660000~20000,de700000~14770000,f3600000~700000,f3db0000~960000,f49e0000~5b00000,fa600000~c00000,fb300000~510000,fbb00000~100000,fbeb0000~450000,fc400000~2b0000,fd400000~400000,fde00000~c00000,ff0b0000~50000,ff200000~800000,ffd60000~10000,fff00000~a0000,100200000~300000,101600000~100000,101750000~300000,102120000~1e0000,1027f0000~a00000,103600000~330000,103b00000~200000,103e60000~4a0000,104310000~c00000,105030000~1200000,106800000~100000,106b20000~400000,107000000~300000,1073e0000~400000,107950000~86b0000,110140000~d0000,110350000~2e0000,110e20000~20000,110eb0000~a0000,110f60000~60000,110fd0000~1f0000,1112a0000~f0000,111420000~30000,1115b0000~30000,111620000~150000,111790000~40000,112560000~180000,112730000~180000,1129b0000~50000,112f90000~4a0000,113840000~c0000,113ea0000~40000,113fb0000~130000,114100000~310000,114470000~10000,114620000~120000,114810000~120000,114a00000~20000,114a90000~f0000,114c60000~e0000,114e80000~20000,114f70000~140000,1150c0000~50000,1151f0000~320000,1155f0000~10000,115670000~226f0000,137e30000~800000,138b50000~400000,139400000~1500000,13ae00000~4500000,13f480000~400000,13f950000~1b6e0000,15b700000~400000,15c000000~300000,15c600000~700000,15d9e0000~1820000,15f400000~c00000,160400000~d00000,1613a0000~630000,1619e0000~d20000,162800000~1b00000,164600000~7550000,170d00000~1800000,172580000~100000,172d70000~1c190000,18f100000~40e0000,193700000~400000,193c90000~6970000,19aa90000~188d0000,1b3770000~c220000,1bff00000~1200000,1c12f0000~400000,1c2c00000~400000,1c3f80000~400000,1c5300000~4200000,1c95c0000~b9a40000,283800000~1800000,289000000~3800000,28d000000~6000000,293800000~2000000,296000000~e000000,2a4800000~465d0000,4b6100000~4c560000,502800000~32500000,cb8500000~10f600000,1139800000~7a000000,1668d30000~3d290000,1794000000~3e800000,191c000000~3ee00000,1a49800000~4110000,1a4f800000~40f0000,1a70800000~4100000,1a76000000~3c10000]
= 0x582550000 : using 0x56d090000(22 GiB)
Any help here would be appreciated, I have stopped out CephFS file
system but our radosgw is also impacted.
[0] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.681.log
[1] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709.repair
Thanks,
derek
--
Derek T. Yarnell
Director of Computing Facilities
University of Maryland
Institute for Advanced Computer Studies
16 Mar
16 Mar
12:47 p.m.
Hi Derek,
first of all some BlueStore design overview to make sure we're on the
same plate.
BlueFS doesn't keep all the BlueStore data but just RocksDB part of it.
In your case BlueFS shares the same device with BlueStore user data.
Some space rebalance procedure takes periodically place to make sure
BlueFS has enough space to keep DB's data.
Hence there is a primary BlueStore space allocator which tracks the
whole volume space. And there is BlueFS one which is gifted by primary
allocator with some space depending on its needs.
Some observation on your case:
1) bluefs-bdev-sizes reports total device space and usage for BlueFS (!)
part of it. I.e. 22GiB are for BlueFS only, it provides no insight about
overall space usage.
2) Looks like Bluestore allocator complains about lack of free space.
Which means BlueFS plus user data took all the space. See:
-14> 2020-03-15 14:43:47.572 7f32925dd700 -1
bluestore(/var/lib/ceph/osd/ceph-681) _do_alloc_write failed to allocate
0x400000 allocated 0x 3ac000 min_alloc_size 0x4000 available 0x 0
-13> 2020-03-15 14:43:47.572 7f32925dd700 -1
bluestore(/var/lib/ceph/osd/ceph-681) _do_write _do_alloc_write failed
with (28) No space left on device
-12> 2020-03-15 14:43:47.572 7f32925dd700 -1
bluestore(/var/lib/ceph/osd/ceph-681) _txc_add_transaction error (28) No
space left on device not handled on operation 10 (op 4, counting from 0)
-11> 2020-03-15 14:43:47.572 7f32925dd700 -1
bluestore(/var/lib/ceph/osd/ceph-681) ENOSPC from bluestore,
misconfigured cluster
3) repair suffers from both lack of space for both allocators. BlueFS
one tries to acquire some additional space from the primary allocator
which fails to do that:
2020-03-15 23:55:14.816 7f0d3fac2c00 -1
bluestore(/var/lib/ceph/osd/ceph-709) allocate_bluefs_freespace failed
to allocate on 0xb000000 min_size 0xb000000 > allocated total 0x80000
bluefs_shared_alloc_size 0x10000 allocated 0x80000 available 0x 8000
2020-03-15 23:55:14.816 7f0d3fac2c00 -1 bluefs _allocate failed to
expand slow device to fit +0xaffa895
2020-03-15 23:55:14.816 7f0d3fac2c00 -1 bluefs _flush_range allocated:
0x0 offset: 0x0 length: 0xaffa895
OSD-709 has been already expanded, right?
What's the error reported by fsck?
4) OSD.681 has a number of checksum verification errors when reading DB
data:
2020-03-15 14:03:52.890 7f6311ffa700 3 rocksdb:
[table/block_based_table_reader.cc:1117] Encountered error while reading
data from compression dictionary block Corruption: block checksum
mismatch: expected 0, got 2324967102 in db/012948.sst offset
18446744073709551615 size 18446744073709551615
Can't say if this is bound to space shortage or not. Wondering if other
OSDs reported(-ing) something similar?
Thanks,
Igor
On 3/16/2020 7:15 AM, Derek Yarnell wrote:
Hi,
We have a production cluster that just suffered an issue with multiple
of our NVMe OSDs. Multiple of them died (>12) with errors that they no
longer had space with a 'ENOSPC from bluestore, misconfigured cluster'
error over 4 nodes. These are all simple one device bluestore osds.
ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus
(stable)
This is an example[0] of one of the logs. In this case each of 8 NVMe
OSDs on a node have 106GB of space allocated to each bluestore NVMe OSD.
The ceph-bluestore-tool bluefs-bdev-sizes output only lists 22GiB for
osd 681. I extended the space of bluestore on a few of the OSDs via LVM
and then the bluefs-bdev-expand command. This worked for a few and not
for others.
Some of the ones that it did work for recovered for a bit then
re-entered the error state. Trying to extend the allocation didn't work
after that. When they failed again I ran the fsck which reported that
it found 1 error and then running repair I got a rather long stack trace[1].
# ceph-bluestore-tool --log-level 30 --command bluefs-bdev-sizes --path
/var/lib/ceph/osd/ceph-681
inferring bluefs devices from bluestore path
slot 1 /var/lib/ceph/osd/ceph-681/block -> /dev/dm-33
1 : device size 0x1a80000000 : own
0x[2480000~10000,24a0000~10000,2520000~60000,25f0000~c0000,2720000~50000,28a0000~110000,2a20000~230000,2cc0000~260000,2f30000~220000,31c0000~6b0000,38a0000~10000,3990000~3e0000,3d80000~530000,42d0000~590000,48d0000~400000,4d00000~7d0000,54f0000~c50000,6150000~10000,6190000~150000,6350000~c0000,6480000~160000,6640000~1e0000,6870000~c0000,6a00000~30000,6a40000~240000,6dd0000~310000,7210000~b0000,73a0000~b0000,76a0000~180000,7830000~80000,78e0000~240000,7b70000~90000,7c50000~b0000,7ef0000~140000,8040000~30000,8180000~250000,8440000~50000,84b0000~110000,8610000~c0000,9e20000~20000,9e50000~b0000,9f10000~60000,9f80000~30000,dd80000~180000,df70000~6a0000,e620000~5ae0000,15200000~3510000,187f0000~bf0000,19490000~1070000,1ab70000~4c0000,1b400000~7d0000,1bbe0000~c20000,1cd10000~340000,1d3a0000~860000,1dd00000~2e00000,20c00000~3f00000,24d00000~700000,25600000~700000,26100000~200000,26400000~300000,26b00000~600000,27400000~400000,27ba0000~6e0000,28500000~1d00000,2a400000~700000,2ac00000~100000,2
b100000~300000,2b470000~120000,2b700000~500000,2c000000~200000,2c400000~400000,2ca00000~100000,2cf00000~300000,2d340000~39b0000,30d00000~1f00000,32e00000~4bf0000,380a0000~3c0000,38500000~c0000,38bd0000~400000,390b0000~340000,39400000~100000,39900000~1000000,3ac00000~5d00000,40b90000~400000,41280000~db50000,4ee00000~700000,4f900000~4500000,54390000~100000,54e00000~18400000,6d800000~20d0000,6f8f0000~1a10000,71400000~4500000,76100000~300000,766e0000~6860000,7dd00000~c00000,7eac0000~a0000,7ef90000~f190000,8e1f0000~80000,8e410000~60000,8e480000~20000,8e4b0000~20000,8e5c0000~50000,8e7e0000~50000,8f160000~60000,8f240000~a0000,90000000~15e90000,a6200000~c3a0000,b25d0000~630000,b3000000~c00000,b3ee0000~90000,b4200000~d00000,b5a70000~160000,b63f0000~2a0000,b6720000~2820000,bab00000~400000,bbf60000~10ad0000,ccb90000~2300000,cf000000~2b00000,d1ca0000~10000,d1e00000~1400000,d3230000~1df0000,d5200000~1a00000,d6d00000~800000,d75e0000~6f0000,d7f00000~d00000,d9100000~400000,d9900000~d00000,da800000~
600000,daf10000~400000,db700000~1600000,dd280000~20000,dd670000~390000,dda30000~400000,de190000~70000,de2a0000~370000,de660000~20000,de700000~14770000,f3600000~700000,f3db0000~960000,f49e0000~5b00000,fa600000~c00000,fb300000~510000,fbb00000~100000,fbeb0000~450000,fc400000~2b0000,fd400000~400000,fde00000~c00000,ff0b0000~50000,ff200000~800000,ffd60000~10000,fff00000~a0000,100200000~300000,101600000~100000,101750000~300000,102120000~1e0000,1027f0000~a00000,103600000~330000,103b00000~200000,103e60000~4a0000,104310000~c00000,105030000~1200000,106800000~100000,106b20000~400000,107000000~300000,1073e0000~400000,107950000~86b0000,110140000~d0000,110350000~2e0000,110e20000~20000,110eb0000~a0000,110f60000~60000,110fd0000~1f0000,1112a0000~f0000,111420000~30000,1115b0000~30000,111620000~150000,111790000~40000,112560000~180000,112730000~180000,1129b0000~50000,112f90000~4a0000,113840000~c0000,113ea0000~40000,113fb0000~130000,114100000~310000,114470000~10000,114620000~120000,114810000~120000,114a0
0000~20000,114a90000~f0000,114c60000~e0000,114e80000~20000,114f70000~140000,1150c0000~50000,1151f0000~320000,1155f0000~10000,115670000~226f0000,137e30000~800000,138b50000~400000,139400000~1500000,13ae00000~4500000,13f480000~400000,13f950000~1b6e0000,15b700000~400000,15c000000~300000,15c600000~700000,15d9e0000~1820000,15f400000~c00000,160400000~d00000,1613a0000~630000,1619e0000~d20000,162800000~1b00000,164600000~7550000,170d00000~1800000,172580000~100000,172d70000~1c190000,18f100000~40e0000,193700000~400000,193c90000~6970000,19aa90000~188d0000,1b3770000~c220000,1bff00000~1200000,1c12f0000~400000,1c2c00000~400000,1c3f80000~400000,1c5300000~4200000,1c95c0000~b9a40000,283800000~1800000,289000000~3800000,28d000000~6000000,293800000~2000000,296000000~e000000,2a4800000~465d0000,4b6100000~4c560000,502800000~32500000,cb8500000~10f600000,1139800000~7a000000,1668d30000~3d290000,1794000000~3e800000,191c000000~3ee00000,1a49800000~4110000,1a4f800000~40f0000,1a70800000~4100000,1a76000000~3c10000]
= 0x582550000 : using 0x56d090000(22 GiB)
Any help here would be appreciated, I have stopped out CephFS file
system but our radosgw is also impacted.
[0] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.681.log
[1] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709.repair
Thanks,
derek
1:26 p.m.
Hi Igor,
Thank you for the help.
On 3/16/20 7:47 AM, Igor Fedotov wrote:
OSD-709 has been already expanded, right?
Correct with 'ceph-bluestore-tool --log-level 30 --path
/var/lib/ceph/osd/ceph-709 --command bluefs-bdev-expand'. Does this
expand bluefs and the data allocation?
Is there a way to ask how much data is used by the non-bluefs portion?
[root@obj21 ceph-709]# pwd
/var/lib/ceph/osd/ceph-709
[root@obj21 ceph-709]# ls -la
total 28
drwxrwxrwt. 2 ceph ceph 180 Mar 15 22:21 .
drwxr-x---. 52 ceph ceph 4096 Jan 8 14:44 ..
lrwxrwxrwx. 1 ceph ceph 30 Mar 15 22:21 block ->
/dev/ceph-data-vol20/data-20-0
-rw-------. 1 ceph ceph 37 Mar 15 22:21 ceph_fsid
-rw-------. 1 ceph ceph 37 Mar 15 22:21 fsid
-rw-------. 1 ceph ceph 57 Mar 15 22:21 keyring
-rw-------. 1 ceph ceph 6 Mar 15 22:21 ready
-rw-------. 1 ceph ceph 10 Mar 15 22:21 type
-rw-------. 1 ceph ceph 4 Mar 15 22:21 whoami
[root@obj21 ceph-709]# lvdisplay /dev/ceph-data-vol20/data-20-0
--- Logical volume ---
LV Path /dev/ceph-data-vol20/data-20-0
LV Name data-20-0
VG Name ceph-data-vol20
LV UUID idjDxJ-CbzN-fb1n-Bhzx-86vP-mSAD-HvIq5n
LV Write Access read/write
LV Creation host, time obj21.umiacs.umd.edu, 2018-10-10 08:15:20 -0400
LV Status available
# open 0
LV Size 200.00 GiB
Current LE 51200
Segments 4
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:33
What's the error reported by fsck?
It doesn't say, but when I try to run the deep fsck it produces
[root@obj21 ceph-709]# ceph-bluestore-tool --log-level 30 --path
/var/lib/ceph/osd/ceph-709 --command fsck
2020-03-16 08:02:16.590 7f5faaa11c00 -1
bluestore(/var/lib/ceph/osd/ceph-709) fsck error: bluefs_extents
inconsistency, downgrade to previous releases might be broken.
fsck found 1 error(s)
[0] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709-fsck-deep
4) OSD.681 has a number of checksum verification
errors when reading DB
data:
2020-03-15 14:03:52.890 7f6311ffa700 3 rocksdb:
[table/block_based_table_reader.cc:1117] Encountered error while reading
data from compression dictionary block Corruption: block checksum
mismatch: expected 0, got 2324967102 in db/012948.sst offset
18446744073709551615 size 18446744073709551615
Can't say if this is bound to space shortage or not. Wondering if other
OSDs reported(-ing) something similar?
Here is another node which at around '2020-03-15 13:51' starts looking
like peering a few pgs and then at '2020-03-15 14:40' on 716 fails, and
then for example 719 it fails 1 min later at '2020-03-15 14:41'.
[1] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.716.log-20200316.gz
[2] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.717.log-20200316.gz
[3] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.718.log-20200316.gz
[4] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.719.log-20200316.gz
[5] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.720.log-20200316.gz
[6] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.721.log-20200316.gz
[7] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.722.log-20200316.gz
[8] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.722.log-20200316.gz
Thanks,
derek
--
Derek T. Yarnell
Director of Computing Facilities
University of Maryland
Institute for Advanced Computer Studies
3:34 p.m.
Derek,
On 3/16/2020 3:26 PM, Derek Yarnell wrote:
So looks like checksum errors appeared after the initial failure. And
trigger recovery which requires additional space...
I think the summary of the issue is as follows:
Cluster had been in 'near full' state when some OSDs started to crash
due to lack of free space.
An attempt to extend device can succeed or not depending on the state
RocksDB is in when the first crash happened.
a) If DB is corrupted and needs for recovery (which is triggered on each
non read-only DB open) it asks for more space which fails again and OSD
fall into a "deadlock" state:
to extend main device one needs DB access which in turn needs more space.
b) If DB isn't corrupted expansion succeeds and OSD starts to get more
data due to peering. Which finally fills it up and OSD tend to get into a).
Some OSDs will presumably allow another expansion though.
Unfortunately I don't know any fix/workaround for the "deadlock" case at
the moment.
Probably migrating DB to a standalone volume (using
ceph-bluestore-tool's bluefs-bdev-migrate commands) will help but I need
to double check that.
And it will definitely expose data to risk of loss so please hold on
until my additional recommendations.
Most probably you will need additional 30GB of free space per each OSD
if going this way. So please let me know if you can afford this.
> Thanks,
> derek
>
Hi Igor,
Thank you for the help.
On 3/16/20 7:47 AM, Igor Fedotov wrote:
I can suggest the following non-straightforward way for now:
1) Check osd startup log for the following line:
2020-03-15 14:43:27.845 7f41bb6baa80 1
bluestore(/var/lib/ceph/osd/ceph-681) _open_alloc loaded 23 GiB in 97
extents
Note 23GiB loaded.
2) Then retriever bluefs usage space for main device from
"bluefs-bdev-sizes" output:
1 : device size 0x1a80000000 : own
...
= 0x582550000 : using 0x56d090000(22 GiB)
3) Actual available space would be around: 1 GiB = 23GiB - 22 GiB
OSD-709 has been already expanded, right?
Correct with 'ceph-bluestore-tool --log-level 30 --path
/var/lib/ceph/osd/ceph-709 --command bluefs-bdev-expand'. Does this
expand bluefs and the data allocation?
Is there a way to ask how much data is used by the non-bluefs portion?
[root@obj21 ceph-709]# pwd
/var/lib/ceph/osd/ceph-709
[root@obj21 ceph-709]# ls -la
total 28
drwxrwxrwt. 2 ceph ceph 180 Mar 15 22:21 .
drwxr-x---. 52 ceph ceph 4096 Jan 8 14:44 ..
lrwxrwxrwx. 1 ceph ceph 30 Mar 15 22:21 block ->
/dev/ceph-data-vol20/data-20-0
-rw-------. 1 ceph ceph 37 Mar 15 22:21 ceph_fsid
-rw-------. 1 ceph ceph 37 Mar 15 22:21 fsid
-rw-------. 1 ceph ceph 57 Mar 15 22:21 keyring
-rw-------. 1 ceph ceph 6 Mar 15 22:21 ready
-rw-------. 1 ceph ceph 10 Mar 15 22:21 type
-rw-------. 1 ceph ceph 4 Mar 15 22:21 whoami
[root@obj21 ceph-709]# lvdisplay /dev/ceph-data-vol20/data-20-0
--- Logical volume ---
LV Path /dev/ceph-data-vol20/data-20-0
LV Name data-20-0
VG Name ceph-data-vol20
LV UUID idjDxJ-CbzN-fb1n-Bhzx-86vP-mSAD-HvIq5n
LV Write Access read/write
LV Creation host, time obj21.umiacs.umd.edu, 2018-10-10 08:15:20 -0400
LV Status available
# open 0
LV Size 200.00 GiB
Current LE 51200
Segments 4
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:33
fsck-deep suffers from the same lack of space. Could you please collect
log for regular fsck?
The latter opens DB in read-only and hence doesn't need additional space.
What's the error reported by fsck?
It
doesn't say, but when I try to run the deep fsck it produces
[root@obj21 ceph-709]# ceph-bluestore-tool --log-level 30 --path
/var/lib/ceph/osd/ceph-709 --command fsck
2020-03-16 08:02:16.590 7f5faaa11c00 -1
bluestore(/var/lib/ceph/osd/ceph-709) fsck error: bluefs_extents
inconsistency, downgrade to previous releases might be broken.
fsck found 1 error(s)
[0] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709-fsck-deep 4) OSD.681 has
a number of checksum verification errors when reading DB
data:
2020-03-15 14:03:52.890 7f6311ffa700 3 rocksdb:
[table/block_based_table_reader.cc:1117] Encountered error while reading
data from compression dictionary block Corruption: block checksum
mismatch: expected 0, got 2324967102 in db/012948.sst offset
18446744073709551615 size 18446744073709551615
Can't say if this is bound to space shortage or not. Wondering if other
OSDs reported(-ing) something similar?
Here is another node which at around
'2020-03-15 13:51' starts looking
like peering a few pgs and then at '2020-03-15 14:40' on 716 fails, and
then for example 719 it fails 1 min later at '2020-03-15 14:41'.
[1] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.716.log-20200316.gz
[2] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.717.log-20200316.gz
[3] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.718.log-20200316.gz
[4] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.719.log-20200316.gz
[5] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.720.log-20200316.gz
[6] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.721.log-20200316.gz
[7] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.722.log-20200316.gz
[8] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.722.log-20200316.gz
5:17 p.m.
Hi Igor,
On 3/16/20 10:34 AM, Igor Fedotov wrote:
That is the log for the regular fsck it only says that it founds 1
error(s) and doesn't give any further information (even with the
--log-level 30 specified).
I can suggest the following non-straightforward way
for now:
1) Check osd startup log for the following line:
2020-03-15 14:43:27.845 7f41bb6baa80 1
bluestore(/var/lib/ceph/osd/ceph-681) _open_alloc loaded 23 GiB in 97
extents
Note 23GiB loaded.
2) Then retriever bluefs usage space for main device from
"bluefs-bdev-sizes" output:
1 : device size 0x1a80000000 : own
...
= 0x582550000 : using 0x56d090000(22 GiB)
3) Actual available space would be around: 1 GiB = 23GiB - 22 GiB
Ok so essentially the data is then the delta of what it reports in the
following line? Why the descrepency from the _open_alloc loaded
information and the bluefs-bdev-sizes output?
bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-681/block
size 106 GiB
[root@obj21
ceph-709]# ceph-bluestore-tool --log-level 30 --path
/var/lib/ceph/osd/ceph-709 --command fsck
2020-03-16 08:02:16.590 7f5faaa11c00 -1
bluestore(/var/lib/ceph/osd/ceph-709) fsck error: bluefs_extents
inconsistency, downgrade to previous releases might be broken.
fsck found 1 error(s)
[0] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709-fsck-deep
fsck-deep suffers from the same lack of space. Could you please collect
log for regular fsck? So looks like checksum errors appeared after the
initial failure. And
trigger recovery which requires additional space...
I think the summary of the issue is as follows:
Cluster had been in 'near full' state when some OSDs started to crash
due to lack of free space.
An attempt to extend device can succeed or not depending on the state
RocksDB is in when the first crash happened.
a) If DB is corrupted and needs for recovery (which is triggered on each
non read-only DB open) it asks for more space which fails again and OSD
fall into a "deadlock" state:
to extend main device one needs DB access which in turn needs more space.
b) If DB isn't corrupted expansion succeeds and OSD starts to get more
data due to peering. Which finally fills it up and OSD tend to get into a).
Some OSDs will presumably allow another expansion though.
Unfortunately I don't know any fix/workaround for the "deadlock" case at
the moment.
I am trying to find creative ways to allow to increase the space
significantly on the OSD but not strand it so I can continue to provide
new space to more of the OSDs. LVM is helpful here
Probably migrating DB to a standalone volume (using
ceph-bluestore-tool's bluefs-bdev-migrate commands) will help but I need
to double check that.
And it will definitely expose data to risk of loss so please hold on
until my additional recommendations.
Most probably you will need additional 30GB of free space per each OSD
if going this way. So please let me know if you can afford this.
Well I had already increased 709's initial space from 106GB to 200GB and
now I gave it 10GB more but it still can not actually resize. Here is
the relevant information I think but the full logs is here[0]. I then
did it with 30G (now total of 240G) and it still failed[1]. I am out of
space without some additional hardware in this node though I have an
idea. If I knew what size it is (and what space it needs for recovery
this would be very helpful).
# ceph-bluestore-tool --log-level 30 --path /var/lib/ceph/osd/ceph-709
--command bluefs-bdev-expand
-4> 2020-03-16 11:33:34.181 7f41d5940c00 -1
bluestore(/var/lib/ceph/osd/ceph-709) allocate_bluefs_freespace failed
to allocate on 0xb000000 min_size 0xb000000 > allocated total 0x80000
bluefs_shared_alloc_size 0x10000 allocated 0x80000 available 0x 8000
-3> 2020-03-16 11:33:34.181 7f41d5940c00 -1 bluefs _allocate failed
to expand slow device to fit +0xaffa895
-2> 2020-03-16 11:33:34.181 7f41d5940c00 -1 bluefs _flush_range
allocated: 0x0 offset: 0x0 length: 0xaffa895
-1> 2020-03-16 11:33:34.184 7f41d5940c00 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.6/rpm/el7/BUILD/ceph-14.2.6/src/os/bluestore/BlueFS.cc:
In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t,
uint64_t)' thread 7f41d5940c00 time 2020-03-16 11:33:34.181884
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.6/rpm/el7/BUILD/ceph-14.2.6/src/os/bluestore/BlueFS.cc:
2269: ceph_abort_msg("bluefs enospc")
[0] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709.bluefs-bdev-expand
[1] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709.bluefs-bdev-expand-2
> Thanks,
> derek
>
--
Derek T. Yarnell
Director of Computing Facilities
University of Maryland
Institute for Advanced Computer Studies
18 Mar
18 Mar
10:30 a.m.
Hi Derek,
On 3/16/2020 7:17 PM, Derek Yarnell wrote:
There is no much sense in increasing main device for this specific OSD
(and similarly failing one, i.e. OSDs mentioning RocksDB recovery in
back trace) at this point.
It's in the "deadlock" state I mentioned before. And hence expand is
unable to proceed.
I'm checking some workarounds to get out of this state at the moment.
Still in progress though.
What I meant before is that you need more available space if workaround
would be the assignment of a new standalone DB volume. It's a
questionable way so I'm trying other ways for now.
> # ceph-bluestore-tool --log-level 30 --path /var/lib/ceph/osd/ceph-709
> --command bluefs-bdev-expand
>
> -4> 2020-03-16 11:33:34.181 7f41d5940c00 -1
> bluestore(/var/lib/ceph/osd/ceph-709) allocate_bluefs_freespace failed
> to allocate on 0xb000000 min_size 0xb000000 > allocated total 0x80000
> bluefs_shared_alloc_size 0x10000 allocated 0x80000 available 0x 8000
> -3> 2020-03-16 11:33:34.181 7f41d5940c00 -1 bluefs _allocate failed
> to expand slow device to fit +0xaffa895
> -2> 2020-03-16 11:33:34.181 7f41d5940c00 -1 bluefs _flush_range
> allocated: 0x0 offset: 0x0 length: 0xaffa895
> -1> 2020-03-16 11:33:34.184 7f41d5940c00 -1
>
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.6/rpm/el7/BUILD/ceph-14.2.6/src/os/bluestore/BlueFS.cc:
> In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t,
> uint64_t)' thread 7f41d5940c00 time 2020-03-16 11:33:34.181884
>
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.6/rpm/el7/BUILD/ceph-14.2.6/src/os/bluestore/BlueFS.cc:
> 2269: ceph_abort_msg("bluefs enospc")
>
>
> [0] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709.bluefs-bdev-expand
> [1] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709.bluefs-bdev-expand-2
>
>>> Thanks,
>>> derek
>>>
Hi Igor,
On 3/16/20 10:34 AM, Igor Fedotov wrote:
Probably - due to value rounding.
I can suggest the following non-straightforward
way for now:
1) Check osd startup log for the following line:
2020-03-15 14:43:27.845 7f41bb6baa80 1
bluestore(/var/lib/ceph/osd/ceph-681) _open_alloc loaded 23 GiB in 97
extents
Note 23GiB loaded.
2) Then retriever bluefs usage space for main device from
"bluefs-bdev-sizes" output:
1 : device size 0x1a80000000 : own
...
= 0x582550000 : using 0x56d090000(22 GiB)
3) Actual available space would be around: 1 GiB = 23GiB - 22 GiB
Ok so
essentially the data is then the delta of what it reports in the
following line? Why the descrepency from the _open_alloc loaded
information and the bluefs-bdev-sizes output?
bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-681/block
size 106 GiB
That is the log for the regular fsck it only says that it
founds 1
error(s) and doesn't give any further information (even with the
--log-level 30 specified).
Missed attachment?
[root@obj21 ceph-709]# ceph-bluestore-tool --log-level 30 --path
/var/lib/ceph/osd/ceph-709 --command fsck
2020-03-16 08:02:16.590 7f5faaa11c00 -1
bluestore(/var/lib/ceph/osd/ceph-709) fsck error: bluefs_extents
inconsistency, downgrade to previous releases might be broken.
fsck found 1 error(s)
[0] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709-fsck-deep
fsck-deep suffers
from the same lack of space. Could you please collect
log for regular fsck? So looks like
checksum errors appeared after the initial failure. And
trigger recovery which requires additional space...
I think the summary of the issue is as follows:
Cluster had been in 'near full' state when some OSDs started to crash
due to lack of free space.
An attempt to extend device can succeed or not depending on the state
RocksDB is in when the first crash happened.
a) If DB is corrupted and needs for recovery (which is triggered on each
non read-only DB open) it asks for more space which fails again and OSD
fall into a "deadlock" state:
to extend main device one needs DB access which in turn needs more space.
b) If DB isn't corrupted expansion succeeds and OSD starts to get more
data due to peering. Which finally fills it up and OSD tend to get into a).
Some OSDs will presumably allow another expansion though.
Unfortunately I don't know any fix/workaround for the "deadlock" case at
the moment.
I am trying to find creative ways to allow to increase the space
significantly on the OSD but not strand it so I can continue to provide
new space to more of the OSDs. LVM is helpful here
Probably migrating DB to a standalone volume
(using
ceph-bluestore-tool's bluefs-bdev-migrate commands) will help but I need
to double check that.
And it will definitely expose data to risk of loss so please hold on
until my additional recommendations.
Most probably you will need additional 30GB of free space per each OSD
if going this way. So please let me know if you can afford this.
Well I had
already increased 709's initial space from 106GB to 200GB and
now I gave it 10GB more but it still can not actually resize. Here is
the relevant information I think but the full logs is here[0]. I then
did it with 30G (now total of 240G) and it still failed[1]. I am out of
space without some additional hardware in this node though I have an
idea. If I knew what size it is (and what space it needs for recovery
this would be very helpful).
1:31 p.m.
Hi Igor,
I just want to thank you for taking the time to help with this issue.
On 3/18/20 5:30 AM, Igor Fedotov wrote:
There is no much sense in increasing main device for this specific OSD
(and similarly failing one, i.e. OSDs mentioning RocksDB recovery in
back trace) at this point.
It's in the "deadlock" state I mentioned before. And hence expand is
unable to proceed.
I'm checking some workarounds to get out of this state at the moment.
Still in progress though.
What I meant before is that you need more available space if workaround
would be the assignment of a new standalone DB volume. It's a
questionable way so I'm trying other ways for now.
I had to go forward with this and migrate the db to a separate partition
as we were production impacting. I did some testing by making a copy
(dd'ing to new LVM) of one of the OSDs and performing the steps before I
tried this. Essential steps (understanding this is a Nautilus cluster)
for someone who might come across this in the future (for each OSD you
need to do this for):
1) create new partition to hold the db volume
lvcreate -L30G -n db-20-6 /dev/ceph-db-vol04
2) migrate with the ceph-bluestore-tool
ceph-bluestore-tool bluefs-bdev-migrate --path
/var/lib/ceph/osd/ceph-715 --dev-target /dev/ceph-db-vol04/db-20-6
--devs-source /var/lib/ceph/osd/ceph-715/block
3) make sure the block device are owned by the ceph user (
chown -h ceph:ceph /var/lib/ceph/osd/ceph-715/block.db
chown ceph:ceph /dev/ceph-db-vol04/db-20-6
4) run the ceph-bluestore-tool repair
ceph-bluestore-tool --log-level 30 --path /var/lib/ceph/osd/ceph-715
--command repair
5) test compaction
ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-715 compact
6) start the OSD
systemctl start ceph-osd@715
Lesson is that bluestore does not handle well when you hit capacity like
this with the simple configuration of co-located rocksdb and data. I
think that even for fast disks such as nvme you should always create a
separate db partition as this deadlock scenario is very problematic if
you don't have additional storage (or can add some quickly).
It seems that we had about 2million cephfs log segments that were behind
on trimming. I am not sure where these segments are kept but I am
guessing in the mds metadata pool which seems to have driven this
maximum space issue. As we are now down to about %17 used in the nvme
class when we were at 100% during this issue.
Thanks,
derek
--
Derek T. Yarnell
Director of Computing Facilities
University of Maryland
Institute for Advanced Computer Studies
Most probably you will need additional 30GB of free
space per each OSD
if going this way. So please let me know if you can afford this.
Well I had
already increased 709's initial space from 106GB to 200GB and
now I gave it 10GB more but it still can not actually resize. Here is
the relevant information I think but the full logs is here[0]. I then
did it with 30G (now total of 240G) and it still failed[1]. I am out of
space without some additional hardware in this node though I have an
idea. If I knew what size it is (and what space it needs for recovery
this would be very helpful).
1494
days inactive
1496
days old
6 comments
2 participants
participants (2)
-
Derek Yarnell -
Igor Fedotov