Hi, We have a production cluster that just suffered an issue with multiple of our NVMe OSDs. Multiple of them died (>12) with errors that they no longer had space with a 'ENOSPC from bluestore, misconfigured cluster' error over 4 nodes. These are all simple one device bluestore osds. ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) This is an example[0] of one of the logs. In this case each of 8 NVMe OSDs on a node have 106GB of space allocated to each bluestore NVMe OSD. The ceph-bluestore-tool bluefs-bdev-sizes output only lists 22GiB for osd 681. I extended the space of bluestore on a few of the OSDs via LVM and then the bluefs-bdev-expand command. This worked for a few and not for others. Some of the ones that it did work for recovered for a bit then re-entered the error state. Trying to extend the allocation didn't work after that. When they failed again I ran the fsck which reported that it found 1 error and then running repair I got a rather long stack trace[1]. # ceph-bluestore-tool --log-level 30 --command bluefs-bdev-sizes --path /var/lib/ceph/osd/ceph-681 inferring bluefs devices from bluestore path slot 1 /var/lib/ceph/osd/ceph-681/block -> /dev/dm-33 1 : device size 0x1a80000000 : own 0x[2480000~10000,24a0000~10000,2520000~60000,25f0000~c0000,2720000~50000,28a0000~110000,2a20000~230000,2cc0000~260000,2f30000~220000,31c0000~6b0000,38a0000~10000,3990000~3e0000,3d80000~530000,42d0000~590000,48d0000~400000,4d00000~7d0000,54f0000~c50000,6150000~10000,6190000~150000,6350000~c0000,6480000~160000,6640000~1e0000,6870000~c0000,6a00000~30000,6a40000~240000,6dd0000~310000,7210000~b0000,73a0000~b0000,76a0000~180000,7830000~80000,78e0000~240000,7b70000~90000,7c50000~b0000,7ef0000~140000,8040000~30000,8180000~250000,8440000~50000,84b0000~110000,8610000~c0000,9e20000~20000,9e50000~b0000,9f10000~60000,9f80000~30000,dd80000~180000,df70000~6a0000,e620000~5ae0000,15200000~3510000,187f0000~bf0000,19490000~1070000,1ab70000~4c0000,1b400000~7d0000,1bbe0000~c20000,1cd10000~340000,1d3a0000~860000,1dd00000~2e00000,20c00000~3f00000,24d00000~700000,25600000~700000,26100000~200000,26400000~300000,26b00000~600000,27400000~400000,27ba0000~6e0000,28500000~1d00000,2a400000~700000,2ac00000~100000,2b100000~300000,2b470000~120000,2b700000~500000,2c000000~200000,2c400000~400000,2ca00000~100000,2cf00000~300000,2d340000~39b0000,30d00000~1f00000,32e00000~4bf0000,380a0000~3c0000,38500000~c0000,38bd0000~400000,390b0000~340000,39400000~100000,39900000~1000000,3ac00000~5d00000,40b90000~400000,41280000~db50000,4ee00000~700000,4f900000~4500000,54390000~100000,54e00000~18400000,6d800000~20d0000,6f8f0000~1a10000,71400000~4500000,76100000~300000,766e0000~6860000,7dd00000~c00000,7eac0000~a0000,7ef90000~f190000,8e1f0000~80000,8e410000~60000,8e480000~20000,8e4b0000~20000,8e5c0000~50000,8e7e0000~50000,8f160000~60000,8f240000~a0000,90000000~15e90000,a6200000~c3a0000,b25d0000~630000,b3000000~c00000,b3ee0000~90000,b4200000~d00000,b5a70000~160000,b63f0000~2a0000,b6720000~2820000,bab00000~400000,bbf60000~10ad0000,ccb90000~2300000,cf000000~2b00000,d1ca0000~10000,d1e00000~1400000,d3230000~1df0000,d5200000~1a00000,d6d00000~800000,d75e0000~6f0000,d7f00000~d00000,d9100000~400000,d9900000~d00000,da800000~600000,daf10000~400000,db700000~1600000,dd280000~20000,dd670000~390000,dda30000~400000,de190000~70000,de2a0000~370000,de660000~20000,de700000~14770000,f3600000~700000,f3db0000~960000,f49e0000~5b00000,fa600000~c00000,fb300000~510000,fbb00000~100000,fbeb0000~450000,fc400000~2b0000,fd400000~400000,fde00000~c00000,ff0b0000~50000,ff200000~800000,ffd60000~10000,fff00000~a0000,100200000~300000,101600000~100000,101750000~300000,102120000~1e0000,1027f0000~a00000,103600000~330000,103b00000~200000,103e60000~4a0000,104310000~c00000,105030000~1200000,106800000~100000,106b20000~400000,107000000~300000,1073e0000~400000,107950000~86b0000,110140000~d0000,110350000~2e0000,110e20000~20000,110eb0000~a0000,110f60000~60000,110fd0000~1f0000,1112a0000~f0000,111420000~30000,1115b0000~30000,111620000~150000,111790000~40000,112560000~180000,112730000~180000,1129b0000~50000,112f90000~4a0000,113840000~c0000,113ea0000~40000,113fb0000~130000,114100000~310000,114470000~10000,114620000~120000,114810000~120000,114a00000~20000,114a90000~f0000,114c60000~e0000,114e80000~20000,114f70000~140000,1150c0000~50000,1151f0000~320000,1155f0000~10000,115670000~226f0000,137e30000~800000,138b50000~400000,139400000~1500000,13ae00000~4500000,13f480000~400000,13f950000~1b6e0000,15b700000~400000,15c000000~300000,15c600000~700000,15d9e0000~1820000,15f400000~c00000,160400000~d00000,1613a0000~630000,1619e0000~d20000,162800000~1b00000,164600000~7550000,170d00000~1800000,172580000~100000,172d70000~1c190000,18f100000~40e0000,193700000~400000,193c90000~6970000,19aa90000~188d0000,1b3770000~c220000,1bff00000~1200000,1c12f0000~400000,1c2c00000~400000,1c3f80000~400000,1c5300000~4200000,1c95c0000~b9a40000,283800000~1800000,289000000~3800000,28d000000~6000000,293800000~2000000,296000000~e000000,2a4800000~465d0000,4b6100000~4c560000,502800000~32500000,cb8500000~10f600000,1139800000~7a000000,1668d30000~3d290000,1794000000~3e800000,191c000000~3ee00000,1a49800000~4110000,1a4f800000~40f0000,1a70800000~4100000,1a76000000~3c10000] = 0x582550000 : using 0x56d090000(22 GiB) Any help here would be appreciated, I have stopped out CephFS file system but our radosgw is also impacted. [0] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.681.log [1] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709.repair Thanks, derek -- Derek T. Yarnell Director of Computing Facilities University of Maryland Institute for Advanced Computer Studies
Hi Derek, first of all some BlueStore design overview to make sure we're on the same plate. BlueFS doesn't keep all the BlueStore data but just RocksDB part of it. In your case BlueFS shares the same device with BlueStore user data. Some space rebalance procedure takes periodically place to make sure BlueFS has enough space to keep DB's data. Hence there is a primary BlueStore space allocator which tracks the whole volume space. And there is BlueFS one which is gifted by primary allocator with some space depending on its needs. Some observation on your case: 1) bluefs-bdev-sizes reports total device space and usage for BlueFS (!) part of it. I.e. 22GiB are for BlueFS only, it provides no insight about overall space usage. 2) Looks like Bluestore allocator complains about lack of free space. Which means BlueFS plus user data took all the space. See: -14> 2020-03-15 14:43:47.572 7f32925dd700 -1 bluestore(/var/lib/ceph/osd/ceph-681) _do_alloc_write failed to allocate 0x400000 allocated 0x 3ac000 min_alloc_size 0x4000 available 0x 0 -13> 2020-03-15 14:43:47.572 7f32925dd700 -1 bluestore(/var/lib/ceph/osd/ceph-681) _do_write _do_alloc_write failed with (28) No space left on device -12> 2020-03-15 14:43:47.572 7f32925dd700 -1 bluestore(/var/lib/ceph/osd/ceph-681) _txc_add_transaction error (28) No space left on device not handled on operation 10 (op 4, counting from 0) -11> 2020-03-15 14:43:47.572 7f32925dd700 -1 bluestore(/var/lib/ceph/osd/ceph-681) ENOSPC from bluestore, misconfigured cluster 3) repair suffers from both lack of space for both allocators. BlueFS one tries to acquire some additional space from the primary allocator which fails to do that: 2020-03-15 23:55:14.816 7f0d3fac2c00 -1 bluestore(/var/lib/ceph/osd/ceph-709) allocate_bluefs_freespace failed to allocate on 0xb000000 min_size 0xb000000 > allocated total 0x80000 bluefs_shared_alloc_size 0x10000 allocated 0x80000 available 0x 8000 2020-03-15 23:55:14.816 7f0d3fac2c00 -1 bluefs _allocate failed to expand slow device to fit +0xaffa895 2020-03-15 23:55:14.816 7f0d3fac2c00 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0xaffa895 OSD-709 has been already expanded, right? What's the error reported by fsck? 4) OSD.681 has a number of checksum verification errors when reading DB data: 2020-03-15 14:03:52.890 7f6311ffa700 3 rocksdb: [table/block_based_table_reader.cc:1117] Encountered error while reading data from compression dictionary block Corruption: block checksum mismatch: expected 0, got 2324967102 in db/012948.sst offset 18446744073709551615 size 18446744073709551615 Can't say if this is bound to space shortage or not. Wondering if other OSDs reported(-ing) something similar? Thanks, Igor On 3/16/2020 7:15 AM, Derek Yarnell wrote:
Hi,
We have a production cluster that just suffered an issue with multiple of our NVMe OSDs. Multiple of them died (>12) with errors that they no longer had space with a 'ENOSPC from bluestore, misconfigured cluster' error over 4 nodes. These are all simple one device bluestore osds.
ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)
This is an example[0] of one of the logs. In this case each of 8 NVMe OSDs on a node have 106GB of space allocated to each bluestore NVMe OSD. The ceph-bluestore-tool bluefs-bdev-sizes output only lists 22GiB for osd 681. I extended the space of bluestore on a few of the OSDs via LVM and then the bluefs-bdev-expand command. This worked for a few and not for others.
Some of the ones that it did work for recovered for a bit then re-entered the error state. Trying to extend the allocation didn't work after that. When they failed again I ran the fsck which reported that it found 1 error and then running repair I got a rather long stack trace[1].
# ceph-bluestore-tool --log-level 30 --command bluefs-bdev-sizes --path /var/lib/ceph/osd/ceph-681 inferring bluefs devices from bluestore path slot 1 /var/lib/ceph/osd/ceph-681/block -> /dev/dm-33 1 : device size 0x1a80000000 : own 0x[2480000~10000,24a0000~10000,2520000~60000,25f0000~c0000,2720000~50000,28a0000~110000,2a20000~230000,2cc0000~260000,2f30000~220000,31c0000~6b0000,38a0000~10000,3990000~3e0000,3d80000~530000,42d0000~590000,48d0000~400000,4d00000~7d0000,54f0000~c50000,6150000~10000,6190000~150000,6350000~c0000,6480000~160000,6640000~1e0000,6870000~c0000,6a00000~30000,6a40000~240000,6dd0000~310000,7210000~b0000,73a0000~b0000,76a0000~180000,7830000~80000,78e0000~240000,7b70000~90000,7c50000~b0000,7ef0000~140000,8040000~30000,8180000~250000,8440000~50000,84b0000~110000,8610000~c0000,9e20000~20000,9e50000~b0000,9f10000~60000,9f80000~30000,dd80000~180000,df70000~6a0000,e620000~5ae0000,15200000~3510000,187f0000~bf0000,19490000~1070000,1ab70000~4c0000,1b400000~7d0000,1bbe0000~c20000,1cd10000~340000,1d3a0000~860000,1dd00000~2e00000,20c00000~3f00000,24d00000~700000,25600000~700000,26100000~200000,26400000~300000,26b00000~600000,27400000~400000,27ba0000~6e0000,28500000~1d00000,2a400000~700000,2ac00000~100000,2 b100000~300000,2b470000~120000,2b700000~500000,2c000000~200000,2c400000~400000,2ca00000~100000,2cf00000~300000,2d340000~39b0000,30d00000~1f00000,32e00000~4bf0000,380a0000~3c0000,38500000~c0000,38bd0000~400000,390b0000~340000,39400000~100000,39900000~1000000,3ac00000~5d00000,40b90000~400000,41280000~db50000,4ee00000~700000,4f900000~4500000,54390000~100000,54e00000~18400000,6d800000~20d0000,6f8f0000~1a10000,71400000~4500000,76100000~300000,766e0000~6860000,7dd00000~c00000,7eac0000~a0000,7ef90000~f190000,8e1f0000~80000,8e410000~60000,8e480000~20000,8e4b0000~20000,8e5c0000~50000,8e7e0000~50000,8f160000~60000,8f240000~a0000,90000000~15e90000,a6200000~c3a0000,b25d0000~630000,b3000000~c00000,b3ee0000~90000,b4200000~d00000,b5a70000~160000,b63f0000~2a0000,b6720000~2820000,bab00000~400000,bbf60000~10ad0000,ccb90000~2300000,cf000000~2b00000,d1ca0000~10000,d1e00000~1400000,d3230000~1df0000,d5200000~1a00000,d6d00000~800000,d75e0000~6f0000,d7f00000~d00000,d9100000~400000,d9900000~d00000,da800000~ 600000,daf10000~400000,db700000~1600000,dd280000~20000,dd670000~390000,dda30000~400000,de190000~70000,de2a0000~370000,de660000~20000,de700000~14770000,f3600000~700000,f3db0000~960000,f49e0000~5b00000,fa600000~c00000,fb300000~510000,fbb00000~100000,fbeb0000~450000,fc400000~2b0000,fd400000~400000,fde00000~c00000,ff0b0000~50000,ff200000~800000,ffd60000~10000,fff00000~a0000,100200000~300000,101600000~100000,101750000~300000,102120000~1e0000,1027f0000~a00000,103600000~330000,103b00000~200000,103e60000~4a0000,104310000~c00000,105030000~1200000,106800000~100000,106b20000~400000,107000000~300000,1073e0000~400000,107950000~86b0000,110140000~d0000,110350000~2e0000,110e20000~20000,110eb0000~a0000,110f60000~60000,110fd0000~1f0000,1112a0000~f0000,111420000~30000,1115b0000~30000,111620000~150000,111790000~40000,112560000~180000,112730000~180000,1129b0000~50000,112f90000~4a0000,113840000~c0000,113ea0000~40000,113fb0000~130000,114100000~310000,114470000~10000,114620000~120000,114810000~120000,114a0 0000~20000,114a90000~f0000,114c60000~e0000,114e80000~20000,114f70000~140000,1150c0000~50000,1151f0000~320000,1155f0000~10000,115670000~226f0000,137e30000~800000,138b50000~400000,139400000~1500000,13ae00000~4500000,13f480000~400000,13f950000~1b6e0000,15b700000~400000,15c000000~300000,15c600000~700000,15d9e0000~1820000,15f400000~c00000,160400000~d00000,1613a0000~630000,1619e0000~d20000,162800000~1b00000,164600000~7550000,170d00000~1800000,172580000~100000,172d70000~1c190000,18f100000~40e0000,193700000~400000,193c90000~6970000,19aa90000~188d0000,1b3770000~c220000,1bff00000~1200000,1c12f0000~400000,1c2c00000~400000,1c3f80000~400000,1c5300000~4200000,1c95c0000~b9a40000,283800000~1800000,289000000~3800000,28d000000~6000000,293800000~2000000,296000000~e000000,2a4800000~465d0000,4b6100000~4c560000,502800000~32500000,cb8500000~10f600000,1139800000~7a000000,1668d30000~3d290000,1794000000~3e800000,191c000000~3ee00000,1a49800000~4110000,1a4f800000~40f0000,1a70800000~4100000,1a76000000~3c10000] = 0x582550000 : using 0x56d090000(22 GiB)
Any help here would be appreciated, I have stopped out CephFS file system but our radosgw is also impacted.
[0] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.681.log [1] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709.repair
Thanks, derek
Hi Igor, Thank you for the help. On 3/16/20 7:47 AM, Igor Fedotov wrote:
OSD-709 has been already expanded, right?
Correct with 'ceph-bluestore-tool --log-level 30 --path /var/lib/ceph/osd/ceph-709 --command bluefs-bdev-expand'. Does this expand bluefs and the data allocation? Is there a way to ask how much data is used by the non-bluefs portion? [root@obj21 ceph-709]# pwd /var/lib/ceph/osd/ceph-709 [root@obj21 ceph-709]# ls -la total 28 drwxrwxrwt. 2 ceph ceph 180 Mar 15 22:21 . drwxr-x---. 52 ceph ceph 4096 Jan 8 14:44 .. lrwxrwxrwx. 1 ceph ceph 30 Mar 15 22:21 block -> /dev/ceph-data-vol20/data-20-0 -rw-------. 1 ceph ceph 37 Mar 15 22:21 ceph_fsid -rw-------. 1 ceph ceph 37 Mar 15 22:21 fsid -rw-------. 1 ceph ceph 57 Mar 15 22:21 keyring -rw-------. 1 ceph ceph 6 Mar 15 22:21 ready -rw-------. 1 ceph ceph 10 Mar 15 22:21 type -rw-------. 1 ceph ceph 4 Mar 15 22:21 whoami [root@obj21 ceph-709]# lvdisplay /dev/ceph-data-vol20/data-20-0 --- Logical volume --- LV Path /dev/ceph-data-vol20/data-20-0 LV Name data-20-0 VG Name ceph-data-vol20 LV UUID idjDxJ-CbzN-fb1n-Bhzx-86vP-mSAD-HvIq5n LV Write Access read/write LV Creation host, time obj21.umiacs.umd.edu, 2018-10-10 08:15:20 -0400 LV Status available # open 0 LV Size 200.00 GiB Current LE 51200 Segments 4 Allocation inherit Read ahead sectors auto - currently set to 8192 Block device 253:33
What's the error reported by fsck?
It doesn't say, but when I try to run the deep fsck it produces [root@obj21 ceph-709]# ceph-bluestore-tool --log-level 30 --path /var/lib/ceph/osd/ceph-709 --command fsck 2020-03-16 08:02:16.590 7f5faaa11c00 -1 bluestore(/var/lib/ceph/osd/ceph-709) fsck error: bluefs_extents inconsistency, downgrade to previous releases might be broken. fsck found 1 error(s) [0] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709-fsck-deep
4) OSD.681 has a number of checksum verification errors when reading DB data:
2020-03-15 14:03:52.890 7f6311ffa700 3 rocksdb: [table/block_based_table_reader.cc:1117] Encountered error while reading data from compression dictionary block Corruption: block checksum mismatch: expected 0, got 2324967102 in db/012948.sst offset 18446744073709551615 size 18446744073709551615
Can't say if this is bound to space shortage or not. Wondering if other OSDs reported(-ing) something similar?
Here is another node which at around '2020-03-15 13:51' starts looking like peering a few pgs and then at '2020-03-15 14:40' on 716 fails, and then for example 719 it fails 1 min later at '2020-03-15 14:41'. [1] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.716.log-20200316.gz [2] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.717.log-20200316.gz [3] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.718.log-20200316.gz [4] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.719.log-20200316.gz [5] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.720.log-20200316.gz [6] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.721.log-20200316.gz [7] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.722.log-20200316.gz [8] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.722.log-20200316.gz Thanks, derek -- Derek T. Yarnell Director of Computing Facilities University of Maryland Institute for Advanced Computer Studies
Derek, On 3/16/2020 3:26 PM, Derek Yarnell wrote:
Hi Igor,
Thank you for the help.
On 3/16/20 7:47 AM, Igor Fedotov wrote:
OSD-709 has been already expanded, right? Correct with 'ceph-bluestore-tool --log-level 30 --path /var/lib/ceph/osd/ceph-709 --command bluefs-bdev-expand'. Does this expand bluefs and the data allocation?
Is there a way to ask how much data is used by the non-bluefs portion?
I can suggest the following non-straightforward way for now: 1) Check osd startup log for the following line: 2020-03-15 14:43:27.845 7f41bb6baa80 1 bluestore(/var/lib/ceph/osd/ceph-681) _open_alloc loaded 23 GiB in 97 extents Note 23GiB loaded. 2) Then retriever bluefs usage space for main device from "bluefs-bdev-sizes" output: 1 : device size 0x1a80000000 : own ... = 0x582550000 : using 0x56d090000(22 GiB) 3) Actual available space would be around: 1 GiB = 23GiB - 22 GiB
[root@obj21 ceph-709]# pwd /var/lib/ceph/osd/ceph-709 [root@obj21 ceph-709]# ls -la total 28 drwxrwxrwt. 2 ceph ceph 180 Mar 15 22:21 . drwxr-x---. 52 ceph ceph 4096 Jan 8 14:44 .. lrwxrwxrwx. 1 ceph ceph 30 Mar 15 22:21 block -> /dev/ceph-data-vol20/data-20-0 -rw-------. 1 ceph ceph 37 Mar 15 22:21 ceph_fsid -rw-------. 1 ceph ceph 37 Mar 15 22:21 fsid -rw-------. 1 ceph ceph 57 Mar 15 22:21 keyring -rw-------. 1 ceph ceph 6 Mar 15 22:21 ready -rw-------. 1 ceph ceph 10 Mar 15 22:21 type -rw-------. 1 ceph ceph 4 Mar 15 22:21 whoami [root@obj21 ceph-709]# lvdisplay /dev/ceph-data-vol20/data-20-0 --- Logical volume --- LV Path /dev/ceph-data-vol20/data-20-0 LV Name data-20-0 VG Name ceph-data-vol20 LV UUID idjDxJ-CbzN-fb1n-Bhzx-86vP-mSAD-HvIq5n LV Write Access read/write LV Creation host, time obj21.umiacs.umd.edu, 2018-10-10 08:15:20 -0400 LV Status available # open 0 LV Size 200.00 GiB Current LE 51200 Segments 4 Allocation inherit Read ahead sectors auto - currently set to 8192 Block device 253:33
What's the error reported by fsck? It doesn't say, but when I try to run the deep fsck it produces
[root@obj21 ceph-709]# ceph-bluestore-tool --log-level 30 --path /var/lib/ceph/osd/ceph-709 --command fsck 2020-03-16 08:02:16.590 7f5faaa11c00 -1 bluestore(/var/lib/ceph/osd/ceph-709) fsck error: bluefs_extents inconsistency, downgrade to previous releases might be broken. fsck found 1 error(s)
[0] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709-fsck-deep
fsck-deep suffers from the same lack of space. Could you please collect log for regular fsck? The latter opens DB in read-only and hence doesn't need additional space.
4) OSD.681 has a number of checksum verification errors when reading DB data:
2020-03-15 14:03:52.890 7f6311ffa700 3 rocksdb: [table/block_based_table_reader.cc:1117] Encountered error while reading data from compression dictionary block Corruption: block checksum mismatch: expected 0, got 2324967102 in db/012948.sst offset 18446744073709551615 size 18446744073709551615
Can't say if this is bound to space shortage or not. Wondering if other OSDs reported(-ing) something similar? Here is another node which at around '2020-03-15 13:51' starts looking like peering a few pgs and then at '2020-03-15 14:40' on 716 fails, and then for example 719 it fails 1 min later at '2020-03-15 14:41'.
[1] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.716.log-20200316.gz [2] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.717.log-20200316.gz [3] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.718.log-20200316.gz [4] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.719.log-20200316.gz [5] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.720.log-20200316.gz [6] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.721.log-20200316.gz [7] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.722.log-20200316.gz [8] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.722.log-20200316.gz
So looks like checksum errors appeared after the initial failure. And trigger recovery which requires additional space... I think the summary of the issue is as follows: Cluster had been in 'near full' state when some OSDs started to crash due to lack of free space. An attempt to extend device can succeed or not depending on the state RocksDB is in when the first crash happened. a) If DB is corrupted and needs for recovery (which is triggered on each non read-only DB open) it asks for more space which fails again and OSD fall into a "deadlock" state: to extend main device one needs DB access which in turn needs more space. b) If DB isn't corrupted expansion succeeds and OSD starts to get more data due to peering. Which finally fills it up and OSD tend to get into a). Some OSDs will presumably allow another expansion though. Unfortunately I don't know any fix/workaround for the "deadlock" case at the moment. Probably migrating DB to a standalone volume (using ceph-bluestore-tool's bluefs-bdev-migrate commands) will help but I need to double check that. And it will definitely expose data to risk of loss so please hold on until my additional recommendations. Most probably you will need additional 30GB of free space per each OSD if going this way. So please let me know if you can afford this.
Thanks, derek
Hi Igor, On 3/16/20 10:34 AM, Igor Fedotov wrote:
I can suggest the following non-straightforward way for now:
1) Check osd startup log for the following line:
2020-03-15 14:43:27.845 7f41bb6baa80 1 bluestore(/var/lib/ceph/osd/ceph-681) _open_alloc loaded 23 GiB in 97 extents
Note 23GiB loaded.
2) Then retriever bluefs usage space for main device from "bluefs-bdev-sizes" output:
1 : device size 0x1a80000000 : own ...
= 0x582550000 : using 0x56d090000(22 GiB)
3) Actual available space would be around: 1 GiB = 23GiB - 22 GiB
Ok so essentially the data is then the delta of what it reports in the following line? Why the descrepency from the _open_alloc loaded information and the bluefs-bdev-sizes output? bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-681/block size 106 GiB
[root@obj21 ceph-709]# ceph-bluestore-tool --log-level 30 --path /var/lib/ceph/osd/ceph-709 --command fsck 2020-03-16 08:02:16.590 7f5faaa11c00 -1 bluestore(/var/lib/ceph/osd/ceph-709) fsck error: bluefs_extents inconsistency, downgrade to previous releases might be broken. fsck found 1 error(s)
[0] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709-fsck-deep
fsck-deep suffers from the same lack of space. Could you please collect log for regular fsck?
That is the log for the regular fsck it only says that it founds 1 error(s) and doesn't give any further information (even with the --log-level 30 specified).
So looks like checksum errors appeared after the initial failure. And trigger recovery which requires additional space...
I think the summary of the issue is as follows:
Cluster had been in 'near full' state when some OSDs started to crash due to lack of free space.
An attempt to extend device can succeed or not depending on the state RocksDB is in when the first crash happened.
a) If DB is corrupted and needs for recovery (which is triggered on each non read-only DB open) it asks for more space which fails again and OSD fall into a "deadlock" state:
to extend main device one needs DB access which in turn needs more space.
b) If DB isn't corrupted expansion succeeds and OSD starts to get more data due to peering. Which finally fills it up and OSD tend to get into a).
Some OSDs will presumably allow another expansion though.
Unfortunately I don't know any fix/workaround for the "deadlock" case at the moment.
I am trying to find creative ways to allow to increase the space significantly on the OSD but not strand it so I can continue to provide new space to more of the OSDs. LVM is helpful here
Probably migrating DB to a standalone volume (using ceph-bluestore-tool's bluefs-bdev-migrate commands) will help but I need to double check that.
And it will definitely expose data to risk of loss so please hold on until my additional recommendations.
Most probably you will need additional 30GB of free space per each OSD if going this way. So please let me know if you can afford this.
Well I had already increased 709's initial space from 106GB to 200GB and now I gave it 10GB more but it still can not actually resize. Here is the relevant information I think but the full logs is here[0]. I then did it with 30G (now total of 240G) and it still failed[1]. I am out of space without some additional hardware in this node though I have an idea. If I knew what size it is (and what space it needs for recovery this would be very helpful). # ceph-bluestore-tool --log-level 30 --path /var/lib/ceph/osd/ceph-709 --command bluefs-bdev-expand -4> 2020-03-16 11:33:34.181 7f41d5940c00 -1 bluestore(/var/lib/ceph/osd/ceph-709) allocate_bluefs_freespace failed to allocate on 0xb000000 min_size 0xb000000 > allocated total 0x80000 bluefs_shared_alloc_size 0x10000 allocated 0x80000 available 0x 8000 -3> 2020-03-16 11:33:34.181 7f41d5940c00 -1 bluefs _allocate failed to expand slow device to fit +0xaffa895 -2> 2020-03-16 11:33:34.181 7f41d5940c00 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0xaffa895 -1> 2020-03-16 11:33:34.184 7f41d5940c00 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.6/rpm/el7/BUILD/ceph-14.2.6/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f41d5940c00 time 2020-03-16 11:33:34.181884 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.6/rpm/el7/BUILD/ceph-14.2.6/src/os/bluestore/BlueFS.cc: 2269: ceph_abort_msg("bluefs enospc") [0] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709.bluefs-bdev-expand [1] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709.bluefs-bdev-expand-2
Thanks, derek
-- Derek T. Yarnell Director of Computing Facilities University of Maryland Institute for Advanced Computer Studies
Hi Derek, On 3/16/2020 7:17 PM, Derek Yarnell wrote:
Hi Igor,
On 3/16/20 10:34 AM, Igor Fedotov wrote:
I can suggest the following non-straightforward way for now:
1) Check osd startup log for the following line:
2020-03-15 14:43:27.845 7f41bb6baa80 1 bluestore(/var/lib/ceph/osd/ceph-681) _open_alloc loaded 23 GiB in 97 extents
Note 23GiB loaded.
2) Then retriever bluefs usage space for main device from "bluefs-bdev-sizes" output:
1 : device size 0x1a80000000 : own ...
= 0x582550000 : using 0x56d090000(22 GiB)
3) Actual available space would be around: 1 GiB = 23GiB - 22 GiB Ok so essentially the data is then the delta of what it reports in the following line? Why the descrepency from the _open_alloc loaded information and the bluefs-bdev-sizes output? Probably - due to value rounding.
bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-681/block size 106 GiB
[root@obj21 ceph-709]# ceph-bluestore-tool --log-level 30 --path /var/lib/ceph/osd/ceph-709 --command fsck 2020-03-16 08:02:16.590 7f5faaa11c00 -1 bluestore(/var/lib/ceph/osd/ceph-709) fsck error: bluefs_extents inconsistency, downgrade to previous releases might be broken. fsck found 1 error(s)
[0] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709-fsck-deep fsck-deep suffers from the same lack of space. Could you please collect log for regular fsck? That is the log for the regular fsck it only says that it founds 1 error(s) and doesn't give any further information (even with the --log-level 30 specified). Missed attachment? So looks like checksum errors appeared after the initial failure. And trigger recovery which requires additional space...
I think the summary of the issue is as follows:
Cluster had been in 'near full' state when some OSDs started to crash due to lack of free space.
An attempt to extend device can succeed or not depending on the state RocksDB is in when the first crash happened.
a) If DB is corrupted and needs for recovery (which is triggered on each non read-only DB open) it asks for more space which fails again and OSD fall into a "deadlock" state:
to extend main device one needs DB access which in turn needs more space.
b) If DB isn't corrupted expansion succeeds and OSD starts to get more data due to peering. Which finally fills it up and OSD tend to get into a).
Some OSDs will presumably allow another expansion though.
Unfortunately I don't know any fix/workaround for the "deadlock" case at the moment. I am trying to find creative ways to allow to increase the space significantly on the OSD but not strand it so I can continue to provide new space to more of the OSDs. LVM is helpful here
Probably migrating DB to a standalone volume (using ceph-bluestore-tool's bluefs-bdev-migrate commands) will help but I need to double check that.
And it will definitely expose data to risk of loss so please hold on until my additional recommendations.
Most probably you will need additional 30GB of free space per each OSD if going this way. So please let me know if you can afford this. Well I had already increased 709's initial space from 106GB to 200GB and now I gave it 10GB more but it still can not actually resize. Here is the relevant information I think but the full logs is here[0]. I then did it with 30G (now total of 240G) and it still failed[1]. I am out of space without some additional hardware in this node though I have an idea. If I knew what size it is (and what space it needs for recovery this would be very helpful).
There is no much sense in increasing main device for this specific OSD (and similarly failing one, i.e. OSDs mentioning RocksDB recovery in back trace) at this point. It's in the "deadlock" state I mentioned before. And hence expand is unable to proceed. I'm checking some workarounds to get out of this state at the moment. Still in progress though. What I meant before is that you need more available space if workaround would be the assignment of a new standalone DB volume. It's a questionable way so I'm trying other ways for now.
# ceph-bluestore-tool --log-level 30 --path /var/lib/ceph/osd/ceph-709 --command bluefs-bdev-expand
-4> 2020-03-16 11:33:34.181 7f41d5940c00 -1 bluestore(/var/lib/ceph/osd/ceph-709) allocate_bluefs_freespace failed to allocate on 0xb000000 min_size 0xb000000 > allocated total 0x80000 bluefs_shared_alloc_size 0x10000 allocated 0x80000 available 0x 8000 -3> 2020-03-16 11:33:34.181 7f41d5940c00 -1 bluefs _allocate failed to expand slow device to fit +0xaffa895 -2> 2020-03-16 11:33:34.181 7f41d5940c00 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0xaffa895 -1> 2020-03-16 11:33:34.184 7f41d5940c00 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.6/rpm/el7/BUILD/ceph-14.2.6/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f41d5940c00 time 2020-03-16 11:33:34.181884 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.6/rpm/el7/BUILD/ceph-14.2.6/src/os/bluestore/BlueFS.cc: 2269: ceph_abort_msg("bluefs enospc")
[0] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709.bluefs-bdev-expand [1] - ftp://ftp.umiacs.umd.edu/pub/derek/ceph-osd.709.bluefs-bdev-expand-2
Thanks, derek
Hi Igor, I just want to thank you for taking the time to help with this issue. On 3/18/20 5:30 AM, Igor Fedotov wrote:
Most probably you will need additional 30GB of free space per each OSD if going this way. So please let me know if you can afford this. Well I had already increased 709's initial space from 106GB to 200GB and now I gave it 10GB more but it still can not actually resize. Here is the relevant information I think but the full logs is here[0]. I then did it with 30G (now total of 240G) and it still failed[1]. I am out of space without some additional hardware in this node though I have an idea. If I knew what size it is (and what space it needs for recovery this would be very helpful).
There is no much sense in increasing main device for this specific OSD (and similarly failing one, i.e. OSDs mentioning RocksDB recovery in back trace) at this point.
It's in the "deadlock" state I mentioned before. And hence expand is unable to proceed.
I'm checking some workarounds to get out of this state at the moment. Still in progress though.
What I meant before is that you need more available space if workaround would be the assignment of a new standalone DB volume. It's a questionable way so I'm trying other ways for now.
I had to go forward with this and migrate the db to a separate partition as we were production impacting. I did some testing by making a copy (dd'ing to new LVM) of one of the OSDs and performing the steps before I tried this. Essential steps (understanding this is a Nautilus cluster) for someone who might come across this in the future (for each OSD you need to do this for): 1) create new partition to hold the db volume lvcreate -L30G -n db-20-6 /dev/ceph-db-vol04 2) migrate with the ceph-bluestore-tool ceph-bluestore-tool bluefs-bdev-migrate --path /var/lib/ceph/osd/ceph-715 --dev-target /dev/ceph-db-vol04/db-20-6 --devs-source /var/lib/ceph/osd/ceph-715/block 3) make sure the block device are owned by the ceph user ( chown -h ceph:ceph /var/lib/ceph/osd/ceph-715/block.db chown ceph:ceph /dev/ceph-db-vol04/db-20-6 4) run the ceph-bluestore-tool repair ceph-bluestore-tool --log-level 30 --path /var/lib/ceph/osd/ceph-715 --command repair 5) test compaction ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-715 compact 6) start the OSD systemctl start ceph-osd@715 Lesson is that bluestore does not handle well when you hit capacity like this with the simple configuration of co-located rocksdb and data. I think that even for fast disks such as nvme you should always create a separate db partition as this deadlock scenario is very problematic if you don't have additional storage (or can add some quickly). It seems that we had about 2million cephfs log segments that were behind on trimming. I am not sure where these segments are kept but I am guessing in the mds metadata pool which seems to have driven this maximum space issue. As we are now down to about %17 used in the nvme class when we were at 100% during this issue. Thanks, derek -- Derek T. Yarnell Director of Computing Facilities University of Maryland Institute for Advanced Computer Studies
participants (2)
-
Derek Yarnell
-
Igor Fedotov