8 Jul
2021
8 Jul
'21
3:20 p.m.
Hi,
I am trying to apply the resharding to a containerized OSD (16.2.4) as described here:
https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#r…
# ceph osd set noout
# ceph orch daemon stop osd.13
# cephadm shell --name osd.13
# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-13 --sharding="m(3) p(3,0-12)
O(3,0-13)=block_cache={type=binned_lru} L P" reshard
2021-07-08T15:12:03.392+0000 7f2e2173b3c0 -1 rocksdb: prepare_for_reshard failure parsing
column options: block_cache={type=binned_lru}
error resharding: (22) Invalid argument
# exit
# ceph orch daemon start osd.13
# ceph osd unset noout
The OSD cannot start any more and has these errors in its log:
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080
0 osd.13:7.OSDShard using op scheduler
ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080
1 bluestore(/var/lib/ceph/osd/ceph-13) _mount path /var/lib/ceph/osd/ceph-13
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080
0 bluestore(/var/lib/ceph/osd/ceph-13) _open_db_and_around read-only:0 repair:0
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080
1 bdev(0x564271210800 /var/lib/ceph/osd/ceph-13/block) open path
/var/lib/ceph/osd/ceph-13/block
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080
1 bdev(0x564271210800 /var/lib/ceph/osd/ceph-13/block) open size 107369988096
(0x18ffc00000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080
1 bluestore(/var/lib/ceph/osd/ceph-13) _set_cache_sizes cache_size 3221225472 meta 0.45 kv
0.45 data 0.06
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080
1 bdev(0x564271210c00 /var/lib/ceph/osd/ceph-13/block) open path
/var/lib/ceph/osd/ceph-13/block
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080
1 bdev(0x564271210c00 /var/lib/ceph/osd/ceph-13/block) open size 107369988096
(0x18ffc00000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080
1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-13/block size 100 GiB
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080
1 bluefs mount
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080
1 bluefs _init_alloc shared, id 1, capacity 0x18ffc00000, block size 0x10000
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.325+0000 7efc32db4080
1 bluefs mount shared_bdev_used = 0
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.325+0000 7efc32db4080
-1 rocksdb: verify_sharding extra columns in rocksdb. rocksdb columns =
[default,m-0,m-1,m-2,p-0,p-1,p-2] target columns = [reshardingXcommencingXlocked]
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.325+0000 7efc32db4080
-1 bluestore(/var/lib/ceph/osd/ceph-13) _open_db erroring opening db:
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.325+0000 7efc32db4080
1 bluefs umount
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.325+0000 7efc32db4080
1 bdev(0x564271210c00 /var/lib/ceph/osd/ceph-13/block) close
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.577+0000 7efc32db4080
1 bdev(0x564271210800 /var/lib/ceph/osd/ceph-13/block) close
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.825+0000 7efc32db4080
-1 osd.13 0 OSD:init: unable to mount object store
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.825+0000 7efc32db4080
-1 ** ERROR: osd init failed: (5) Input/output error
How do I correct the issue?
Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin
http://www.heinlein-support.de
Tel: 030 / 405051-43
Fax: 030 / 405051-19
Zwangsangaben lt. §35a GmbHG:
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin