5 Feb
2021
5 Feb
'21
4:34 a.m.
Hi,
With 15.2.8, run "ceph orch rm osd 12 --replace --force",
PGs on osd.12 are remapped, osd.12 is removed from "ceph osd tree",
the daemon is removed from "ceph orch ps", the device is "available"
in "ceph orch device ls". Everything seems good at this point.
Then dry-run service spec.
```
# cat osd-spec.yaml
service_type: osd
service_id: osd-spec
placement:
hosts:
- ceph-osd-1
data_devices:
rotational: 1
db_devices:
rotational: 0
# ceph orch apply osd -i osd-spec.yaml --dry-run
+---------+----------+------------+----------+----------+-----+
|SERVICE |NAME |HOST |DATA |DB |WAL |
+---------+----------+------------+----------+----------+-----+
|osd |osd-spec |ceph-osd-3 |/dev/sdd |/dev/sdb |- |
+---------+----------+------------+----------+----------+-----+
```
It looks as expected.
Then "ceph orch apply osd -i osd-spec.yaml".
Here is the log of cephadm.
```
/bin/docker:stderr --> relative data size: 1.0
/bin/docker:stderr --> passed block_db devices: 1 physical, 0 LVM
/bin/docker:stderr Running command: /usr/bin/ceph-authtool --gen-print-key
/bin/docker:stderr Running command: /usr/bin/ceph --cluster ceph --name
client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd tree -f json
/bin/docker:stderr Running command: /usr/bin/ceph --cluster ceph --name
client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new
b05c3c90-b7d5-4f13-8a58-f72761c1971b 12
/bin/docker:stderr Running command: /usr/sbin/vgcreate --force --yes
ceph-a3886f74-3de9-4e6e-a983-8330eda0bd64 /dev/sdd
/bin/docker:stderr stdout: Physical volume "/dev/sdd" successfully created.
/bin/docker:stderr stdout: Volume group
"ceph-a3886f74-3de9-4e6e-a983-8330eda0bd64" successfully created
/bin/docker:stderr Running command: /usr/sbin/lvcreate --yes -l 572318 -n
osd-block-b05c3c90-b7d5-4f13-8a58-f72761c1971b ceph-a3886f74-3de9-4e6e-a983-8330eda0bd64
/bin/docker:stderr stderr: Volume group
"ceph-a3886f74-3de9-4e6e-a983-8330eda0bd64" has insufficient free space (572317
extents): 572318 required.
/bin/docker:stderr --> Was unable to complete a new OSD, will rollback changes
```
Q1, why VG name (ceph-<id>) is different from others (ceph-block-<id>)?
Q2, where is that 572318 from? Since all HDDs are the same model, VG
"Total PE" of all HDDs is 572317.
Has anyone seen similar issues? Anything I am missing?
Thanks!
Tony