Hey, no problem and thank you! This is the output of lsblk: sda 8:0 0 14.6T 0 disk └─ceph--937823b8--204b--4190--9bd1--f867e64621db-osd--block--a4bbaa5d--eb2d--41f3--8f4e--f8c5a2747012 253:24 0 14.6T 0 lvm sdb 8:16 0 14.6T 0 disk └─ceph--169752b2--8095--41e9--87f2--cc6e962231ed-osd--block--8802416f--2e00--4388--847e--0e44b4a3afe2 253:35 0 14.6T 0 lvm sdc 8:32 0 14.6T 0 disk └─ceph--8373a100--9457--43e9--a7c4--d573c2be9c0c-osd--block--7f582b9b--1433--4151--a644--6528e6991f75 253:25 0 14.6T 0 lvm sdd 8:48 0 14.6T 0 disk └─ceph--5f830838--dbcf--4071--9886--5fff17a44590-osd--block--d9b18b2c--0bd3--4a9f--a22c--7a5b27abda3c 253:30 0 14.6T 0 lvm sde 8:64 0 14.6T 0 disk └─ceph--aea40ec8--547e--4ebc--86af--b773a45ac857-osd--block--6dce53bc--1aef--4ee5--95c0--b2b9bcdb5733 253:19 0 14.6T 0 lvm sdf 8:80 0 14.6T 0 disk └─ceph--0a6669a8--7c6f--4fef--9b49--46b0a6a62ece-osd--block--859896ad--e839--42d6--a84c--da244454bc6d 253:34 0 14.6T 0 lvm sdg 8:96 0 14.6T 0 disk └─ceph--cf96ae54--eb04--4ef1--a162--6c647e23a139-osd--block--65a1f8a8--bdaa--4ffd--9c1b--2d60381a33fc 253:29 0 14.6T 0 lvm sdh 8:112 0 14.6T 0 disk └─ceph--67deea47--0236--4392--b0c1--ecd5b30be3c6-osd--block--a7a03d63--91a6--4207--8192--ab078d4d596b 253:22 0 14.6T 0 lvm sdi 8:128 0 14.6T 0 disk └─ceph--eef96e23--7f9e--458e--bc54--028c1161e11a-osd--block--10c6bdc6--9071--4c2f--b29e--7e5320a19f01 253:20 0 14.6T 0 lvm sdj 8:144 0 14.6T 0 disk └─ceph--979b74d7--ac00--4e40--8e20--c45c274d8c3e-osd--block--eecdd1d8--1b2e--4e96--9914--b91623932bae 253:33 0 14.6T 0 lvm sdk 8:160 0 14.6T 0 disk └─ceph--c94872dc--2567--4d73--b12c--0ab5bf700889-osd--block--72a52d58--06df--423f--89d6--a20b92f784b7 253:32 0 14.6T 0 lvm sdl 8:176 0 14.6T 0 disk └─ceph--e480c934--f576--4a4c--821a--8328e2b23137-osd--block--9d67203d--37bb--4f36--9153--81550e1389db 253:28 0 14.6T 0 lvm sdm 8:192 0 14.6T 0 disk └─ceph--1a529e9a--b15c--4b04--afe9--64f18a728c63-osd--block--70abcdcf--88c2--4085--b5e0--2f028ad946a1 253:36 0 14.6T 0 lvm sdn 8:208 0 14.6T 0 disk └─ceph--3e15ae4d--72f3--4c8d--979d--e00fabc5fe99-osd--block--ff7ebdca--afe0--477c--81a7--0773b6449487 253:26 0 14.6T 0 lvm sdo 8:224 0 14.6T 0 disk └─ceph--81805753--ae18--4416--87f7--3a996ece90f3-osd--block--4ae6d812--2e33--4fbd--8d79--f2260af1765f 253:23 0 14.6T 0 lvm sdp 8:240 0 14.6T 0 disk └─ceph--6776033b--3a45--47fe--aa5e--45e82602a10d-osd--block--67f8fe64--8cc7--43e1--857a--8d59fd5d0f81 253:31 0 14.6T 0 lvm sdq 65:0 0 14.6T 0 disk └─ceph--a3ee82aa--0811--4a4d--80fc--ada3ba93376f-osd--block--f5d90973--eed3--4a1b--923d--a39263cb8546 253:21 0 14.6T 0 lvm sdr 65:16 0 14.6T 0 disk └─ceph--29846433--9bf8--4c2c--89b8--054af1f301c7-osd--block--f2910199--3df5--488e--aa2a--4ccd98ba6453 253:27 0 14.6T 0 lvm sds 65:32 0 400G 0 disk ├─sds1 65:33 0 1G 0 part /boot/efi ├─sds2 65:34 0 2G 0 part /boot └─sds3 65:35 0 396.9G 0 part └─ubuntu--vg-ubuntu--lv 253:18 0 300G 0 lvm / nvme1n1 259:2 0 5.8T 0 disk ├─ceph--b38117e8--8e50--48dd--95f2--b4226286bfde-osd--wal--09370408--a1a5--4d32--9a15--9da8ccac931d 253:0 0 331.2G 0 lvm ├─ceph--b38117e8--8e50--48dd--95f2--b4226286bfde-osd--wal--324bb7f9--afac--49d3--910a--73643f6c09b2 253:1 0 331.2G 0 lvm ├─ceph--b38117e8--8e50--48dd--95f2--b4226286bfde-osd--wal--e65bf85c--96d2--4ae2--b9d1--9a60f957ab97 253:2 0 331.2G 0 lvm ├─ceph--b38117e8--8e50--48dd--95f2--b4226286bfde-osd--wal--a7f4cf98--8e1a--4d8a--b622--787d5db4ee3e 253:3 0 331.2G 0 lvm ├─ceph--b38117e8--8e50--48dd--95f2--b4226286bfde-osd--wal--bb925b1b--6d02--4be6--aa4f--254e4f1e8202 253:4 0 331.2G 0 lvm ├─ceph--b38117e8--8e50--48dd--95f2--b4226286bfde-osd--wal--fff0b13d--a90c--4ada--a6fa--20e98cb956c8 253:5 0 331.2G 0 lvm ├─ceph--b38117e8--8e50--48dd--95f2--b4226286bfde-osd--wal--139637be--7bc9--4398--8461--3e88382a9eaa 253:6 0 331.2G 0 lvm ├─ceph--b38117e8--8e50--48dd--95f2--b4226286bfde-osd--wal--a274b742--65ba--4bf3--918e--166023565c69 253:7 0 331.2G 0 lvm └─ceph--b38117e8--8e50--48dd--95f2--b4226286bfde-osd--wal--d7efd132--5079--4ae5--aaa2--d57e05e86fb6 253:8 0 331.2G 0 lvm nvme0n1 259:3 0 5.8T 0 disk ├─ceph--3a336b8e--ed39--4532--a199--ac6a3730840b-osd--wal--50092d2d--f06a--47f9--adce--ca9344d5615f 253:9 0 331.2G 0 lvm ├─ceph--3a336b8e--ed39--4532--a199--ac6a3730840b-osd--wal--cc1d8423--d699--4b69--a426--87f01d289e2d 253:10 0 331.2G 0 lvm ├─ceph--3a336b8e--ed39--4532--a199--ac6a3730840b-osd--wal--f5b7018b--8537--4153--b6f5--2ecbe4d1f109 253:11 0 331.2G 0 lvm ├─ceph--3a336b8e--ed39--4532--a199--ac6a3730840b-osd--wal--5d845dba--8b55--4984--890b--547fbdaff10c 253:12 0 331.2G 0 lvm ├─ceph--3a336b8e--ed39--4532--a199--ac6a3730840b-osd--wal--e100d725--fa1e--47d5--844b--d7fdca8093ca 253:13 0 331.2G 0 lvm ├─ceph--3a336b8e--ed39--4532--a199--ac6a3730840b-osd--wal--62f7a57b--dc82--45c9--a2df--6adbfe445893 253:14 0 331.2G 0 lvm ├─ceph--3a336b8e--ed39--4532--a199--ac6a3730840b-osd--wal--9d42a505--9e3f--48e1--8b4d--f88896f022d0 253:15 0 331.2G 0 lvm ├─ceph--3a336b8e--ed39--4532--a199--ac6a3730840b-osd--wal--15993faf--32e3--47ef--b527--903ef37624a2 253:16 0 331.2G 0 lvm └─ceph--3a336b8e--ed39--4532--a199--ac6a3730840b-osd--wal--45c7a924--ec0d--45d8--a321--b660f86ad83c 253:17 0 331.2G 0 lvm Best Ken On 07.02.23 15:30, Guillaume Abrioux wrote:
Hello,
I'm sorry for not getting back to you sooner.
[2023-01-26 16:25:00,785][ceph_volume.process][INFO ] stdout ceph.block_device=/dev/ceph-808efc2a-54fd-47cc-90e2-c5cc96bdd825/osd-block-2a1d1bf0-300e-4160-ac55-047837a5af0b,ceph.block_uuid=b4WDQQ-eMTb-AN1U-D7dk-yD2q-4dPZ-KyFrHi,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=8038f09a-27a0-11ed-8de8-55262cdd5a37,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=0,ceph.osd_fsid=2a1d1bf0-300e-4160-ac55-047837a5af0b,ceph.osd_id=232,ceph.osdspec_affinity=dashboard-admin-1661788934732,ceph.type=wal,ceph.vdo=0,ceph.wal_device=/dev/ceph-3a336b8e-ed39-4532-a199-ac6a3730840b/osd-wal-5d845dba-8b55-4984-890b-547fbdaff10c,ceph.wal_uuid=dquBMJ-s8ou-Wp6M-NY8Z-QoFh-6L4b-9Lwqm0";"/dev/ceph-3a336b8e-ed39-4532-a199-ac6a3730840b/osd-wal-5d845dba-8b55-4984-890b-547fbdaff10c";"osd-wal-5d845dba-8b55-4984-890b-547fbdaff10c";"ceph-3a336b8e-ed39-4532-a199-ac6a3730840b";"dquBMJ-s8ou-Wp6M-NY8Z-QoFh-6L4b-9Lwqm0";"355622453248
From that line, I read that osd.232 has block on /dev/ceph-808efc2a-54fd-47cc-90e2-c5cc96bdd825/osd-block-2a1d1bf0-300e-4160-ac55-047837a5af0b and block.wal on /dev/ceph-3a336b8e-ed39-4532-a199-ac6a3730840b/osd-wal-5d845dba-8b55-4984-890b-547fbdaff10c from there, check if that device is well an LV member of the NVME device. Can you share the full output of lsblk ?
Thanks,
On Wed, 1 Feb 2023 at 17:22, mailing-lists <mailing-lists@indane.de> wrote:
I've pulled a few lines from the log and i've attached this to this mail. (I hope this works for this mailinglist?)
I found the line 135
[2023-01-26 16:25:00,785][ceph_volume.process][INFO ] stdout ceph.block_device=/dev/ceph-808efc2a-54fd-47cc-90e2-c5cc96bdd825/osd-block-2a1d1bf0-300e-4160-ac55-047837a5af0b,ceph.block_uuid=b4WDQQ-eMTb-AN1U-D7dk-yD2q-4dPZ-KyFrHi,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=8038f09a-27a0-11ed-8de8-55262cdd5a37,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=0,ceph.osd_fsid=2a1d1bf0-300e-4160-ac55-047837a5af0b,ceph.osd_id=232,ceph.osdspec_affinity=dashboard-admin-1661788934732,ceph.type=wal,ceph.vdo=0,ceph.wal_device=/dev/ceph-3a336b8e-ed39-4532-a199-ac6a3730840b/osd-wal-5d845dba-8b55-4984-890b-547fbdaff10c,ceph.wal_uuid=dquBMJ-s8ou-Wp6M-NY8Z-QoFh-6L4b-9Lwqm0";"/dev/ceph-3a336b8e-ed39-4532-a199-ac6a3730840b/osd-wal-5d845dba-8b55-4984-890b-547fbdaff10c";"osd-wal-5d845dba-8b55-4984-890b-547fbdaff10c";"ceph-3a336b8e-ed39-4532-a199-ac6a3730840b";"dquBMJ-s8ou-Wp6M-NY8Z-QoFh-6L4b-9Lwqm0";"355622453248
Which indicates that this OSD is in fact using a WAL, since WAL and DB should both be on the NVME, i would guess it is just a visual bug in the dashboard?
From line 135:
ceph.wal_device=/dev/ceph-3a336b8e-ed39-4532-a199-ac6a3730840b/osd-wal-5d845dba-8b55-4984-890b-547fbdaff10c
From lsblk:
├─ceph--3a336b8e--ed39--4532--a199--ac6a3730840b-osd--wal--5d845dba--8b55--4984--890b--547fbdaff10c 253:12 0 331.2G 0 lvm
So it looks like it is using that lvm group right there. Yet, the dashboard doesn't show a nvme. (please compare screenshot osd_232.png and osd_218.png)
Can I somehow confirm, that my osd 232 is really using the nvme as wal/db?
Thanks and best regards
Ken
On 01.02.23 10:35, Guillaume Abrioux wrote:
Any chance you can share the ceph-volume.log (from the corresponding host)? It should be in /var/log/ceph/<cluster fsid>/ceph-volume.log. Note that there might be several log files (log rotation). Ideally, the one that includes the recreation steps.
Thanks,
On Wed, 1 Feb 2023 at 10:13, mailing-lists <mailing-lists@indane.de> wrote:
Ah, nice.
service_type: osd service_id: dashboard-admin-1661788934732 service_name: osd.dashboard-admin-1661788934732 placement: host_pattern: '*' spec: data_devices: model: MG08SCA16TEY db_devices: model: Dell Ent NVMe AGN MU AIC 6.4TB filter_logic: AND objectstore: bluestore wal_devices: model: Dell Ent NVMe AGN MU AIC 6.4TB status: created: '2022-08-29T16:02:22.822027Z' last_refresh: '2023-02-01T09:03:22.853860Z' running: 306 size: 306
Best
Ken
On 31.01.23 23:51, Guillaume Abrioux wrote:
On Tue, 31 Jan 2023 at 22:31, mailing-lists <mailing-lists@indane.de> wrote:
I am not sure. I didn't find it... It should be somewhere, right? I used the dashboard to create the osd service.
what does a `cephadm shell -- ceph orch ls osd --format yaml` say?
-- *Guillaume Abrioux *Senior Software Engineer
-- *Guillaume Abrioux *Senior Software Engineer
-- *Guillaume Abrioux *Senior Software Engineer