Hi Joshua and thanks for this quick reply. At this step I have only one node. I was checking what ceph was returning with different commands on this host before adding new hosts. Just to compare with my first Octopus install. As this hardware is for testing only, it remains easy for me to break everything and reinstall again. [root@mostha1 ~]# cephadm check-host podman (/usr/bin/podman) version 4.2.0 is present systemctl is present lvcreate is present Unit chronyd.service is enabled and running Host looks OK [ceph: root@mostha1 /]# ceph -s   cluster:     id:     4b7a6504-f0be-11ed-be1a-00266cf8869c     health: HEALTH_WARN             OSD count 0 < osd_pool_default_size 3   services:     mon: 1 daemons, quorum mostha1.legi.grenoble-inp.fr (age 5h)     mgr: mostha1.legi.grenoble-inp.fr.hogwuz(active, since 5h)     osd: 0 osds: 0 up, 0 in   data:     pools:   0 pools, 0 pgs     objects: 0 objects, 0 B     usage:   0 B used, 0 B / 0 B avail     pgs: [ceph: root@mostha1 /]# ceph orch ls NAME           PORTS RUNNING  REFRESHED  AGE  PLACEMENT alertmanager   ?:9093,9094      1/1  6m ago     6h   count:1 crash                           1/1  6m ago     6h   * grafana        ?:3000           1/1  6m ago     6h   count:1 mgr                             1/2  6m ago     6h   count:2 mon                             1/5  6m ago     6h   count:5 node-exporter  ?:9100           1/1  6m ago     6h   * prometheus     ?:9095           1/1  6m ago     6h count:1 [ceph: root@mostha1 /]# ceph orch ls osd -export No services reported [ceph: root@mostha1 /]# ceph orch host ls HOST ADDR           LABELS  STATUS mostha1.legi.grenoble-inp.fr 194.254.66.34  _admin 1 hosts in cluster [ceph: root@mostha1 /]# ceph log last cephadm ... 2023-05-12T15:19:58.754655+0000 mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1876 : cephadm [INF] Zap device mostha1.legi.grenoble-inp.fr:/dev/sdb 2023-05-12T15:19:58.756639+0000 mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1877 : cephadm [ERR] Device path '/dev/sdb' not found on host 'mostha1.legi.grenoble-inp.fr' Traceback (most recent call last):   File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper     return OrchResult(f(*args, **kwargs))   File "/usr/share/ceph/mgr/cephadm/module.py", line 2275, in zap_device     f"Device path '{path}' not found on host '{host}'") orchestrator._interface.OrchestratorError: Device path '/dev/sdb' not found on host 'mostha1.legi.grenoble-inp.fr' .... [ceph: root@mostha1 /]# ls -l /dev/sdb brw-rw---- 1 root disk 8, 16 May 12 15:16 /dev/sdb [ceph: root@mostha1 /]# lsblk /dev/sdb NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT sdb      8:16   1 465.8G  0 disk `-sdb1   8:17   1 465.8G  0 part I have crated a full partition on /dev/sdb (for testing) and /dev/sdc has no partition table (removed). But all seams fine with these commands. Patrick Le 12/05/2023 à 20:19, Beaman, Joshua a écrit :

I don’t quite understand why that zap would not work.  But, here’s where I’d start.

1. cephadm check-host 1. Run this on each of your hosts to make sure cephadm, podman and all other prerequisites are installed and recognized 2. ceph orch ls 1. This should show at least a mon, mgr, and osd spec deployed 3. ceph orch ls osd –export 1. This will show the OSD placement service specifications that orchestrator uses to identify devices to deploy as OSDs 4. ceph orch host ls 1. This will list the hosts that have been added to orchestrator’s inventory, and what labels are applied which correlate to the service placement labels 5. ceph log last cephadm 1. This will show you what orchestrator has been trying to do, and how it may be failing

Also, it’s never un-helpful to have a look at “ceph -s” and “ceph health detail”, particularly for any people trying to help you without access to your systems.

Best of luck,

Josh Beaman

*From: *Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> *Date: *Friday, May 12, 2023 at 10:45 AM *To: *ceph-users <ceph-users@ceph.io> *Subject: *[EXTERNAL] [ceph-users] [Pacific] ceph orch device ls do not returns any HDD

Hi everyone

I'm new to CEPH, just a french 4 days training session with Octopus on VMs that convince me to build my first cluster.

At this time I have 4 old identical nodes for testing with 3 HDDs each, 2 network interfaces and running Alma Linux8 (el8). I try to replay the training session but it fails, breaking the web interface because of some problems with podman 4.2 not compatible with Octopus.

So I try to deploy Pacific with cephadm tool on my first node (mostha1) (to enable testing also an upgrade later).

    dnf -y install https://urldefense.com/v3/__https://download.ceph.com/rpm-16.2.13/el8/noarch... <https://urldefense.com/v3/__https:/download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noarch.rpm__;!!CQl3mcHX2A!H9cwNCJyKXYQ4BbGA3gwHHRitjOS4lBCZT9wlnBZ-8IDue0MvdcPD8Dnv5yQCZw_eA4BNDYaEq1eouKQcQO7HshgdUJ0SJ-EgLfaBGBmCQ$>

    monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }')     cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \                        --initial-dashboard-user admceph \                        --allow-fqdn-hostname --cluster-network 10.1.0.0/16

This was sucessfull.

But running "*c**eph orch device ls*" do not show any HDD even if I have /dev/sda (used by the OS), /dev/sdb and /dev/sdc

The web interface shows a row capacity which is an aggregate of the sizes of the 3 HDDs for the node.

I've also tried to reset /dev/sdb but cephadm do not see it:

    [ceph: root@mostha1 /]# ceph orch device zap     mostha1.legi.grenoble-inp.fr /dev/sdb --force     Error EINVAL: Device path '/dev/sdb' not found on host     'mostha1.legi.grenoble-inp.fr'

On my first attempt with octopus, I was able to list the available HDD with this command line. Before moving to Pacific, the OS on this node has been reinstalled from scratch.

Any advices for a CEPH beginner ?

Thanks

Patrick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Joshua, I've tried these commands but it looks like CEPH is unable to see and configure these HDDs. [root@mostha1 ~]# cephadm ceph-volume inventory Inferring fsid 4b7a6504-f0be-11ed-be1a-00266cf8869c Using recent ceph image quay.io/ceph/ceph@sha256:e6919776f0ff8331a8e9c4b18d36c5e9eed31e1a80da62ae8454e42d10e95544 Device Path               Size         Device nodes    rotates available Model name [root@mostha1 ~]# cephadm shell [ceph: root@mostha1 /]# ceph orch apply osd --all-available-devices Scheduled osd.all-available-devices update... [ceph: root@mostha1 /]# ceph orch device ls[ceph: root@mostha1 /]# ceph-volume lvm zap /dev/sdb --> Zapping: /dev/sdb --> --destroy was not specified, but zapping a whole device will remove the partition table Running command: /usr/bin/dd if=/dev/zero of=/dev/sdb bs=1M count=10 conv=fsync  stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.10039 s, 104 MB/s --> Zapping successful for: <Raw Device: /dev/sdb> I can check that /dev/sdb1 has been erased, so previous command is successful [ceph: root@mostha1 ceph]# lsblk NAME                 MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT sda                    8:0    1 232.9G  0 disk |-sda1                 8:1    1   3.9G  0 part /rootfs/boot |-sda2                 8:2    1  78.1G  0 part | `-osvg-rootvol     253:0    0  48.8G  0 lvm  /rootfs |-sda3                 8:3    1   3.9G  0 part [SWAP] `-sda4                 8:4    1 146.9G  0 part   |-secretvg-homevol 253:1    0   9.8G  0 lvm  /rootfs/home   |-secretvg-tmpvol  253:2    0   9.8G  0 lvm  /rootfs/tmp   `-secretvg-varvol  253:3    0   9.8G  0 lvm  /rootfs/var sdb                    8:16   1 465.8G  0 disk sdc                    8:32   1 232.9G  0 disk But still no visible HDD: [ceph: root@mostha1 ceph]# ceph orch apply osd --all-available-devices Scheduled osd.all-available-devices update... [ceph: root@mostha1 ceph]# ceph orch device ls [ceph: root@mostha1 ceph]# May be I have done something bad at install time as in the container I've unintentionally run: dnf -y install https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noarc... (an awful copy/paste launching the command). Can this break The container ? I do not know what should be available as ceph packages in the container to remove properly this install (no dnf.log file in the container) Patrick Le 12/05/2023 à 21:38, Beaman, Joshua a écrit :

The most significant point I see there, is you have no OSD service spec to tell orchestrator how to deploy OSDs.  The easiest fix for that would be “cephorchapplyosd--all-available-devices”

This will create a simple spec that should work for a test environment.  Most likely it will collocate the block, block.db, and WAL all on the same device.  Not ideal for prod environments, but fine for practice and testing.

The other command I should have had you try is “cephadm ceph-volume inventory”.  That should show you the devices available for OSD deployment, and hopefully matches up to what your “lsblk” shows.  If you need to zap HDDs and orchestrator is still not seeing them, you can try “cephadm ceph-volume lvm zap /dev/sdb”

Thank you,

Josh Beaman

*From: *Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> *Date: *Friday, May 12, 2023 at 2:22 PM *To: *Beaman, Joshua <Joshua_Beaman@comcast.com>, ceph-users <ceph-users@ceph.io> *Subject: *Re: [EXTERNAL] [ceph-users] [Pacific] ceph orch device ls do not returns any HDD

Hi Joshua and thanks for this quick reply.

At this step I have only one node. I was checking what ceph was returning with different commands on this host before adding new hosts. Just to compare with my first Octopus install. As this hardware is for testing only, it remains easy for me to break everything and reinstall again.

[root@mostha1 ~]# cephadm check-host

podman (/usr/bin/podman) version 4.2.0 is present systemctl is present lvcreate is present Unit chronyd.service is enabled and running Host looks OK

[ceph: root@mostha1 /]# ceph -s

  cluster:     id:     4b7a6504-f0be-11ed-be1a-00266cf8869c     health: HEALTH_WARN             OSD count 0 < osd_pool_default_size 3

  services:     mon: 1 daemons, quorum mostha1.legi.grenoble-inp.fr (age 5h)     mgr: mostha1.legi.grenoble-inp.fr.hogwuz(active, since 5h)     osd: 0 osds: 0 up, 0 in

  data:     pools:   0 pools, 0 pgs     objects: 0 objects, 0 B     usage:   0 B used, 0 B / 0 B avail     pgs:

[ceph: root@mostha1 /]# ceph orch ls

NAME           PORTS        RUNNING REFRESHED  AGE  PLACEMENT alertmanager   ?:9093,9094      1/1  6m ago     6h count:1 crash                           1/1  6m ago     6h * grafana        ?:3000           1/1  6m ago     6h count:1 mgr                             1/2  6m ago     6h count:2 mon                             1/5  6m ago     6h count:5 node-exporter  ?:9100           1/1  6m ago     6h * prometheus     ?:9095           1/1  6m ago     6h count:1

[ceph: root@mostha1 /]# ceph orch ls osd -export

No services reported

[ceph: root@mostha1 /]# ceph orch host ls

HOST                          ADDR LABELS  STATUS mostha1.legi.grenoble-inp.fr  194.254.66.34 _admin 1 hosts in cluster

[ceph: root@mostha1 /]# ceph log last cephadm

... 2023-05-12T15:19:58.754655+0000 mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1876 : cephadm [INF] Zap device mostha1.legi.grenoble-inp.fr:/dev/sdb 2023-05-12T15:19:58.756639+0000 mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1877 : cephadm [ERR] Device path '/dev/sdb' not found on host 'mostha1.legi.grenoble-inp.fr' Traceback (most recent call last):   File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper     return OrchResult(f(*args, **kwargs))   File "/usr/share/ceph/mgr/cephadm/module.py", line 2275, in zap_device     f"Device path '{path}' not found on host '{host}'") orchestrator._interface.OrchestratorError: Device path '/dev/sdb' not found on host 'mostha1.legi.grenoble-inp.fr' ....

[ceph: root@mostha1 /]# ls -l /dev/sdb

brw-rw---- 1 root disk 8, 16 May 12 15:16 /dev/sdb

[ceph: root@mostha1 /]# lsblk /dev/sdb

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT sdb      8:16   1 465.8G  0 disk `-sdb1   8:17   1 465.8G  0 part

I have crated a full partition on /dev/sdb (for testing) and /dev/sdc has no partition table (removed).

But all seams fine with these commands.

Patrick

Le 12/05/2023 à 20:19, Beaman, Joshua a écrit :

I don’t quite understand why that zap would not work.  But, here’s where I’d start.

1. cephadm check-host

1. Run this on each of your hosts to make sure cephadm, podman and all other prerequisites are installed and recognized

2. ceph orch ls

1. This should show at least a mon, mgr, and osd spec deployed

3. ceph orch ls osd –export

1. This will show the OSD placement service specifications that orchestrator uses to identify devices to deploy as OSDs

4. ceph orch host ls

1. This will list the hosts that have been added to orchestrator’s inventory, and what labels are applied which correlate to the service placement labels

5. ceph log last cephadm

1. This will show you what orchestrator has been trying to do, and how it may be failing

Also, it’s never un-helpful to have a look at “ceph -s” and “ceph health detail”, particularly for any people trying to help you without access to your systems.

Best of luck,

Josh Beaman

*From: *Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> <mailto:Patrick.Begou@univ-grenoble-alpes.fr> *Date: *Friday, May 12, 2023 at 10:45 AM *To: *ceph-users <ceph-users@ceph.io> <mailto:ceph-users@ceph.io> *Subject: *[EXTERNAL] [ceph-users] [Pacific] ceph orch device ls do not returns any HDD

Hi everyone

I'm new to CEPH, just a french 4 days training session with Octopus on VMs that convince me to build my first cluster.

At this time I have 4 old identical nodes for testing with 3 HDDs each, 2 network interfaces and running Alma Linux8 (el8). I try to replay the training session but it fails, breaking the web interface because of some problems with podman 4.2 not compatible with Octopus.

So I try to deploy Pacific with cephadm tool on my first node (mostha1) (to enable testing also an upgrade later).

    dnf -y install https://urldefense.com/v3/__https://download.ceph.com/rpm-16.2.13/el8/noarch... <https://urldefense.com/v3/__https:/download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noarch.rpm__;!!CQl3mcHX2A!H9cwNCJyKXYQ4BbGA3gwHHRitjOS4lBCZT9wlnBZ-8IDue0MvdcPD8Dnv5yQCZw_eA4BNDYaEq1eouKQcQO7HshgdUJ0SJ-EgLfaBGBmCQ$>

    monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }')     cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \                        --initial-dashboard-user admceph \                        --allow-fqdn-hostname --cluster-network 10.1.0.0/16

This was sucessfull.

But running "*c**eph orch device ls*" do not show any HDD even if I have /dev/sda (used by the OS), /dev/sdb and /dev/sdc

The web interface shows a row capacity which is an aggregate of the sizes of the 3 HDDs for the node.

I've also tried to reset /dev/sdb but cephadm do not see it:

    [ceph: root@mostha1 /]# ceph orch device zap     mostha1.legi.grenoble-inp.fr /dev/sdb --force     Error EINVAL: Device path '/dev/sdb' not found on host     'mostha1.legi.grenoble-inp.fr'

On my first attempt with octopus, I was able to list the available HDD with this command line. Before moving to Pacific, the OS on this node has been reinstalled from scratch.

Any advices for a CEPH beginner ?

Thanks

Patrick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

As you've already seem to have figured out, "ceph orch device ls" is populated with the results from "ceph-volume inventory". My best guess to try and debug this would be to manually run "cephadm ceph-volume -- inventory" (the same as "cephadm ceph-volume inventory", I just like to separate the ceph-volume command from cephadm itself with the " -- ") and then check /var/log/ceph/<fsid>/ceph-volume.log from when you ran the command onward to try and see why it isn't seeing your devices. For example I can see a line like [2023-05-15 19:11:58,048][ceph_volume.main][INFO ] Running command: ceph-volume inventory in there. Then if I look onward from there I can see it ran things like lsblk -P -o NAME,KNAME,PKNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL as part of getting my device list. So if I was having issues I would try running that directly and see what I got. Will note that ceph-volume on certain more recent versions (not sure about octopus) runs commands through nsenter, so you'd have to look past that part in the log lines to the underlying command being used, typically something with lsblk, blkid, udevadm, lvs, or pvs. Also, if you want to see if it's an issue with a certain version of ceph-volume, you can use different versions by passing the image flag to cephadm. E.g. cephadm --image quay.io/ceph/ceph:v17.2.6 ceph-volume -- inventory would use the 17.2.6 version of ceph-volume for the inventory. It works by running ceph-volume through the container, so you don't have to have to worry about installing different packages to try them and it should pull the container image on its own if it isn't on the machine already (but note that means the command will take longer as it pulls the image the first time). On Sat, May 13, 2023 at 4:34 AM Patrick Begou < Patrick.Begou@univ-grenoble-alpes.fr> wrote:

Hi Joshua,

I've tried these commands but it looks like CEPH is unable to see and configure these HDDs. [root@mostha1 ~]# cephadm ceph-volume inventory

Inferring fsid 4b7a6504-f0be-11ed-be1a-00266cf8869c Using recent ceph image

quay.io/ceph/ceph@sha256:e6919776f0ff8331a8e9c4b18d36c5e9eed31e1a80da62ae8454e42d10e95544

Device Path Size Device nodes rotates available Model name

[root@mostha1 ~]# cephadm shell

[ceph: root@mostha1 /]# ceph orch apply osd --all-available-devices

Scheduled osd.all-available-devices update...

[ceph: root@mostha1 /]# ceph orch device ls[ceph: root@mostha1 /]# ceph-volume lvm zap /dev/sdb

--> Zapping: /dev/sdb --> --destroy was not specified, but zapping a whole device will remove the partition table Running command: /usr/bin/dd if=/dev/zero of=/dev/sdb bs=1M count=10 conv=fsync stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.10039 s, 104 MB/s --> Zapping successful for: <Raw Device: /dev/sdb>

I can check that /dev/sdb1 has been erased, so previous command is successful [ceph: root@mostha1 ceph]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 1 232.9G 0 disk |-sda1 8:1 1 3.9G 0 part /rootfs/boot |-sda2 8:2 1 78.1G 0 part | `-osvg-rootvol 253:0 0 48.8G 0 lvm /rootfs |-sda3 8:3 1 3.9G 0 part [SWAP] `-sda4 8:4 1 146.9G 0 part |-secretvg-homevol 253:1 0 9.8G 0 lvm /rootfs/home |-secretvg-tmpvol 253:2 0 9.8G 0 lvm /rootfs/tmp `-secretvg-varvol 253:3 0 9.8G 0 lvm /rootfs/var sdb 8:16 1 465.8G 0 disk sdc 8:32 1 232.9G 0 disk

But still no visible HDD:

[ceph: root@mostha1 ceph]# ceph orch apply osd --all-available-devices

Scheduled osd.all-available-devices update...

[ceph: root@mostha1 ceph]# ceph orch device ls [ceph: root@mostha1 ceph]#

May be I have done something bad at install time as in the container I've unintentionally run:

dnf -y install

https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noarc...

(an awful copy/paste launching the command). Can this break The container ? I do not know what should be available as ceph packages in the container to remove properly this install (no dnf.log file in the container)

Patrick

Le 12/05/2023 à 21:38, Beaman, Joshua a écrit :

...

The most significant point I see there, is you have no OSD service spec to tell orchestrator how to deploy OSDs. The easiest fix for that would be “cephorchapplyosd--all-available-devices”

This will create a simple spec that should work for a test environment. Most likely it will collocate the block, block.db, and WAL all on the same device. Not ideal for prod environments, but fine for practice and testing.

The other command I should have had you try is “cephadm ceph-volume inventory”. That should show you the devices available for OSD deployment, and hopefully matches up to what your “lsblk” shows. If you need to zap HDDs and orchestrator is still not seeing them, you can try “cephadm ceph-volume lvm zap /dev/sdb”

Thank you,

Josh Beaman

*From: *Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> *Date: *Friday, May 12, 2023 at 2:22 PM *To: *Beaman, Joshua <Joshua_Beaman@comcast.com>, ceph-users <ceph-users@ceph.io> *Subject: *Re: [EXTERNAL] [ceph-users] [Pacific] ceph orch device ls do not returns any HDD

Hi Joshua and thanks for this quick reply.

At this step I have only one node. I was checking what ceph was returning with different commands on this host before adding new hosts. Just to compare with my first Octopus install. As this hardware is for testing only, it remains easy for me to break everything and reinstall again.

[root@mostha1 ~]# cephadm check-host

podman (/usr/bin/podman) version 4.2.0 is present systemctl is present lvcreate is present Unit chronyd.service is enabled and running Host looks OK

[ceph: root@mostha1 /]# ceph -s

cluster: id: 4b7a6504-f0be-11ed-be1a-00266cf8869c health: HEALTH_WARN OSD count 0 < osd_pool_default_size 3

services: mon: 1 daemons, quorum mostha1.legi.grenoble-inp.fr (age 5h) mgr: mostha1.legi.grenoble-inp.fr.hogwuz(active, since 5h) osd: 0 osds: 0 up, 0 in

data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs:

[ceph: root@mostha1 /]# ceph orch ls

NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 6m ago 6h count:1 crash 1/1 6m ago 6h * grafana ?:3000 1/1 6m ago 6h count:1 mgr 1/2 6m ago 6h count:2 mon 1/5 6m ago 6h count:5 node-exporter ?:9100 1/1 6m ago 6h * prometheus ?:9095 1/1 6m ago 6h count:1

[ceph: root@mostha1 /]# ceph orch ls osd -export

No services reported

[ceph: root@mostha1 /]# ceph orch host ls

HOST ADDR LABELS STATUS mostha1.legi.grenoble-inp.fr 194.254.66.34 _admin 1 hosts in cluster

[ceph: root@mostha1 /]# ceph log last cephadm

... 2023-05-12T15:19:58.754655+0000 mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1876 : cephadm [INF] Zap device mostha1.legi.grenoble-inp.fr:/dev/sdb 2023-05-12T15:19:58.756639+0000 mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1877 : cephadm [ERR] Device path '/dev/sdb' not found on host 'mostha1.legi.grenoble-inp.fr' Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper return OrchResult(f(*args, **kwargs)) File "/usr/share/ceph/mgr/cephadm/module.py", line 2275, in zap_device f"Device path '{path}' not found on host '{host}'") orchestrator._interface.OrchestratorError: Device path '/dev/sdb' not found on host 'mostha1.legi.grenoble-inp.fr' ....

[ceph: root@mostha1 /]# ls -l /dev/sdb

brw-rw---- 1 root disk 8, 16 May 12 15:16 /dev/sdb

[ceph: root@mostha1 /]# lsblk /dev/sdb

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdb 8:16 1 465.8G 0 disk `-sdb1 8:17 1 465.8G 0 part

I have crated a full partition on /dev/sdb (for testing) and /dev/sdc has no partition table (removed).

But all seams fine with these commands.

Patrick

Le 12/05/2023 à 20:19, Beaman, Joshua a écrit :

I don’t quite understand why that zap would not work. But, here’s where I’d start.

1. cephadm check-host

1. Run this on each of your hosts to make sure cephadm, podman and all other prerequisites are installed and recognized

2. ceph orch ls

1. This should show at least a mon, mgr, and osd spec deployed

3. ceph orch ls osd –export

1. This will show the OSD placement service specifications that orchestrator uses to identify devices to deploy as OSDs

4. ceph orch host ls

1. This will list the hosts that have been added to orchestrator’s inventory, and what labels are applied which correlate to the service placement labels

5. ceph log last cephadm

1. This will show you what orchestrator has been trying to do, and how it may be failing

Also, it’s never un-helpful to have a look at “ceph -s” and “ceph health detail”, particularly for any people trying to help you without access to your systems.

Best of luck,

Josh Beaman

*From: *Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> <mailto:Patrick.Begou@univ-grenoble-alpes.fr> *Date: *Friday, May 12, 2023 at 10:45 AM *To: *ceph-users <ceph-users@ceph.io> <mailto:ceph-users@ceph.io> *Subject: *[EXTERNAL] [ceph-users] [Pacific] ceph orch device ls do not returns any HDD

Hi everyone

I'm new to CEPH, just a french 4 days training session with Octopus on VMs that convince me to build my first cluster.

At this time I have 4 old identical nodes for testing with 3 HDDs each, 2 network interfaces and running Alma Linux8 (el8). I try to replay the training session but it fails, breaking the web interface because of some problems with podman 4.2 not compatible with Octopus.

So I try to deploy Pacific with cephadm tool on my first node (mostha1) (to enable testing also an upgrade later).

dnf -y install

https://urldefense.com/v3/__https://download.ceph.com/rpm-16.2.13/el8/noarch...

...

<

https://urldefense.com/v3/__https:/download.ceph.com/rpm-16.2.13/el8/noarch/...

...

monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }') cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \ --initial-dashboard-user admceph \ --allow-fqdn-hostname --cluster-network 10.1.0.0/16

This was sucessfull.

But running "*c**eph orch device ls*" do not show any HDD even if I have /dev/sda (used by the OS), /dev/sdb and /dev/sdc

The web interface shows a row capacity which is an aggregate of the sizes of the 3 HDDs for the node.

I've also tried to reset /dev/sdb but cephadm do not see it:

[ceph: root@mostha1 /]# ceph orch device zap mostha1.legi.grenoble-inp.fr /dev/sdb --force Error EINVAL: Device path '/dev/sdb' not found on host 'mostha1.legi.grenoble-inp.fr'

On my first attempt with octopus, I was able to list the available HDD with this command line. Before moving to Pacific, the OS on this node has been reinstalled from scratch.

Any advices for a CEPH beginner ?

Thanks

Patrick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Patrick, It is weird, we have a couple of clusters with cephadm and running pacify or quincy and ceph orch device works well. Have you looked at the cephadm logs (ceph log last cephadm)? Except if you are using a very specific hardware, I suspect Ceph is suffering of a problem outside it... Cheers, Michel Sent from my mobile Le 26 mai 2023 17:02:50 Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> a écrit :

Hi,

I'm back working on this problem.

First of all, I saw that I had a hardware memory error so I had to solve this first. It's done.

I've tested some different Ceph deployments, each time starting with a full OS re-install (it requires some time for each test).

Using Octopus, the devices are found:

dnf -y install \ https://download.ceph.com/rpm-15.2.12/el8/noarch/cephadm-15.2.12-0.el8.noarc... monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }')) cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \ --allow-fqdn-hostname

[ceph: root@mostha1 /]# *ceph orch device ls* Hostname Path Type Serial Size Health Ident Fault Available mostha1.legi.grenoble-inp.fr /dev/sda hdd S2B5J90ZA02494 250G Unknown N/A N/A Yes mostha1.legi.grenoble-inp.fr /dev/sdc hdd WD-WMAYP0982329 500G Unknown N/A N/A Yes

But with Pacific or Quincy the command returns nothing.

With Pacific:

dnf -y install \ https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noarc... monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }') cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \ --allow-fqdn-hostname

"ceph orch device ls" doesn't return anything but "cephadm shell lsmcli ldl" list all the devices.

[ceph: root@mostha1 /]# *ceph orch device ls --wide* [ceph: root@mostha1 /]# *lsblk* NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 1 232.9G 0 disk |-sda1 8:1 1 3.9G 0 part /rootfs/boot |-sda2 8:2 1 78.1G 0 part | `-osvg-rootvol 253:0 0 48.8G 0 lvm /rootfs |-sda3 8:3 1 3.9G 0 part [SWAP] `-sda4 8:4 1 146.9G 0 part |-secretvg-homevol 253:1 0 9.8G 0 lvm /rootfs/home |-secretvg-tmpvol 253:2 0 9.8G 0 lvm /rootfs/tmp `-secretvg-varvol 253:3 0 9.8G 0 lvm /rootfs/var sdb 8:16 1 232.9G 0 disk sdc 8:32 1 465.8G 0 disk [ceph: root@mostha1 /]# exit [root@mostha1 ~]# *cephadm ceph-volume inventory* Inferring fsid 2e3e85a8-fbcf-11ed-84e5-00266cf8869c Using ceph image with id '0dc91bca92c2' and tag 'v17' created on 2023-05-25 16:26:31 +0000 UTC quay.io/ceph/ceph@sha256:b8df01a568f4dec7bac6d5040f9391dcca14e00ec7f4de8a3dcf3f2a6502d3a9

Device Path Size Device nodes rotates available Model name

[root@mostha1 ~]# *cephadm shell lsmcli ldl* Inferring fsid 4d54823c-fb05-11ed-aecf-00266cf8869c Inferring config /var/lib/ceph/4d54823c-fb05-11ed-aecf-00266cf8869c/mon.mostha1/config Using ceph image with id 'c9a1062f7289' and tag 'v17' created on 2023-04-25 16:04:33 +0000 UTC quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e Path | SCSI VPD 0x83 | Link Type | Serial Number | Health Status ------------------------------------------------------------------------- */dev/sda | 50024e92039e4f1c | PATA/SATA | S2B5J90ZA10142 | Good** **/dev/sdc | 50014ee0ad5953c9 | PATA/SATA | WD-WMAYP0982329 | Good** **/dev/sdb | 50024e920387fa2c | PATA/SATA | S2B5J90ZA02494 | Good** *

Could it be a bug in ceph-volume ? Adam suggest looking to the underlying commands (lsblk, blkid, udevadm, lvs, or pvs) but I'm not very comfortable with blkid and udevadm. Is there a "debug flag" to set ceph more verbose ?

Thanks

Patrick

Le 15/05/2023 à 21:20, Adam King a écrit :

...

As you've already seem to have figured out, "ceph orch device ls" is populated with the results from "ceph-volume inventory". My best guess to try and debug this would be to manually run "cephadm ceph-volume -- inventory" (the same as "cephadm ceph-volume inventory", I just like to separate the ceph-volume command from cephadm itself with the " -- ") and then check /var/log/ceph/<fsid>/ceph-volume.log from when you ran the command onward to try and see why it isn't seeing your devices. For example I can see a line like

[2023-05-15 19:11:58,048][ceph_volume.main][INFO ] Running command: ceph-volume inventory

in there. Then if I look onward from there I can see it ran things like

lsblk -P -o NAME,KNAME,PKNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL

as part of getting my device list. So if I was having issues I would try running that directly and see what I got. Will note that ceph-volume on certain more recent versions (not sure about octopus) runs commands through nsenter, so you'd have to look past that part in the log lines to the underlying command being used, typically something with lsblk, blkid, udevadm, lvs, or pvs.

Also, if you want to see if it's an issue with a certain version of ceph-volume, you can use different versions by passing the image flag to cephadm. E.g.

cephadm --image quay.io/ceph/ceph:v17.2.6 <http://quay.io/ceph/ceph:v17.2.6> ceph-volume -- inventory

would use the 17.2.6 version of ceph-volume for the inventory. It works by running ceph-volume through the container, so you don't have to have to worry about installing different packages to try them and it should pull the container image on its own if it isn't on the machine already (but note that means the command will take longer as it pulls the image the first time).

On Sat, May 13, 2023 at 4:34 AM Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> wrote:

Hi Joshua,

I've tried these commands but it looks like CEPH is unable to see and configure these HDDs. [root@mostha1 ~]# cephadm ceph-volume inventory

Inferring fsid 4b7a6504-f0be-11ed-be1a-00266cf8869c Using recent ceph image quay.io/ceph/ceph@sha256:e6919776f0ff8331a8e9c4b18d36c5e9eed31e1a80da62ae8454e42d10e95544 <http://quay.io/ceph/ceph@sha256:e6919776f0ff8331a8e9c4b18d36c5e9eed31e1a80da62ae8454e42d10e95544>

Device Path Size Device nodes rotates available Model name

[root@mostha1 ~]# cephadm shell

[ceph: root@mostha1 /]# ceph orch apply osd --all-available-devices

Scheduled osd.all-available-devices update...

[ceph: root@mostha1 /]# ceph orch device ls[ceph: root@mostha1 /]# ceph-volume lvm zap /dev/sdb

--> Zapping: /dev/sdb --> --destroy was not specified, but zapping a whole device will remove the partition table Running command: /usr/bin/dd if=/dev/zero of=/dev/sdb bs=1M count=10 conv=fsync stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.10039 s, 104 MB/s --> Zapping successful for: <Raw Device: /dev/sdb>

I can check that /dev/sdb1 has been erased, so previous command is successful [ceph: root@mostha1 ceph]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 1 232.9G 0 disk |-sda1 8:1 1 3.9G 0 part /rootfs/boot |-sda2 8:2 1 78.1G 0 part | `-osvg-rootvol 253:0 0 48.8G 0 lvm /rootfs |-sda3 8:3 1 3.9G 0 part [SWAP] `-sda4 8:4 1 146.9G 0 part |-secretvg-homevol 253:1 0 9.8G 0 lvm /rootfs/home |-secretvg-tmpvol 253:2 0 9.8G 0 lvm /rootfs/tmp `-secretvg-varvol 253:3 0 9.8G 0 lvm /rootfs/var sdb 8:16 1 465.8G 0 disk sdc 8:32 1 232.9G 0 disk

But still no visible HDD:

[ceph: root@mostha1 ceph]# ceph orch apply osd --all-available-devices

Scheduled osd.all-available-devices update...

[ceph: root@mostha1 ceph]# ceph orch device ls [ceph: root@mostha1 ceph]#

May be I have done something bad at install time as in the container I've unintentionally run:

dnf -y install https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noarc...

(an awful copy/paste launching the command). Can this break The container ? I do not know what should be available as ceph packages in the container to remove properly this install (no dnf.log file in the container)

Patrick

Le 12/05/2023 à 21:38, Beaman, Joshua a écrit : > The most significant point I see there, is you have no OSD service > spec to tell orchestrator how to deploy OSDs. The easiest fix for > that would be “cephorchapplyosd--all-available-devices” > > This will create a simple spec that should work for a test > environment. Most likely it will collocate the block, block.db, and > WAL all on the same device. Not ideal for prod environments, but fine > for practice and testing. > > The other command I should have had you try is “cephadm ceph-volume > inventory”. That should show you the devices available for OSD > deployment, and hopefully matches up to what your “lsblk” shows. If > you need to zap HDDs and orchestrator is still not seeing them, you > can try “cephadm ceph-volume lvm zap /dev/sdb” > > Thank you, > > Josh Beaman > > *From: *Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> > *Date: *Friday, May 12, 2023 at 2:22 PM > *To: *Beaman, Joshua <Joshua_Beaman@comcast.com>, ceph-users > <ceph-users@ceph.io> > *Subject: *Re: [EXTERNAL] [ceph-users] [Pacific] ceph orch device ls > do not returns any HDD > > Hi Joshua and thanks for this quick reply. > > At this step I have only one node. I was checking what ceph was > returning with different commands on this host before adding new > hosts. Just to compare with my first Octopus install. As this hardware > is for testing only, it remains easy for me to break everything and > reinstall again. > > [root@mostha1 ~]# cephadm check-host > > podman (/usr/bin/podman) version 4.2.0 is present > systemctl is present > lvcreate is present > Unit chronyd.service is enabled and running > Host looks OK > > [ceph: root@mostha1 /]# ceph -s > > cluster: > id: 4b7a6504-f0be-11ed-be1a-00266cf8869c > health: HEALTH_WARN > OSD count 0 < osd_pool_default_size 3 > > services: > mon: 1 daemons, quorum mostha1.legi.grenoble-inp.fr <http://mostha1.legi.grenoble-inp.fr> (age 5h) > mgr: mostha1.legi.grenoble-inp.fr <http://mostha1.legi.grenoble-inp.fr>.hogwuz(active, since 5h) > osd: 0 osds: 0 up, 0 in > > data: > pools: 0 pools, 0 pgs > objects: 0 objects, 0 B > usage: 0 B used, 0 B / 0 B avail > pgs: > > [ceph: root@mostha1 /]# ceph orch ls > > NAME PORTS RUNNING REFRESHED AGE PLACEMENT > alertmanager ?:9093,9094 1/1 6m ago 6h count:1 > crash 1/1 6m ago 6h * > grafana ?:3000 1/1 6m ago 6h count:1 > mgr 1/2 6m ago 6h count:2 > mon 1/5 6m ago 6h count:5 > node-exporter ?:9100 1/1 6m ago 6h * > prometheus ?:9095 1/1 6m ago 6h count:1 > > [ceph: root@mostha1 /]# ceph orch ls osd -export > > No services reported > > [ceph: root@mostha1 /]# ceph orch host ls > > HOST ADDR LABELS STATUS > mostha1.legi.grenoble-inp.fr <http://mostha1.legi.grenoble-inp.fr> 194.254.66.34 _admin > 1 hosts in cluster > > [ceph: root@mostha1 /]# ceph log last cephadm > > ... > 2023-05-12T15:19:58.754655+0000 > mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1876 : cephadm > [INF] Zap device mostha1.legi.grenoble-inp.fr:/dev/sdb > 2023-05-12T15:19:58.756639+0000 > mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1877 : cephadm > [ERR] Device path '/dev/sdb' not found on host > 'mostha1.legi.grenoble-inp.fr <http://mostha1.legi.grenoble-inp.fr>' > Traceback (most recent call last): > File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, > in wrapper > return OrchResult(f(*args, **kwargs)) > File "/usr/share/ceph/mgr/cephadm/module.py", line 2275, in > zap_device > f"Device path '{path}' not found on host '{host}'") > orchestrator._interface.OrchestratorError: Device path '/dev/sdb' > not found on host 'mostha1.legi.grenoble-inp.fr <http://mostha1.legi.grenoble-inp.fr>' > .... > > [ceph: root@mostha1 /]# ls -l /dev/sdb > > brw-rw---- 1 root disk 8, 16 May 12 15:16 /dev/sdb > > [ceph: root@mostha1 /]# lsblk /dev/sdb > > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > sdb 8:16 1 465.8G 0 disk > `-sdb1 8:17 1 465.8G 0 part > > I have crated a full partition on /dev/sdb (for testing) and /dev/sdc > has no partition table (removed). > > But all seams fine with these commands. > > Patrick > > Le 12/05/2023 à 20:19, Beaman, Joshua a écrit : > > I don’t quite understand why that zap would not work. But, here’s > where I’d start. > > 1. cephadm check-host > > 1. Run this on each of your hosts to make sure cephadm, > podman and all other prerequisites are installed and > recognized > > 2. ceph orch ls > > 1. This should show at least a mon, mgr, and osd spec deployed > > 3. ceph orch ls osd –export > > 1. This will show the OSD placement service specifications > that orchestrator uses to identify devices to deploy as OSDs > > 4. ceph orch host ls > > 1. This will list the hosts that have been added to > orchestrator’s inventory, and what labels are applied > which correlate to the service placement labels > > 5. ceph log last cephadm > > 1. This will show you what orchestrator has been trying to > do, and how it may be failing > > Also, it’s never un-helpful to have a look at “ceph -s” and “ceph > health detail”, particularly for any people trying to help you > without access to your systems. > > Best of luck, > > Josh Beaman > > *From: *Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> > <mailto:Patrick.Begou@univ-grenoble-alpes.fr> > *Date: *Friday, May 12, 2023 at 10:45 AM > *To: *ceph-users <ceph-users@ceph.io> <mailto:ceph-users@ceph.io> > *Subject: *[EXTERNAL] [ceph-users] [Pacific] ceph orch device ls > do not returns any HDD > > Hi everyone > > I'm new to CEPH, just a french 4 days training session with > Octopus on > VMs that convince me to build my first cluster. > > At this time I have 4 old identical nodes for testing with 3 HDDs > each, > 2 network interfaces and running Alma Linux8 (el8). I try to > replay the > training session but it fails, breaking the web interface because of > some problems with podman 4.2 not compatible with Octopus. > > So I try to deploy Pacific with cephadm tool on my first node > (mostha1) > (to enable testing also an upgrade later). > > dnf -y install > https://urldefense.com/v3/__https://download.ceph.com/rpm-16.2.13/el8/noarch... > <https://urldefense.com/v3/__https:/download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noarch.rpm__;!!CQl3mcHX2A!H9cwNCJyKXYQ4BbGA3gwHHRitjOS4lBCZT9wlnBZ-8IDue0MvdcPD8Dnv5yQCZw_eA4BNDYaEq1eouKQcQO7HshgdUJ0SJ-EgLfaBGBmCQ$> > > > monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }') > cephadm bootstrap --mon-ip $monip --initial-dashboard-password > xxxxx \ > --initial-dashboard-user admceph \ > --allow-fqdn-hostname --cluster-network > 10.1.0.0/16 <http://10.1.0.0/16> > > This was sucessfull. > > But running "*c**eph orch device ls*" do not show any HDD even if > I have > /dev/sda (used by the OS), /dev/sdb and /dev/sdc > > The web interface shows a row capacity which is an aggregate of the > sizes of the 3 HDDs for the node. > > I've also tried to reset /dev/sdb but cephadm do not see it: > > [ceph: root@mostha1 /]# ceph orch device zap > mostha1.legi.grenoble-inp.fr <http://mostha1.legi.grenoble-inp.fr> /dev/sdb --force > Error EINVAL: Device path '/dev/sdb' not found on host > 'mostha1.legi.grenoble-inp.fr <http://mostha1.legi.grenoble-inp.fr>' > > On my first attempt with octopus, I was able to list the available > HDD > with this command line. Before moving to Pacific, the OS on this node > has been reinstalled from scratch. > > Any advices for a CEPH beginner ? > > Thanks > > Patrick > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Patrick, I can only say that I would not expect a specific problem due to your hardware. Upgrading the firmware is generally a good idea but I wouldn't expect it helps in your case if the osk (lsblk) sees the disk. As for starting with octopus I don't know if it will help... But we are also using the same os as you (centos stream in fact but basically the same). We have been running octopus (with cephadm) on this os version without problem and upgraded since then in pacific and quincy. In fact on of our cluster started in infernetis, the other in luminous and they have been upgraded without problems since then... Michel Sent from my mobile Le 26 mai 2023 18:34:22 Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> a écrit :

Hi Michel, I do not notice anything strange in the logs files (looking for errors or warnings). The hardware is a DELL C6100 sled (from 2011) running Alma Linux8 up-to-date. It uses 3 sata disks. Is there a way to force osd installation by hand with providing the device /dev/sdc for example ? A "do what I say" approach... Is it a good try to deploy Octopus on the nodes, configure the osd (even if podman 4.2.0 is not validated for Octopus) and then upgrade to Pacific? Could this be a workaround for this sort of regression from Octopus to Pacific ? May be updating the BIOS from 1.7.1 to 1.8.1 ?

All this is a little bit confusing for me as I'm trying to discover Ceph 😁 Thanks Patrick

Le 26/05/2023 à 17:19, Michel Jouvin a écrit :

...

Hi Patrick,

It is weird, we have a couple of clusters with cephadm and running pacify or quincy and ceph orch device works well. Have you looked at the cephadm logs (ceph log last cephadm)?

Except if you are using a very specific hardware, I suspect Ceph is suffering of a problem outside it...

Cheers,

Michel Sent from my mobile

Le 26 mai 2023 17:02:50 Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> a écrit :

...

Hi,

I'm back working on this problem.

First of all, I saw that I had a hardware memory error so I had to solve this first. It's done.

I've tested some different Ceph deployments, each time starting with a full OS re-install (it requires some time for each test).

Using Octopus, the devices are found:

dnf -y install \ https://download.ceph.com/rpm-15.2.12/el8/noarch/cephadm-15.2.12-0.el8.noarc... monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }')) cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \ --allow-fqdn-hostname

[ceph: root@mostha1 /]# *ceph orch device ls* Hostname Path Type Serial Size Health Ident Fault Available mostha1.legi.grenoble-inp.fr /dev/sda hdd S2B5J90ZA02494 250G Unknown N/A N/A Yes mostha1.legi.grenoble-inp.fr /dev/sdc hdd WD-WMAYP0982329 500G Unknown N/A N/A Yes

But with Pacific or Quincy the command returns nothing.

With Pacific:

dnf -y install \ https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noarc... monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }') cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \ --allow-fqdn-hostname

"ceph orch device ls" doesn't return anything but "cephadm shell lsmcli ldl" list all the devices.

[ceph: root@mostha1 /]# *ceph orch device ls --wide* [ceph: root@mostha1 /]# *lsblk* NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 1 232.9G 0 disk |-sda1 8:1 1 3.9G 0 part /rootfs/boot |-sda2 8:2 1 78.1G 0 part | `-osvg-rootvol 253:0 0 48.8G 0 lvm /rootfs |-sda3 8:3 1 3.9G 0 part [SWAP] `-sda4 8:4 1 146.9G 0 part |-secretvg-homevol 253:1 0 9.8G 0 lvm /rootfs/home |-secretvg-tmpvol 253:2 0 9.8G 0 lvm /rootfs/tmp `-secretvg-varvol 253:3 0 9.8G 0 lvm /rootfs/var sdb 8:16 1 232.9G 0 disk sdc 8:32 1 465.8G 0 disk [ceph: root@mostha1 /]# exit [root@mostha1 ~]# *cephadm ceph-volume inventory* Inferring fsid 2e3e85a8-fbcf-11ed-84e5-00266cf8869c Using ceph image with id '0dc91bca92c2' and tag 'v17' created on 2023-05-25 16:26:31 +0000 UTC quay.io/ceph/ceph@sha256:b8df01a568f4dec7bac6d5040f9391dcca14e00ec7f4de8a3dcf3f2a6502d3a9

Device Path Size Device nodes rotates available Model name

[root@mostha1 ~]# *cephadm shell lsmcli ldl* Inferring fsid 4d54823c-fb05-11ed-aecf-00266cf8869c Inferring config /var/lib/ceph/4d54823c-fb05-11ed-aecf-00266cf8869c/mon.mostha1/config Using ceph image with id 'c9a1062f7289' and tag 'v17' created on 2023-04-25 16:04:33 +0000 UTC quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e Path | SCSI VPD 0x83 | Link Type | Serial Number | Health Status ------------------------------------------------------------------------- */dev/sda | 50024e92039e4f1c | PATA/SATA | S2B5J90ZA10142 | Good** **/dev/sdc | 50014ee0ad5953c9 | PATA/SATA | WD-WMAYP0982329 | Good** **/dev/sdb | 50024e920387fa2c | PATA/SATA | S2B5J90ZA02494 | Good** *

Could it be a bug in ceph-volume ? Adam suggest looking to the underlying commands (lsblk, blkid, udevadm, lvs, or pvs) but I'm not very comfortable with blkid and udevadm. Is there a "debug flag" to set ceph more verbose ?

Thanks

Patrick

Le 15/05/2023 à 21:20, Adam King a écrit :

...

As you've already seem to have figured out, "ceph orch device ls" is populated with the results from "ceph-volume inventory". My best guess to try and debug this would be to manually run "cephadm ceph-volume -- inventory" (the same as "cephadm ceph-volume inventory", I just like to separate the ceph-volume command from cephadm itself with the " -- ") and then check /var/log/ceph/<fsid>/ceph-volume.log from when you ran the command onward to try and see why it isn't seeing your devices. For example I can see a line like

[2023-05-15 19:11:58,048][ceph_volume.main][INFO ] Running command: ceph-volume inventory

in there. Then if I look onward from there I can see it ran things like

lsblk -P -o NAME,KNAME,PKNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL

as part of getting my device list. So if I was having issues I would try running that directly and see what I got. Will note that ceph-volume on certain more recent versions (not sure about octopus) runs commands through nsenter, so you'd have to look past that part in the log lines to the underlying command being used, typically something with lsblk, blkid, udevadm, lvs, or pvs.

Also, if you want to see if it's an issue with a certain version of ceph-volume, you can use different versions by passing the image flag to cephadm. E.g.

cephadm --image quay.io/ceph/ceph:v17.2.6 <http://quay.io/ceph/ceph:v17.2.6> ceph-volume -- inventory

would use the 17.2.6 version of ceph-volume for the inventory. It works by running ceph-volume through the container, so you don't have to have to worry about installing different packages to try them and it should pull the container image on its own if it isn't on the machine already (but note that means the command will take longer as it pulls the image the first time).

On Sat, May 13, 2023 at 4:34 AM Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> wrote:

Hi Joshua,

I've tried these commands but it looks like CEPH is unable to see and configure these HDDs. [root@mostha1 ~]# cephadm ceph-volume inventory

Inferring fsid 4b7a6504-f0be-11ed-be1a-00266cf8869c Using recent ceph image quay.io/ceph/ceph@sha256:e6919776f0ff8331a8e9c4b18d36c5e9eed31e1a80da62ae8454e42d10e95544 <http://quay.io/ceph/ceph@sha256:e6919776f0ff8331a8e9c4b18d36c5e9eed31e1a80da62ae8454e42d10e95544>

Device Path Size Device nodes rotates available Model name

[root@mostha1 ~]# cephadm shell

[ceph: root@mostha1 /]# ceph orch apply osd --all-available-devices

Scheduled osd.all-available-devices update...

[ceph: root@mostha1 /]# ceph orch device ls[ceph: root@mostha1 /]# ceph-volume lvm zap /dev/sdb

--> Zapping: /dev/sdb --> --destroy was not specified, but zapping a whole device will remove the partition table Running command: /usr/bin/dd if=/dev/zero of=/dev/sdb bs=1M count=10 conv=fsync stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.10039 s, 104 MB/s --> Zapping successful for: <Raw Device: /dev/sdb>

I can check that /dev/sdb1 has been erased, so previous command is successful [ceph: root@mostha1 ceph]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 1 232.9G 0 disk |-sda1 8:1 1 3.9G 0 part /rootfs/boot |-sda2 8:2 1 78.1G 0 part | `-osvg-rootvol 253:0 0 48.8G 0 lvm /rootfs |-sda3 8:3 1 3.9G 0 part [SWAP] `-sda4 8:4 1 146.9G 0 part |-secretvg-homevol 253:1 0 9.8G 0 lvm /rootfs/home |-secretvg-tmpvol 253:2 0 9.8G 0 lvm /rootfs/tmp `-secretvg-varvol 253:3 0 9.8G 0 lvm /rootfs/var sdb 8:16 1 465.8G 0 disk sdc 8:32 1 232.9G 0 disk

But still no visible HDD:

[ceph: root@mostha1 ceph]# ceph orch apply osd --all-available-devices

Scheduled osd.all-available-devices update...

[ceph: root@mostha1 ceph]# ceph orch device ls [ceph: root@mostha1 ceph]#

May be I have done something bad at install time as in the container I've unintentionally run:

dnf -y install https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noarc...

(an awful copy/paste launching the command). Can this break The container ? I do not know what should be available as ceph packages in the container to remove properly this install (no dnf.log file in the container)

Patrick

...

The most significant point I see there, is you have no OSD service spec to tell orchestrator how to deploy OSDs. The easiest fix for that would be “cephorchapplyosd--all-available-devices”

This will create a simple spec that should work for a test environment. Most likely it will collocate the block, block.db, and WAL all on the same device. Not ideal for prod environments, but fine for practice and testing.

The other command I should have had you try is “cephadm ceph-volume inventory”. That should show you the devices available for OSD deployment, and hopefully matches up to what your “lsblk” shows. If you need to zap HDDs and orchestrator is still not seeing them, you can try “cephadm ceph-volume lvm zap /dev/sdb”

Thank you,

Josh Beaman

*From: *Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> *Date: *Friday, May 12, 2023 at 2:22 PM *To: *Beaman, Joshua <Joshua_Beaman@comcast.com>, ceph-users <ceph-users@ceph.io> *Subject: *Re: [EXTERNAL] [ceph-users] [Pacific] ceph orch device ls do not returns any HDD

Hi Joshua and thanks for this quick reply.

At this step I have only one node. I was checking what ceph was returning with different commands on this host before adding new hosts. Just to compare with my first Octopus install. As this hardware is for testing only, it remains easy for me to break everything and reinstall again.

[root@mostha1 ~]# cephadm check-host

podman (/usr/bin/podman) version 4.2.0 is present systemctl is present lvcreate is present Unit chronyd.service is enabled and running Host looks OK

[ceph: root@mostha1 /]# ceph -s

cluster: id: 4b7a6504-f0be-11ed-be1a-00266cf8869c health: HEALTH_WARN OSD count 0 < osd_pool_default_size 3

services: mon: 1 daemons, quorum mostha1.legi.grenoble-inp.fr <http://mostha1.legi.grenoble-inp.fr> (age 5h) mgr: mostha1.legi.grenoble-inp.fr <http://mostha1.legi.grenoble-inp.fr>.hogwuz(active, since 5h) osd: 0 osds: 0 up, 0 in

data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs:

[ceph: root@mostha1 /]# ceph orch ls

NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 6m ago 6h count:1 crash 1/1 6m ago 6h * grafana ?:3000 1/1 6m ago 6h count:1 mgr 1/2 6m ago 6h count:2 mon 1/5 6m ago 6h count:5 node-exporter ?:9100 1/1 6m ago 6h * prometheus ?:9095 1/1 6m ago 6h count:1

[ceph: root@mostha1 /]# ceph orch ls osd -export

No services reported

[ceph: root@mostha1 /]# ceph orch host ls

HOST ADDR LABELS STATUS mostha1.legi.grenoble-inp.fr <http://mostha1.legi.grenoble-inp.fr> 194.254.66.34 _admin 1 hosts in cluster

[ceph: root@mostha1 /]# ceph log last cephadm

... 2023-05-12T15:19:58.754655+0000 mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1876 : cephadm [INF] Zap device mostha1.legi.grenoble-inp.fr:/dev/sdb 2023-05-12T15:19:58.756639+0000 mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1877 : cephadm [ERR] Device path '/dev/sdb' not found on host 'mostha1.legi.grenoble-inp.fr <http://mostha1.legi.grenoble-inp.fr>' Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py",

Le 12/05/2023 à 21:38, Beaman, Joshua a écrit : line 125,

...

in wrapper return OrchResult(f(*args, **kwargs)) File "/usr/share/ceph/mgr/cephadm/module.py", line 2275, in zap_device f"Device path '{path}' not found on host '{host}'") orchestrator._interface.OrchestratorError: Device path '/dev/sdb' not found on host 'mostha1.legi.grenoble-inp.fr <http://mostha1.legi.grenoble-inp.fr>' ....

[ceph: root@mostha1 /]# ls -l /dev/sdb

brw-rw---- 1 root disk 8, 16 May 12 15:16 /dev/sdb

[ceph: root@mostha1 /]# lsblk /dev/sdb

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdb 8:16 1 465.8G 0 disk `-sdb1 8:17 1 465.8G 0 part

I have crated a full partition on /dev/sdb (for testing) and /dev/sdc has no partition table (removed).

But all seams fine with these commands.

Patrick

Le 12/05/2023 à 20:19, Beaman, Joshua a écrit :

I don’t quite understand why that zap would not work. But, here’s where I’d start.

1. cephadm check-host

1. Run this on each of your hosts to make sure cephadm, podman and all other prerequisites are installed and recognized

2. ceph orch ls

1. This should show at least a mon, mgr, and osd spec deployed

3. ceph orch ls osd –export

1. This will show the OSD placement service specifications that orchestrator uses to identify devices to deploy as OSDs

4. ceph orch host ls

1. This will list the hosts that have been added to orchestrator’s inventory, and what labels are applied which correlate to the service placement labels

5. ceph log last cephadm

1. This will show you what orchestrator has been trying to do, and how it may be failing

Also, it’s never un-helpful to have a look at “ceph -s” and “ceph health detail”, particularly for any people trying to help you without access to your systems.

Best of luck,

Josh Beaman

*From: *Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr> <mailto:Patrick.Begou@univ-grenoble-alpes.fr> *Date: *Friday, May 12, 2023 at 10:45 AM *To: *ceph-users <ceph-users@ceph.io> <mailto:ceph-users@ceph.io> *Subject: *[EXTERNAL] [ceph-users] [Pacific] ceph orch device ls do not returns any HDD

Hi everyone

I'm new to CEPH, just a french 4 days training session with Octopus on VMs that convince me to build my first cluster.

At this time I have 4 old identical nodes for testing with 3 HDDs each, 2 network interfaces and running Alma Linux8 (el8). I try to replay the training session but it fails, breaking the web interface because of some problems with podman 4.2 not compatible with Octopus.

So I try to deploy Pacific with cephadm tool on my first node (mostha1) (to enable testing also an upgrade later).

dnf -y install https://urldefense.com/v3/__https://download.ceph.com/rpm-16.2.13/el8/noarch...

<https://urldefense.com/v3/__https:/download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noarch.rpm__;!!CQl3mcHX2A!H9cwNCJyKXYQ4BbGA3gwHHRitjOS4lBCZT9wlnBZ-8IDue0MvdcPD8Dnv5yQCZw_eA4BNDYaEq1eouKQcQO7HshgdUJ0SJ-EgLfaBGBmCQ$>

...

monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print

...

cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \ --initial-dashboard-user admceph \ --allow-fqdn-hostname --cluster-network 10.1.0.0/16 <http://10.1.0.0/16>

This was sucessfull.

But running "*c**eph orch device ls*" do not show any HDD even if I have /dev/sda (used by the OS), /dev/sdb and /dev/sdc

The web interface shows a row capacity which is an aggregate of the sizes of the 3 HDDs for the node.

I've also tried to reset /dev/sdb but cephadm do not see it:

[ceph: root@mostha1 /]# ceph orch device zap mostha1.legi.grenoble-inp.fr <http://mostha1.legi.grenoble-inp.fr> /dev/sdb --force Error EINVAL: Device path '/dev/sdb' not found on host 'mostha1.legi.grenoble-inp.fr <http://mostha1.legi.grenoble-inp.fr>'

On my first attempt with octopus, I was able to list the available HDD with this command line. Before moving to Pacific, the OS on

$1 }') this node

...

has been reinstalled from scratch.

Any advices for a CEPH beginner ?

Thanks

Patrick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi all, still stuck with this problem. I've deployed octopus and all my HDD have been setup as osd. Fine. I've upgraded to pacific and 2 osd have failed. They have been automatically removed and upgrade finishes. Cluster Health is finaly OK, no data loss. But now I cannot re-add these osd with pacific (I had previous troubles on these old HDDs, lost one osd in octopus and was able to reset and re-add it). I've tried manually to add the first osd on the node where it is located, following https://docs.ceph.com/en/pacific/rados/operations/bluestore-migration/ (not sure it's the best idea...) but it fails too. This node was the one used for deploying the cluster. [ceph: root@mostha1 /]# *ceph-volume lvm zap /dev/sdc* --> Zapping: /dev/sdc --> --destroy was not specified, but zapping a whole device will remove the partition table Running command: /usr/bin/dd if=/dev/zero of=/dev/sdc bs=1M count=10 conv=fsync  stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.663425 s, 15.8 MB/s --> Zapping successful for: <Raw Device: /dev/sdc> [ceph: root@mostha1 /]# *ceph-volume lvm create --bluestore --data /dev/sdc* Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 9f1eb8ee-41e6-4350-ad73-1be21234ec7c  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 AuthRegistry(0x7fb4e405c4d8) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 AuthRegistry(0x7fb4e40601d0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 AuthRegistry(0x7fb4eb8bee90) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx  stderr: 2023-10-02T16:09:29.858+0000 7fb4e965c700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]  stderr: 2023-10-02T16:09:29.858+0000 7fb4e9e5d700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]  stderr: 2023-10-02T16:09:29.858+0000 7fb4e8e5b700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]  stderr: 2023-10-02T16:09:29.858+0000 7fb4eb8c0700 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication  stderr: [errno 13] RADOS permission denied (error connecting to the cluster) -->  RuntimeError: Unable to create a new OSD id Any idea of what is wrong ? Thanks Patrick

Hi, just wondering if 'ceph-volume lvm zap --destroy /dev/sdc' would help here. From your previous output you didn't specify the --destroy flag. Which cephadm version is installed on the host? Did you also upgrade the OS when moving to Pacific? (Sorry if I missed that. Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

Le 02/10/2023 à 18:22, Patrick Bégou a écrit :

...

Hi all,

still stuck with this problem.

I've deployed octopus and all my HDD have been setup as osd. Fine. I've upgraded to pacific and 2 osd have failed. They have been automatically removed and upgrade finishes. Cluster Health is finaly OK, no data loss.

But now I cannot re-add these osd with pacific (I had previous troubles on these old HDDs, lost one osd in octopus and was able to reset and re-add it).

I've tried manually to add the first osd on the node where it is located, following https://docs.ceph.com/en/pacific/rados/operations/bluestore-migration/ (not sure it's the best idea...) but it fails too. This node was the one used for deploying the cluster.

[ceph: root@mostha1 /]# *ceph-volume lvm zap /dev/sdc* --> Zapping: /dev/sdc --> --destroy was not specified, but zapping a whole device will remove the partition table Running command: /usr/bin/dd if=/dev/zero of=/dev/sdc bs=1M count=10 conv=fsync  stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.663425 s, 15.8 MB/s --> Zapping successful for: <Raw Device: /dev/sdc>

[ceph: root@mostha1 /]# *ceph-volume lvm create --bluestore --data /dev/sdc* Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 9f1eb8ee-41e6-4350-ad73-1be21234ec7c  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 AuthRegistry(0x7fb4e405c4d8) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 AuthRegistry(0x7fb4e40601d0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 AuthRegistry(0x7fb4eb8bee90) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx  stderr: 2023-10-02T16:09:29.858+0000 7fb4e965c700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]  stderr: 2023-10-02T16:09:29.858+0000 7fb4e9e5d700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]  stderr: 2023-10-02T16:09:29.858+0000 7fb4e8e5b700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]  stderr: 2023-10-02T16:09:29.858+0000 7fb4eb8c0700 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication  stderr: [errno 13] RADOS permission denied (error connecting to the cluster) -->  RuntimeError: Unable to create a new OSD id

Any idea of what is wrong ?

Thanks

Patrick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

I'm still trying to understand what can be wrong or how to debug this situation where Ceph cannot see the devices.

The device :dev/sdc exists:

[root@mostha1 ~]# cephadm shell lsmcli ldl Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c Using recent ceph image

quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e Path     | SCSI VPD 0x83    | Link Type | Serial Number   | Health Status ------------------------------------------------------------------------- /dev/sda | 50024e92039e4f1c | PATA/SATA | S2B5J90ZA10142  | Good /dev/sdb | 50014ee0ad5953c9 | PATA/SATA | WD-WMAYP0982329 | Good /dev/sdc | 50024e920387fa2c | PATA/SATA | S2B5J90ZA02494  | Good

But I cannot do anything with it:

[root@mostha1 ~]# cephadm shell ceph orch device zap mostha1.legi.grenoble-inp.fr /dev/sdc --force Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c Using recent ceph image

quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e Error EINVAL: Device path '/dev/sdc' not found on host 'mostha1.legi.grenoble-inp.fr'

Since I moved from octopus to Pacific.

Patrick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Don't use ceph-volume manually to deploy OSDs if your cluster is managed by cephadm. I just wanted to point out that you hadn't wiped the disk properly to be able to re-use it. Let the orchestrator handle the OSD creation and activation. I recommend to remove the OSD again, wipe it properly (cephadm ceph-volume lvm zap --destroy /dev/sdc) and then let the orchestrator add it as an OSD. Depending on your drivegroup configuration it will happen automatically (if "all-available-devices" is enabled or your osd specs are already applied). If it doesn't happen automatically, deploy it with 'ceph orch daemon add osd *<host>*:*<device-path>*' [1]. [1] https://docs.ceph.com/en/quincy/cephadm/services/osd/#deploy-osds Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

Hi Eugen,

- the OS is Alma Linux 8 with latests updates.

- this morning I've worked with ceph-volume but it ends with a strange final state. I was connected on host mostha1 where /dev/sdc was not reconized. These are the steps followed based on the ceph-volume documentation I've read:

*[root@mostha1 ~]# cephadm shell** **[ceph: root@mostha1 /]# ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring** **[ceph: root@mostha1 /]# ceph-volume lvm prepare --bluestore --data /dev/sdc** * *[ceph: root@mostha1 /]# ceph-volume lvm list

====== osd.2 =======

*  [block]

/dev/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c/osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632

      block device

/dev/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c/osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632       block uuid Pq0XeH-LJct-t4yH-f56F-d5jk-JzGQ-zITfhE       cephx lockbox secret       cluster fsid 250f9864-0142-11ee-8e5f-00266cf8869c       cluster name              ceph       crush device class       encrypted                 0 *osd fsid 45c8e92c-caf9-4fe7-9a42-7b45a0794632*       osd id                    2       osdspec affinity       type                      block       vdo                       0 *      devices                   /dev/sdc

*

Now lsblk command shows sdc as an osd: .... sdb 8:16   1 465.8G  0 disk `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 253:1    0 465.8G  0 lvm *sdc 8:32   1 232.9G  0 disk ** **`-ceph--b27d7a07--278d--4ee2--b84e--53256ef8de4c-osd--block--45c8e92c--caf9--4fe7--9a42--7b45a0794632 253:5    0 232.8G  0 lvm ** *

But this osd.2 is "down" and "out" with a strange status (no related cluster host, no weight....) and I cannot activate it as within the podman container systemctl is not working.

[ceph: root@mostha1 /]# ceph osd tree ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT  PRI-AFF -1         1.72823  root default -5         0.45477      host dean  0    hdd  0.22739          osd.0         up   1.00000  1.00000  4    hdd  0.22739          osd.4         up   1.00000  1.00000 -9         0.22739      host ekman  6    hdd  0.22739          osd.6         up   1.00000  1.00000 -7         0.45479      host mostha1  5    hdd  0.45479          osd.5         up   1.00000  1.00000 -3         0.59128      host mostha2  1    hdd  0.22739          osd.1         up   1.00000  1.00000  3    hdd  0.36389          osd.3         up   1.00000  1.00000 *2               0  osd.2               down         0 1.00000*

My attempt to activate the osd:

[ceph: root@mostha1 /]# ceph-volume lvm activate 2 45c8e92c-caf9-4fe7-9a42-7b45a0794632 Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c/osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632 --path /var/lib/ceph/osd/ceph-2 --no-mon-config Running command: /usr/bin/ln -snf /dev/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c/osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632 /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/systemctl enable ceph-volume@lvm-2-45c8e92c-caf9-4fe7-9a42-7b45a0794632  stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-2-45c8e92c-caf9-4fe7-9a42-7b45a0794632.service -> /usr/lib/systemd/system/ceph-volume@.service. Running command: /usr/bin/systemctl enable --runtime ceph-osd@2  stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd@2.service -> /usr/lib/systemd/system/ceph-osd@.service. Running command: /usr/bin/systemctl start ceph-osd@2  stderr: Failed to connect to bus: No such file or directory -->  RuntimeError: command returned non-zero exit status: 1

Patrick

Le 11/10/2023 à 11:00, Eugen Block a écrit :

...

Hi,

just wondering if 'ceph-volume lvm zap --destroy /dev/sdc' would help here. From your previous output you didn't specify the --destroy flag. Which cephadm version is installed on the host? Did you also upgrade the OS when moving to Pacific? (Sorry if I missed that.

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Le 02/10/2023 à 18:22, Patrick Bégou a écrit :

...

Hi all,

still stuck with this problem.

I've deployed octopus and all my HDD have been setup as osd. Fine. I've upgraded to pacific and 2 osd have failed. They have been automatically removed and upgrade finishes. Cluster Health is finaly OK, no data loss.

But now I cannot re-add these osd with pacific (I had previous troubles on these old HDDs, lost one osd in octopus and was able to reset and re-add it).

I've tried manually to add the first osd on the node where it is located, following https://docs.ceph.com/en/pacific/rados/operations/bluestore-migration/ (not sure it's the best idea...) but it fails too. This node was the one used for deploying the cluster.

[ceph: root@mostha1 /]# *ceph-volume lvm zap /dev/sdc* --> Zapping: /dev/sdc --> --destroy was not specified, but zapping a whole device will remove the partition table Running command: /usr/bin/dd if=/dev/zero of=/dev/sdc bs=1M count=10 conv=fsync  stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.663425 s, 15.8 MB/s --> Zapping successful for: <Raw Device: /dev/sdc>

[ceph: root@mostha1 /]# *ceph-volume lvm create --bluestore --data /dev/sdc* Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 9f1eb8ee-41e6-4350-ad73-1be21234ec7c  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 AuthRegistry(0x7fb4e405c4d8) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 AuthRegistry(0x7fb4e40601d0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 AuthRegistry(0x7fb4eb8bee90) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx  stderr: 2023-10-02T16:09:29.858+0000 7fb4e965c700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]  stderr: 2023-10-02T16:09:29.858+0000 7fb4e9e5d700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]  stderr: 2023-10-02T16:09:29.858+0000 7fb4e8e5b700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]  stderr: 2023-10-02T16:09:29.858+0000 7fb4eb8c0700 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication  stderr: [errno 13] RADOS permission denied (error connecting to the cluster) -->  RuntimeError: Unable to create a new OSD id

Any idea of what is wrong ?

Thanks

Patrick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

I'm still trying to understand what can be wrong or how to debug this situation where Ceph cannot see the devices.

The device :dev/sdc exists:

   [root@mostha1 ~]# cephadm shell lsmcli ldl    Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c    Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e    Path     | SCSI VPD 0x83    | Link Type | Serial Number   | Health    Status -------------------------------------------------------------------------    /dev/sda | 50024e92039e4f1c | PATA/SATA | S2B5J90ZA10142  | Good    /dev/sdb | 50014ee0ad5953c9 | PATA/SATA | WD-WMAYP0982329 | Good    /dev/sdc | 50024e920387fa2c | PATA/SATA | S2B5J90ZA02494  | Good

But I cannot do anything with it:

   [root@mostha1 ~]# cephadm shell ceph orch device zap    mostha1.legi.grenoble-inp.fr /dev/sdc --force    Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c    Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e    Error EINVAL: Device path '/dev/sdc' not found on host    'mostha1.legi.grenoble-inp.fr'

Since I moved from octopus to Pacific.

Patrick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Eugen, - the OS is Alma Linux 8 with latests updates. - this morning I've worked with ceph-volume but it ends with a strange final state. I was connected on host mostha1 where /dev/sdc was not reconized. These are the steps followed based on the ceph-volume documentation I've read: [root@mostha1 ~]# cephadm shell [ceph: root@mostha1 /]# ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring [ceph: root@mostha1 /]# ceph-volume lvm prepare --bluestore --data /dev/sdc Now lsblk command shows sdc as an osd: .... sdb 8:16   1 465.8G  0 disk `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 253:1    0 465.8G  0 lvm sdc 8:32   1 232.9G  0 disk `-ceph--b27d7a07--278d--4ee2--b84e--53256ef8de4c-osd--block--45c8e92c--caf9--4fe7--9a42--7b45a0794632 253:5    0 232.8G  0 lvm Then I've tried to activate this osd but it fails as in podman I have not access to systemctl: [ceph: root@mostha1 /]# ceph-volume lvm activate 2 45c8e92c-caf9-4fe7-9a42-7b45a0794632 ..... Running command: /usr/bin/systemctl start ceph-osd@2  stderr: Failed to connect to bus: No such file or directory -->  RuntimeError: command returned non-zero exit status: 1 [ceph: root@mostha1 /]# ceph osd tree And now I have now I have a strange status for this osd.2: [ceph: root@mostha1 /]# ceph osd tree ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT  PRI-AFF -1         1.72823  root default -5         0.45477      host dean  0    hdd  0.22739          osd.0         up   1.00000  1.00000  4    hdd  0.22739          osd.4         up   1.00000  1.00000 -9         0.22739      host ekman  6    hdd  0.22739          osd.6         up   1.00000  1.00000 -7         0.45479      host mostha1  5    hdd  0.45479          osd.5         up   1.00000  1.00000 -3         0.59128      host mostha2  1    hdd  0.22739          osd.1         up   1.00000  1.00000  3    hdd  0.36389          osd.3         up   1.00000  1.00000  2               0  osd.2               down         0  1.00000 I've tried to destroy the osd as you suggest but even if the command returns no error I still have this osd even if "lsblk" do not show any more /dev/sdc as a ceph osd device. *[ceph: root@mostha1 /]# ceph-volume lvm zap --destroy /dev/sdc** *--> Zapping: /dev/sdc --> Zapping lvm member /dev/sdc. lv_path is /dev/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c/osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632 --> Unmounting /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-2  stderr: umount: /var/lib/ceph/osd/ceph-2 unmounted Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c/osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632 bs=1M count=10 conv=fsync  stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.575633 s, 18.2 MB/s --> Only 1 LV left in VG, will proceed to destroy volume group ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c  stderr: Removing ceph--b27d7a07--278d--4ee2--b84e--53256ef8de4c-osd--block--45c8e92c--caf9--4fe7--9a42--7b45a0794632 (253:1)  stderr: Releasing logical volume "osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632"  stderr: Archiving volume group "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" metadata (seqno 5).  stdout: Logical volume "osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632" successfully removed.  stderr: Removing physical volume "/dev/sdc" from volume group "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c"  stdout: Volume group "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" successfully removed  stderr: Creating volume group backup "/etc/lvm/backup/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" (seqno 6). Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/sdc  stdout: Labels on physical volume "/dev/sdc" successfully wiped. Running command: /usr/bin/dd if=/dev/zero of=/dev/sdc bs=1M count=10 conv=fsync  stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.590652 s, 17.8 MB/s *--> Zapping successful for: <Raw Device: /dev/sdc>* * * *[ceph: root@mostha1 /]# ceph osd tree** *ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT PRI-AFF -1         1.72823  root default -5         0.45477      host dean  0    hdd  0.22739          osd.0         up   1.00000  1.00000  4    hdd  0.22739          osd.4         up   1.00000  1.00000 -9         0.22739      host ekman  6    hdd  0.22739          osd.6         up   1.00000  1.00000 -7         0.45479      host mostha1  5    hdd  0.45479          osd.5         up   1.00000  1.00000 -3         0.59128      host mostha2  1    hdd  0.22739          osd.1         up   1.00000  1.00000  3    hdd  0.36389          osd.3         up   1.00000  1.00000  2               0  osd.2               down         0  1.00000 * * *[ceph: root@mostha1 /]# lsblk** *NAME MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT sda 8:0    1 232.9G  0 disk |-sda1 8:1    1   3.9G  0 part /rootfs/boot |-sda2 8:2    1   3.9G  0 part [SWAP] `-sda3 8:3    1   225G  0 part |-al8vg-rootvol 253:0    0  48.8G  0 lvm  /rootfs |-al8vg-homevol 253:3    0   9.8G  0 lvm  /rootfs/home |-al8vg-tmpvol 253:4    0   9.8G  0 lvm  /rootfs/tmp `-al8vg-varvol 253:5    0  19.8G  0 lvm  /rootfs/var sdb 8:16   1 465.8G  0 disk `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 253:2    0 465.8G  0 lvm *sdc * Patrick Le 11/10/2023 à 11:00, Eugen Block a écrit :

Hi,

just wondering if 'ceph-volume lvm zap --destroy /dev/sdc' would help here. From your previous output you didn't specify the --destroy flag. Which cephadm version is installed on the host? Did you also upgrade the OS when moving to Pacific? (Sorry if I missed that.

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Le 02/10/2023 à 18:22, Patrick Bégou a écrit :

...

Hi all,

still stuck with this problem.

I've deployed octopus and all my HDD have been setup as osd. Fine. I've upgraded to pacific and 2 osd have failed. They have been automatically removed and upgrade finishes. Cluster Health is finaly OK, no data loss.

But now I cannot re-add these osd with pacific (I had previous troubles on these old HDDs, lost one osd in octopus and was able to reset and re-add it).

I've tried manually to add the first osd on the node where it is located, following https://docs.ceph.com/en/pacific/rados/operations/bluestore-migration/ (not sure it's the best idea...) but it fails too. This node was the one used for deploying the cluster.

[ceph: root@mostha1 /]# *ceph-volume lvm zap /dev/sdc* --> Zapping: /dev/sdc --> --destroy was not specified, but zapping a whole device will remove the partition table Running command: /usr/bin/dd if=/dev/zero of=/dev/sdc bs=1M count=10 conv=fsync  stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.663425 s, 15.8 MB/s --> Zapping successful for: <Raw Device: /dev/sdc>

[ceph: root@mostha1 /]# *ceph-volume lvm create --bluestore --data /dev/sdc* Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 9f1eb8ee-41e6-4350-ad73-1be21234ec7c  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 AuthRegistry(0x7fb4e405c4d8) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 AuthRegistry(0x7fb4e40601d0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 AuthRegistry(0x7fb4eb8bee90) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx  stderr: 2023-10-02T16:09:29.858+0000 7fb4e965c700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]  stderr: 2023-10-02T16:09:29.858+0000 7fb4e9e5d700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]  stderr: 2023-10-02T16:09:29.858+0000 7fb4e8e5b700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]  stderr: 2023-10-02T16:09:29.858+0000 7fb4eb8c0700 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication  stderr: [errno 13] RADOS permission denied (error connecting to the cluster) -->  RuntimeError: Unable to create a new OSD id

Any idea of what is wrong ?

Thanks

Patrick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

I'm still trying to understand what can be wrong or how to debug this situation where Ceph cannot see the devices.

The device :dev/sdc exists:

   [root@mostha1 ~]# cephadm shell lsmcli ldl    Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c    Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e    Path     | SCSI VPD 0x83    | Link Type | Serial Number   | Health    Status -------------------------------------------------------------------------    /dev/sda | 50024e92039e4f1c | PATA/SATA | S2B5J90ZA10142  | Good    /dev/sdb | 50014ee0ad5953c9 | PATA/SATA | WD-WMAYP0982329 | Good    /dev/sdc | 50024e920387fa2c | PATA/SATA | S2B5J90ZA02494  | Good

But I cannot do anything with it:

   [root@mostha1 ~]# cephadm shell ceph orch device zap    mostha1.legi.grenoble-inp.fr /dev/sdc --force    Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c    Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e    Error EINVAL: Device path '/dev/sdc' not found on host    'mostha1.legi.grenoble-inp.fr'

Since I moved from octopus to Pacific.

Patrick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Eugen, first many thanks for the time spent on this problem. "ceph osd purge 2 --force --yes-i-really-mean-it" works and clean all the bas status. *[root@mostha1 ~]# cephadm shell *Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e * * *[ceph: root@mostha1 /]# ceph osd purge 2 --force --yes-i-really-mean-it * purged osd.2 * * *[ceph: root@mostha1 /]# ceph osd tree* ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT  PRI-AFF -1         1.72823  root default -5         0.45477      host dean  0    hdd  0.22739          osd.0         up   1.00000  1.00000  4    hdd  0.22739          osd.4         up   1.00000  1.00000 -9         0.22739      host ekman  6    hdd  0.22739          osd.6         up   1.00000  1.00000 -7         0.45479      host mostha1  5    hdd  0.45479          osd.5         up   1.00000  1.00000 -3         0.59128      host mostha2  1    hdd  0.22739          osd.1         up   1.00000  1.00000  3    hdd  0.36389          osd.3         up   1.00000  1.00000 * * *[ceph: root@mostha1 /]# lsblk* NAME MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT sda 8:0    1 232.9G  0 disk |-sda1 8:1    1   3.9G  0 part /rootfs/boot |-sda2 8:2    1   3.9G  0 part [SWAP] `-sda3 8:3    1   225G  0 part |-al8vg-rootvol 253:0    0  48.8G  0 lvm  /rootfs |-al8vg-homevol 253:2    0   9.8G  0 lvm  /rootfs/home |-al8vg-tmpvol 253:3    0   9.8G  0 lvm  /rootfs/tmp `-al8vg-varvol 253:4    0  19.8G  0 lvm  /rootfs/var sdb 8:16   1 465.8G  0 disk `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 253:1    0 465.8G  0 lvm sdc 8:32   1 232.9G  0 disk "cephadm ceph-volume inventory" returns nothing: *[root@mostha1 ~]# cephadm ceph-volume inventory ** *Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e Device Path               Size         Device nodes    rotates available Model name [root@mostha1 ~]# But running the same command within cephadm 15.2.17 works: *[root@mostha1 ~]# cephadm --image 93146564743f ceph-volume inventory* Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c Device Path               Size         rotates available Model name /dev/sdc                  232.83 GB    True    True      SAMSUNG HE253GJ /dev/sda                  232.83 GB    True    False     SAMSUNG HE253GJ /dev/sdb                  465.76 GB    True    False     WDC WD5003ABYX-1 [root@mostha1 ~]# *[root@mostha1 ~]# podman images -a** *REPOSITORY                        TAG         IMAGE ID CREATED        SIZE quay.io/ceph/ceph                 v16.2.14    f13d80acdbb5  2 weeks ago    1.21 GB quay.io/ceph/ceph                 v15.2.17    93146564743f  14 months ago  1.24 GB .... Patrick Le 11/10/2023 à 15:14, Eugen Block a écrit :

Your response is a bit confusing since it seems to be mixed up with the previous answer. So you still need to remove the OSD properly, so purge it from the crush tree:

ceph osd purge 2 --force --yes-i-really-mean-it (only in a test cluster!)

If everything is clean (OSD has been removed, disk has been zapped, lsblk shows no LVs for that disk) you can check the inventory:

cephadm ceph-volume inventory

Please also add the output of 'ceph orch ls osd --export'.

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Hi Eugen,

- the OS is Alma Linux 8 with latests updates.

- this morning I've worked with ceph-volume but it ends with a strange final state. I was connected on host mostha1 where /dev/sdc was not reconized. These are the steps followed based on the ceph-volume documentation I've read: [root@mostha1 ~]# cephadm shell [ceph: root@mostha1 /]# ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring [ceph: root@mostha1 /]# ceph-volume lvm prepare --bluestore --data /dev/sdc

Now lsblk command shows sdc as an osd: .... sdb 8:16   1 465.8G  0 disk `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 253:1    0 465.8G  0 lvm sdc 8:32   1 232.9G  0 disk `-ceph--b27d7a07--278d--4ee2--b84e--53256ef8de4c-osd--block--45c8e92c--caf9--4fe7--9a42--7b45a0794632 253:5    0 232.8G  0 lvm

Then I've tried to activate this osd but it fails as in podman I have not access to systemctl:

[ceph: root@mostha1 /]# ceph-volume lvm activate 2 45c8e92c-caf9-4fe7-9a42-7b45a0794632 ..... Running command: /usr/bin/systemctl start ceph-osd@2  stderr: Failed to connect to bus: No such file or directory -->  RuntimeError: command returned non-zero exit status: 1 [ceph: root@mostha1 /]# ceph osd tree

And now I have now I have a strange status for this osd.2:

[ceph: root@mostha1 /]# ceph osd tree ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT  PRI-AFF -1         1.72823  root default -5         0.45477      host dean  0    hdd  0.22739          osd.0         up   1.00000  1.00000  4    hdd  0.22739          osd.4         up   1.00000  1.00000 -9         0.22739      host ekman  6    hdd  0.22739          osd.6         up   1.00000  1.00000 -7         0.45479      host mostha1  5    hdd  0.45479          osd.5         up   1.00000  1.00000 -3         0.59128      host mostha2  1    hdd  0.22739          osd.1         up   1.00000  1.00000  3    hdd  0.36389          osd.3         up   1.00000  1.00000  2               0  osd.2               down         0  1.00000

I've tried to destroy the osd as you suggest but even if the command returns no error I still have this osd even if "lsblk" do not show any more /dev/sdc as a ceph osd device.

*[ceph: root@mostha1 /]# ceph-volume lvm zap --destroy /dev/sdc** *--> Zapping: /dev/sdc --> Zapping lvm member /dev/sdc. lv_path is /dev/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c/osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632 --> Unmounting /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-2  stderr: umount: /var/lib/ceph/osd/ceph-2 unmounted Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c/osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632 bs=1M count=10 conv=fsync  stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.575633 s, 18.2 MB/s --> Only 1 LV left in VG, will proceed to destroy volume group ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c  stderr: Removing ceph--b27d7a07--278d--4ee2--b84e--53256ef8de4c-osd--block--45c8e92c--caf9--4fe7--9a42--7b45a0794632 (253:1)  stderr: Releasing logical volume "osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632"  stderr: Archiving volume group "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" metadata (seqno 5).  stdout: Logical volume "osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632" successfully removed.  stderr: Removing physical volume "/dev/sdc" from volume group "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c"  stdout: Volume group "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" successfully removed  stderr: Creating volume group backup "/etc/lvm/backup/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" (seqno 6). Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/sdc  stdout: Labels on physical volume "/dev/sdc" successfully wiped. Running command: /usr/bin/dd if=/dev/zero of=/dev/sdc bs=1M count=10 conv=fsync  stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.590652 s, 17.8 MB/s *--> Zapping successful for: <Raw Device: /dev/sdc>* * * *[ceph: root@mostha1 /]# ceph osd tree** *ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT PRI-AFF -1         1.72823  root default -5         0.45477      host dean  0    hdd  0.22739          osd.0         up   1.00000  1.00000  4    hdd  0.22739          osd.4         up   1.00000  1.00000 -9         0.22739      host ekman  6    hdd  0.22739          osd.6         up   1.00000  1.00000 -7         0.45479      host mostha1  5    hdd  0.45479          osd.5         up   1.00000  1.00000 -3         0.59128      host mostha2  1    hdd  0.22739          osd.1         up   1.00000  1.00000  3    hdd  0.36389          osd.3         up   1.00000  1.00000  2               0  osd.2               down         0  1.00000 * * *[ceph: root@mostha1 /]# lsblk** *NAME MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT sda 8:0    1 232.9G  0 disk |-sda1 8:1    1   3.9G  0 part /rootfs/boot |-sda2 8:2    1   3.9G  0 part [SWAP] `-sda3 8:3    1   225G  0 part |-al8vg-rootvol 253:0    0  48.8G  0 lvm  /rootfs |-al8vg-homevol 253:3    0   9.8G  0 lvm  /rootfs/home |-al8vg-tmpvol 253:4    0   9.8G  0 lvm  /rootfs/tmp `-al8vg-varvol 253:5    0  19.8G  0 lvm  /rootfs/var sdb 8:16   1 465.8G  0 disk `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 253:2    0 465.8G  0 lvm *sdc *

Patrick Le 11/10/2023 à 11:00, Eugen Block a écrit :

...

Hi,

just wondering if 'ceph-volume lvm zap --destroy /dev/sdc' would help here. From your previous output you didn't specify the --destroy flag. Which cephadm version is installed on the host? Did you also upgrade the OS when moving to Pacific? (Sorry if I missed that.

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Le 02/10/2023 à 18:22, Patrick Bégou a écrit :

...

Hi all,

still stuck with this problem.

I've deployed octopus and all my HDD have been setup as osd. Fine. I've upgraded to pacific and 2 osd have failed. They have been automatically removed and upgrade finishes. Cluster Health is finaly OK, no data loss.

But now I cannot re-add these osd with pacific (I had previous troubles on these old HDDs, lost one osd in octopus and was able to reset and re-add it).

I've tried manually to add the first osd on the node where it is located, following https://docs.ceph.com/en/pacific/rados/operations/bluestore-migration/ (not sure it's the best idea...) but it fails too. This node was the one used for deploying the cluster.

[ceph: root@mostha1 /]# *ceph-volume lvm zap /dev/sdc* --> Zapping: /dev/sdc --> --destroy was not specified, but zapping a whole device will remove the partition table Running command: /usr/bin/dd if=/dev/zero of=/dev/sdc bs=1M count=10 conv=fsync  stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.663425 s, 15.8 MB/s --> Zapping successful for: <Raw Device: /dev/sdc>

[ceph: root@mostha1 /]# *ceph-volume lvm create --bluestore --data /dev/sdc* Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 9f1eb8ee-41e6-4350-ad73-1be21234ec7c  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 AuthRegistry(0x7fb4e405c4d8) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 AuthRegistry(0x7fb4e40601d0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 AuthRegistry(0x7fb4eb8bee90) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx  stderr: 2023-10-02T16:09:29.858+0000 7fb4e965c700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]  stderr: 2023-10-02T16:09:29.858+0000 7fb4e9e5d700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]  stderr: 2023-10-02T16:09:29.858+0000 7fb4e8e5b700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]  stderr: 2023-10-02T16:09:29.858+0000 7fb4eb8c0700 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication  stderr: [errno 13] RADOS permission denied (error connecting to the cluster) -->  RuntimeError: Unable to create a new OSD id

Any idea of what is wrong ?

Thanks

Patrick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

I'm still trying to understand what can be wrong or how to debug this situation where Ceph cannot see the devices.

The device :dev/sdc exists:

   [root@mostha1 ~]# cephadm shell lsmcli ldl    Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c    Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e

   Path     | SCSI VPD 0x83    | Link Type | Serial Number | Health    Status ceph osd purge 2 --force --yes-i-really-mean-it -------------------------------------------------------------------------

   /dev/sda | 50024e92039e4f1c | PATA/SATA | S2B5J90ZA10142 | Good    /dev/sdb | 50014ee0ad5953c9 | PATA/SATA | WD-WMAYP0982329 | Good    /dev/sdc | 50024e920387fa2c | PATA/SATA | S2B5J90ZA02494 | Good

But I cannot do anything with it:

   [root@mostha1 ~]# cephadm shell ceph orch device zap    mostha1.legi.grenoble-inp.fr /dev/sdc --force    Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c    Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e

   Error EINVAL: Device path '/dev/sdc' not found on host    'mostha1.legi.grenoble-inp.fr'

Since I moved from octopus to Pacific.

Patrick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Eugen, [root@mostha1 ~]# rpm -q cephadm cephadm-16.2.14-0.el8.noarch Log associated to the 2023-10-11 16:16:02,167 7f820515fb80 DEBUG -------------------------------------------------------------------------------- cephadm ['gather-facts'] 2023-10-11 16:16:02,208 7f820515fb80 DEBUG /bin/podman: 4.4.1 2023-10-11 16:16:02,313 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,317 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,322 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,326 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,329 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,333 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:04,474 7ff2a5c08b80 DEBUG -------------------------------------------------------------------------------- cephadm ['ceph-volume', 'inventory'] 2023-10-11 16:16:04,516 7ff2a5c08b80 DEBUG /usr/bin/podman: 4.4.1 2023-10-11 16:16:04,520 7ff2a5c08b80 DEBUG Using default config: /etc/ceph/ceph.conf 2023-10-11 16:16:04,573 7ff2a5c08b80 DEBUG /usr/bin/podman: 0d28d71358d7,445.8MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 2084faaf4d54,13.27MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 61073c53805d,512.7MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 6b9f0b72d668,361.1MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 7493a28808ad,163.7MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: a89672a3accf,59.22MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: b45271cc9726,54.24MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: e00ec13ab138,707.3MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,35.55MB / 50.32GB 2023-10-11 16:16:04,630 7ff2a5c08b80 DEBUG /usr/bin/podman: 0d28d71358d7,1.28% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 2084faaf4d54,0.00% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 61073c53805d,1.19% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 6b9f0b72d668,1.03% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 7493a28808ad,0.78% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: a89672a3accf,0.11% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: b45271cc9726,1.35% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: e00ec13ab138,0.43% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,0.02% 2023-10-11 16:16:04,634 7ff2a5c08b80 INFO Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c 2023-10-11 16:16:04,691 7ff2a5c08b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:16:04,692 7ff2a5c08b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:c08064dde4bba4e72a1f55d90ca32df9ef5aafab82efe2e0a0722444a5aaacca 2023-10-11 16:16:04,692 7ff2a5c08b80 DEBUG /usr/bin/podman: docker.io/ceph/ceph@sha256:056637972a107df4096f10951e4216b21fcd8ae0b9fb4552e628d35df3f61139 2023-10-11 16:16:04,694 7ff2a5c08b80 INFO Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:16:05,094 7ff2a5c08b80 DEBUG stat: 167 167 2023-10-11 16:16:05,903 7ff2a5c08b80 DEBUG Acquiring lock 140679815723776 on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:16:05,903 7ff2a5c08b80 DEBUG Lock 140679815723776 acquired on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:16:05,929 7ff2a5c08b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:05,933 7ff2a5c08b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:06,700 7ff2a5c08b80 DEBUG /usr/bin/podman: 2023-10-11 16:16:06,701 7ff2a5c08b80 DEBUG /usr/bin/podman: Device Path               Size         Device nodes    rotates available Model name I have only one version of cephadm in /var/lib/ceph/{fsid} : [root@mostha1 ~]# ls -lrt /var/lib/ceph/250f9864-0142-11ee-8e5f-00266cf8869c/cephadm* -rw-r--r-- 1 root root 350889 28 sept. 16:39 /var/lib/ceph/250f9864-0142-11ee-8e5f-00266cf8869c/cephadm.f6868821c084cd9740b59c7c5eb59f0dd47f6e3b1e6fecb542cb44134ace8d78 Running " python3 /var/lib/ceph/250f9864-0142-11ee-8e5f-00266cf8869c/cephadm.f6868821c084cd9740b59c7c5eb59f0dd47f6e3b1e6fecb542cb44134ace8d78 ceph-volume inventory" give the same output and the same log (execpt the valu of the lock): 2023-10-11 16:21:35,965 7f467cf31b80 DEBUG -------------------------------------------------------------------------------- cephadm ['ceph-volume', 'inventory'] 2023-10-11 16:21:36,009 7f467cf31b80 DEBUG /usr/bin/podman: 4.4.1 2023-10-11 16:21:36,012 7f467cf31b80 DEBUG Using default config: /etc/ceph/ceph.conf 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 0d28d71358d7,452.1MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 2084faaf4d54,13.27MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 61073c53805d,513.6MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 6b9f0b72d668,322.4MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 7493a28808ad,164MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: a89672a3accf,58.5MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: b45271cc9726,54.69MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: e00ec13ab138,707.1MB / 50.32GB 2023-10-11 16:21:36,068 7f467cf31b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,36.28MB / 50.32GB 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 0d28d71358d7,1.27% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 2084faaf4d54,0.00% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 61073c53805d,1.16% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 6b9f0b72d668,1.02% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 7493a28808ad,0.78% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: a89672a3accf,0.11% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: b45271cc9726,1.35% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: e00ec13ab138,0.41% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,0.02% 2023-10-11 16:21:36,128 7f467cf31b80 INFO Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c 2023-10-11 16:21:36,186 7f467cf31b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:21:36,187 7f467cf31b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:c08064dde4bba4e72a1f55d90ca32df9ef5aafab82efe2e0a0722444a5aaacca 2023-10-11 16:21:36,187 7f467cf31b80 DEBUG /usr/bin/podman: docker.io/ceph/ceph@sha256:056637972a107df4096f10951e4216b21fcd8ae0b9fb4552e628d35df3f61139 2023-10-11 16:21:36,189 7f467cf31b80 INFO Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:21:36,549 7f467cf31b80 DEBUG stat: 167 167 2023-10-11 16:21:36,942 7f467cf31b80 DEBUG Acquiring lock 139940396923424 on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:21:36,942 7f467cf31b80 DEBUG Lock 139940396923424 acquired on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:21:36,969 7f467cf31b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:21:36,972 7f467cf31b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:21:37,749 7f467cf31b80 DEBUG /usr/bin/podman: 2023-10-11 16:21:37,750 7f467cf31b80 DEBUG /usr/bin/podman: Device Path               Size         Device nodes    rotates available Model name Patrick Le 11/10/2023 à 15:59, Eugen Block a écrit :

Can you check which cephadm version is installed on the host? And then please add (only the relevant) output from the cephadm.log when you run the inventory (without the --image <octopus>). Sometimes the version mismatch on the host and the one the orchestrator uses can cause some disruptions. You could try the same with the latest cephadm you have in /var/lib/ceph/${fsid}/ (ls -lrt /var/lib/ceph/${fsid}/cephadm.*). I mentioned that in this thread [1]. So you could try the following:

$ chmod +x /var/lib/ceph/{fsid}/cephadm.{latest}

$ python3 /var/lib/ceph/{fsid}/cephadm.{latest} ceph-volume inventory

Does the output differ? Paste the relevant cephadm.log from that attempt as well.

[1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LASBJCSPFGDY...

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Hi Eugen,

first many thanks for the time spent on this problem.

"ceph osd purge 2 --force --yes-i-really-mean-it" works and clean all the bas status.

*[root@mostha1 ~]# cephadm shell *Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e * * *[ceph: root@mostha1 /]# ceph osd purge 2 --force --yes-i-really-mean-it * purged osd.2 * * *[ceph: root@mostha1 /]# ceph osd tree* ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT  PRI-AFF -1         1.72823  root default -5         0.45477      host dean  0    hdd  0.22739          osd.0         up   1.00000  1.00000  4    hdd  0.22739          osd.4         up   1.00000  1.00000 -9         0.22739      host ekman  6    hdd  0.22739          osd.6         up   1.00000  1.00000 -7         0.45479      host mostha1  5    hdd  0.45479          osd.5         up   1.00000  1.00000 -3         0.59128      host mostha2  1    hdd  0.22739          osd.1         up   1.00000  1.00000  3    hdd  0.36389          osd.3         up   1.00000  1.00000 * * *[ceph: root@mostha1 /]# lsblk* NAME MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT sda 8:0    1 232.9G  0 disk |-sda1 8:1    1   3.9G  0 part /rootfs/boot |-sda2 8:2    1   3.9G  0 part [SWAP] `-sda3 8:3    1   225G  0 part |-al8vg-rootvol 253:0    0  48.8G  0 lvm  /rootfs |-al8vg-homevol 253:2    0   9.8G  0 lvm  /rootfs/home |-al8vg-tmpvol 253:3    0   9.8G  0 lvm  /rootfs/tmp `-al8vg-varvol 253:4    0  19.8G  0 lvm  /rootfs/var sdb 8:16   1 465.8G  0 disk `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 253:1    0 465.8G  0 lvm sdc 8:32   1 232.9G  0 disk

"cephadm ceph-volume inventory" returns nothing:

*[root@mostha1 ~]# cephadm ceph-volume inventory ** *Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e

Device Path               Size         Device nodes    rotates available Model name

[root@mostha1 ~]#

But running the same command within cephadm 15.2.17 works:

*[root@mostha1 ~]# cephadm --image 93146564743f ceph-volume inventory* Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c

Device Path               Size         rotates available Model name /dev/sdc                  232.83 GB    True    True      SAMSUNG HE253GJ /dev/sda                  232.83 GB    True    False     SAMSUNG HE253GJ /dev/sdb                  465.76 GB    True    False     WDC WD5003ABYX-1

[root@mostha1 ~]#

*[root@mostha1 ~]# podman images -a** *REPOSITORY                        TAG         IMAGE ID CREATED        SIZE quay.io/ceph/ceph                 v16.2.14    f13d80acdbb5  2 weeks ago    1.21 GB quay.io/ceph/ceph                 v15.2.17    93146564743f  14 months ago  1.24 GB ....

Patrick

Le 11/10/2023 à 15:14, Eugen Block a écrit :

...

Your response is a bit confusing since it seems to be mixed up with the previous answer. So you still need to remove the OSD properly, so purge it from the crush tree:

ceph osd purge 2 --force --yes-i-really-mean-it (only in a test cluster!)

If everything is clean (OSD has been removed, disk has been zapped, lsblk shows no LVs for that disk) you can check the inventory:

cephadm ceph-volume inventory

Please also add the output of 'ceph orch ls osd --export'.

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Hi Eugen,

- the OS is Alma Linux 8 with latests updates.

- this morning I've worked with ceph-volume but it ends with a strange final state. I was connected on host mostha1 where /dev/sdc was not reconized. These are the steps followed based on the ceph-volume documentation I've read: [root@mostha1 ~]# cephadm shell [ceph: root@mostha1 /]# ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring [ceph: root@mostha1 /]# ceph-volume lvm prepare --bluestore --data /dev/sdc

Now lsblk command shows sdc as an osd: .... sdb 8:16   1 465.8G  0 disk `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 253:1    0 465.8G  0 lvm sdc 8:32   1 232.9G  0 disk `-ceph--b27d7a07--278d--4ee2--b84e--53256ef8de4c-osd--block--45c8e92c--caf9--4fe7--9a42--7b45a0794632 253:5    0 232.8G  0 lvm

Then I've tried to activate this osd but it fails as in podman I have not access to systemctl:

[ceph: root@mostha1 /]# ceph-volume lvm activate 2 45c8e92c-caf9-4fe7-9a42-7b45a0794632 ..... Running command: /usr/bin/systemctl start ceph-osd@2  stderr: Failed to connect to bus: No such file or directory -->  RuntimeError: command returned non-zero exit status: 1 [ceph: root@mostha1 /]# ceph osd tree

And now I have now I have a strange status for this osd.2:

[ceph: root@mostha1 /]# ceph osd tree ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT PRI-AFF -1         1.72823  root default -5         0.45477      host dean  0    hdd  0.22739          osd.0         up   1.00000 1.00000  4    hdd  0.22739          osd.4         up   1.00000 1.00000 -9         0.22739      host ekman  6    hdd  0.22739          osd.6         up   1.00000 1.00000 -7         0.45479      host mostha1  5    hdd  0.45479          osd.5         up   1.00000 1.00000 -3         0.59128      host mostha2  1    hdd  0.22739          osd.1         up   1.00000 1.00000  3    hdd  0.36389          osd.3         up   1.00000 1.00000  2               0  osd.2               down         0 1.00000

I've tried to destroy the osd as you suggest but even if the command returns no error I still have this osd even if "lsblk" do not show any more /dev/sdc as a ceph osd device.

*[ceph: root@mostha1 /]# ceph-volume lvm zap --destroy /dev/sdc** *--> Zapping: /dev/sdc --> Zapping lvm member /dev/sdc. lv_path is /dev/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c/osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632 --> Unmounting /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-2  stderr: umount: /var/lib/ceph/osd/ceph-2 unmounted Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c/osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632 bs=1M count=10 conv=fsync  stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.575633 s, 18.2 MB/s --> Only 1 LV left in VG, will proceed to destroy volume group ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c  stderr: Removing ceph--b27d7a07--278d--4ee2--b84e--53256ef8de4c-osd--block--45c8e92c--caf9--4fe7--9a42--7b45a0794632 (253:1)  stderr: Releasing logical volume "osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632"  stderr: Archiving volume group "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" metadata (seqno 5).  stdout: Logical volume "osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632" successfully removed.  stderr: Removing physical volume "/dev/sdc" from volume group "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c"  stdout: Volume group "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" successfully removed  stderr: Creating volume group backup "/etc/lvm/backup/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" (seqno 6). Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/sdc  stdout: Labels on physical volume "/dev/sdc" successfully wiped. Running command: /usr/bin/dd if=/dev/zero of=/dev/sdc bs=1M count=10 conv=fsync  stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.590652 s, 17.8 MB/s *--> Zapping successful for: <Raw Device: /dev/sdc>* * * *[ceph: root@mostha1 /]# ceph osd tree** *ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT PRI-AFF -1         1.72823  root default -5         0.45477      host dean  0    hdd  0.22739          osd.0         up   1.00000 1.00000  4    hdd  0.22739          osd.4         up   1.00000 1.00000 -9         0.22739      host ekman  6    hdd  0.22739          osd.6         up   1.00000 1.00000 -7         0.45479      host mostha1  5    hdd  0.45479          osd.5         up   1.00000 1.00000 -3         0.59128      host mostha2  1    hdd  0.22739          osd.1         up   1.00000 1.00000  3    hdd  0.36389          osd.3         up   1.00000 1.00000  2               0  osd.2               down         0 1.00000 * * *[ceph: root@mostha1 /]# lsblk** *NAME MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT sda 8:0    1 232.9G  0 disk |-sda1 8:1    1   3.9G  0 part /rootfs/boot |-sda2 8:2    1   3.9G  0 part [SWAP] `-sda3 8:3    1   225G  0 part |-al8vg-rootvol 253:0    0  48.8G  0 lvm  /rootfs |-al8vg-homevol 253:3    0   9.8G  0 lvm  /rootfs/home |-al8vg-tmpvol 253:4    0   9.8G  0 lvm  /rootfs/tmp `-al8vg-varvol 253:5    0  19.8G  0 lvm  /rootfs/var sdb 8:16   1 465.8G  0 disk `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 253:2    0 465.8G  0 lvm *sdc *

Patrick Le 11/10/2023 à 11:00, Eugen Block a écrit :

...

Hi,

just wondering if 'ceph-volume lvm zap --destroy /dev/sdc' would help here. From your previous output you didn't specify the --destroy flag. Which cephadm version is installed on the host? Did you also upgrade the OS when moving to Pacific? (Sorry if I missed that.

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Le 02/10/2023 à 18:22, Patrick Bégou a écrit : > Hi all, > > still stuck with this problem. > > I've deployed octopus and all my HDD have been setup as osd. Fine. > I've upgraded to pacific and 2 osd have failed. They have been > automatically removed and upgrade finishes. Cluster Health is > finaly OK, no data loss. > > But now I cannot re-add these osd with pacific (I had previous > troubles on these old HDDs, lost one osd in octopus and was able > to reset and re-add it). > > I've tried manually to add the first osd on the node where it is > located, following > https://docs.ceph.com/en/pacific/rados/operations/bluestore-migration/ > (not sure it's the best idea...) but it fails too. This node was > the one used for deploying the cluster. > > [ceph: root@mostha1 /]# *ceph-volume lvm zap /dev/sdc* > --> Zapping: /dev/sdc > --> --destroy was not specified, but zapping a whole device will > remove the partition table > Running command: /usr/bin/dd if=/dev/zero of=/dev/sdc bs=1M > count=10 conv=fsync >  stderr: 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.663425 s, 15.8 MB/s > --> Zapping successful for: <Raw Device: /dev/sdc> > > > [ceph: root@mostha1 /]# *ceph-volume lvm create --bluestore > --data /dev/sdc* > Running command: /usr/bin/ceph-authtool --gen-print-key > Running command: /usr/bin/ceph --cluster ceph --name > client.bootstrap-osd --keyring > /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new > 9f1eb8ee-41e6-4350-ad73-1be21234ec7c >  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 auth: > unable to find a keyring on > /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or > directory >  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 > AuthRegistry(0x7fb4e405c4d8) no keyring found at > /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx >  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 auth: > unable to find a keyring on > /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or > directory >  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 > AuthRegistry(0x7fb4e40601d0) no keyring found at > /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx >  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 auth: > unable to find a keyring on > /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or > directory >  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 > AuthRegistry(0x7fb4eb8bee90) no keyring found at > /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx >  stderr: 2023-10-02T16:09:29.858+0000 7fb4e965c700 -1 > monclient(hunting): handle_auth_bad_method server > allowed_methods [2] but i only support [1] >  stderr: 2023-10-02T16:09:29.858+0000 7fb4e9e5d700 -1 > monclient(hunting): handle_auth_bad_method server > allowed_methods [2] but i only support [1] >  stderr: 2023-10-02T16:09:29.858+0000 7fb4e8e5b700 -1 > monclient(hunting): handle_auth_bad_method server > allowed_methods [2] but i only support [1] >  stderr: 2023-10-02T16:09:29.858+0000 7fb4eb8c0700 -1 monclient: > authenticate NOTE: no keyring found; disabled cephx authentication >  stderr: [errno 13] RADOS permission denied (error connecting to > the cluster) > -->  RuntimeError: Unable to create a new OSD id > > Any idea of what is wrong ? > > Thanks > > Patrick > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io

I'm still trying to understand what can be wrong or how to debug this situation where Ceph cannot see the devices.

The device :dev/sdc exists:

   [root@mostha1 ~]# cephadm shell lsmcli ldl    Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c    Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e    Path     | SCSI VPD 0x83    | Link Type | Serial Number | Health    Status ceph osd purge 2 --force --yes-i-really-mean-it -------------------------------------------------------------------------    /dev/sda | 50024e92039e4f1c | PATA/SATA | S2B5J90ZA10142 | Good    /dev/sdb | 50014ee0ad5953c9 | PATA/SATA | WD-WMAYP0982329 | Good    /dev/sdc | 50024e920387fa2c | PATA/SATA | S2B5J90ZA02494 | Good

But I cannot do anything with it:

   [root@mostha1 ~]# cephadm shell ceph orch device zap    mostha1.legi.grenoble-inp.fr /dev/sdc --force    Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c    Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e    Error EINVAL: Device path '/dev/sdc' not found on host    'mostha1.legi.grenoble-inp.fr'

Since I moved from octopus to Pacific.

Patrick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

This afternoon I have a look at the python file but do not manage how it works with containers as I am only a Fortran HPC programmer... but I found that "cephadm gather-facts" shows all the HDD in Pacific. Some quick tests show: ============== Nautilus ============== [root@mostha1 ~]# cephadm --image quay.io/ceph/ceph:v14 ceph-volume inventory Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c Device Path               Size         rotates available Model name /dev/sdc                  232.83 GB    True    True      SAMSUNG HE253GJ /dev/sda                  232.83 GB    True    False     SAMSUNG HE253GJ /dev/sdb                  465.76 GB    True    False     WDC WD5003ABYX-1 ============== Octopus ============== [root@mostha1 ~]# cephadm --image quay.io/ceph/ceph:v15 ceph-volume inventory Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c Device Path               Size         rotates available Model name /dev/sdc                  232.83 GB    True    True      SAMSUNG HE253GJ /dev/sda                  232.83 GB    True    False     SAMSUNG HE253GJNautilus /dev/sdb                  465.76 GB    True    False     WDC WD5003ABYX-1 ============== Pacific ============== [root@mostha1 ~]# cephadm --image quay.io/ceph/ceph:v16 ceph-volume inventory Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c Device Path               Size         Device nodes    rotates available Model name ============== Quincy ============== [root@mostha1 ~]# cephadm --image quay.io/ceph/ceph:v17 ceph-volume inventory Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c Device Path               Size         Device nodes    rotates available Model name ============== Reef ============== [root@mostha1 ~]# cephadm --image quay.io/ceph/ceph:v18 ceph-volume inventory Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c Device Path               Size         Device nodes    rotates available Model name Could it be related to deprecated hardware support in Ceph with SATA drives ? Patrick Le 11/10/2023 à 17:38, Eugen Block a écrit :

That's really strange. Just out of curiosity, have you tried Quincy (and/or Reef) as well? I don't recall what inventory does in the background exactly, I believe Adam King mentioned that in some thread, maybe that can help here. I'll search for that thread tomorrow.

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Hi Eugen,

[root@mostha1 ~]# rpm -q cephadm cephadm-16.2.14-0.el8.noarch

Log associated to the

2023-10-11 16:16:02,167 7f820515fb80 DEBUG -------------------------------------------------------------------------------- cephadm ['gather-facts'] 2023-10-11 16:16:02,208 7f820515fb80 DEBUG /bin/podman: 4.4.1 2023-10-11 16:16:02,313 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,317 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,322 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,326 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,329 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,333 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:04,474 7ff2a5c08b80 DEBUG -------------------------------------------------------------------------------- cephadm ['ceph-volume', 'inventory'] 2023-10-11 16:16:04,516 7ff2a5c08b80 DEBUG /usr/bin/podman: 4.4.1 2023-10-11 16:16:04,520 7ff2a5c08b80 DEBUG Using default config: /etc/ceph/ceph.conf 2023-10-11 16:16:04,573 7ff2a5c08b80 DEBUG /usr/bin/podman: 0d28d71358d7,445.8MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 2084faaf4d54,13.27MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 61073c53805d,512.7MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 6b9f0b72d668,361.1MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 7493a28808ad,163.7MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: a89672a3accf,59.22MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: b45271cc9726,54.24MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: e00ec13ab138,707.3MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,35.55MB / 50.32GB 2023-10-11 16:16:04,630 7ff2a5c08b80 DEBUG /usr/bin/podman: 0d28d71358d7,1.28% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 2084faaf4d54,0.00% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 61073c53805d,1.19% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 6b9f0b72d668,1.03% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 7493a28808ad,0.78% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: a89672a3accf,0.11% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: b45271cc9726,1.35% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: e00ec13ab138,0.43% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,0.02% 2023-10-11 16:16:04,634 7ff2a5c08b80 INFO Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c 2023-10-11 16:16:04,691 7ff2a5c08b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:16:04,692 7ff2a5c08b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:c08064dde4bba4e72a1f55d90ca32df9ef5aafab82efe2e0a0722444a5aaacca 2023-10-11 16:16:04,692 7ff2a5c08b80 DEBUG /usr/bin/podman: docker.io/ceph/ceph@sha256:056637972a107df4096f10951e4216b21fcd8ae0b9fb4552e628d35df3f61139 2023-10-11 16:16:04,694 7ff2a5c08b80 INFO Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:16:05,094 7ff2a5c08b80 DEBUG stat: 167 167 2023-10-11 16:16:05,903 7ff2a5c08b80 DEBUG Acquiring lock 140679815723776 on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:16:05,903 7ff2a5c08b80 DEBUG Lock 140679815723776 acquired on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:16:05,929 7ff2a5c08b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:05,933 7ff2a5c08b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:06,700 7ff2a5c08b80 DEBUG /usr/bin/podman: 2023-10-11 16:16:06,701 7ff2a5c08b80 DEBUG /usr/bin/podman: Device Path               Size         Device nodes    rotates available Model name

I have only one version of cephadm in /var/lib/ceph/{fsid} : [root@mostha1 ~]# ls -lrt /var/lib/ceph/250f9864-0142-11ee-8e5f-00266cf8869c/cephadm* -rw-r--r-- 1 root root 350889 28 sept. 16:39 /var/lib/ceph/250f9864-0142-11ee-8e5f-00266cf8869c/cephadm.f6868821c084cd9740b59c7c5eb59f0dd47f6e3b1e6fecb542cb44134ace8d78

Running " python3 /var/lib/ceph/250f9864-0142-11ee-8e5f-00266cf8869c/cephadm.f6868821c084cd9740b59c7c5eb59f0dd47f6e3b1e6fecb542cb44134ace8d78 ceph-volume inventory" give the same output and the same log (execpt the valu of the lock):

2023-10-11 16:21:35,965 7f467cf31b80 DEBUG -------------------------------------------------------------------------------- cephadm ['ceph-volume', 'inventory'] 2023-10-11 16:21:36,009 7f467cf31b80 DEBUG /usr/bin/podman: 4.4.1 2023-10-11 16:21:36,012 7f467cf31b80 DEBUG Using default config: /etc/ceph/ceph.conf 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 0d28d71358d7,452.1MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 2084faaf4d54,13.27MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 61073c53805d,513.6MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 6b9f0b72d668,322.4MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 7493a28808ad,164MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: a89672a3accf,58.5MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: b45271cc9726,54.69MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: e00ec13ab138,707.1MB / 50.32GB 2023-10-11 16:21:36,068 7f467cf31b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,36.28MB / 50.32GB 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 0d28d71358d7,1.27% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 2084faaf4d54,0.00% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 61073c53805d,1.16% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 6b9f0b72d668,1.02% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 7493a28808ad,0.78% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: a89672a3accf,0.11% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: b45271cc9726,1.35% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: e00ec13ab138,0.41% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,0.02% 2023-10-11 16:21:36,128 7f467cf31b80 INFO Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c 2023-10-11 16:21:36,186 7f467cf31b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:21:36,187 7f467cf31b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:c08064dde4bba4e72a1f55d90ca32df9ef5aafab82efe2e0a0722444a5aaacca 2023-10-11 16:21:36,187 7f467cf31b80 DEBUG /usr/bin/podman: docker.io/ceph/ceph@sha256:056637972a107df4096f10951e4216b21fcd8ae0b9fb4552e628d35df3f61139 2023-10-11 16:21:36,189 7f467cf31b80 INFO Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:21:36,549 7f467cf31b80 DEBUG stat: 167 167 2023-10-11 16:21:36,942 7f467cf31b80 DEBUG Acquiring lock 139940396923424 on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:21:36,942 7f467cf31b80 DEBUG Lock 139940396923424 acquired on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:21:36,969 7f467cf31b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:21:36,972 7f467cf31b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:21:37,749 7f467cf31b80 DEBUG /usr/bin/podman: 2023-10-11 16:21:37,750 7f467cf31b80 DEBUG /usr/bin/podman: Device Path               Size         Device nodes    rotates available Model name

Patrick

Le 11/10/2023 à 15:59, Eugen Block a écrit :

...

Can you check which cephadm version is installed on the host? And then please add (only the relevant) output from the cephadm.log when you run the inventory (without the --image <octopus>). Sometimes the version mismatch on the host and the one the orchestrator uses can cause some disruptions. You could try the same with the latest cephadm you have in /var/lib/ceph/${fsid}/ (ls -lrt /var/lib/ceph/${fsid}/cephadm.*). I mentioned that in this thread [1]. So you could try the following:

$ chmod +x /var/lib/ceph/{fsid}/cephadm.{latest}

$ python3 /var/lib/ceph/{fsid}/cephadm.{latest} ceph-volume inventory

Does the output differ? Paste the relevant cephadm.log from that attempt as well.

[1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LASBJCSPFGDY...

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Hi Eugen,

first many thanks for the time spent on this problem.

"ceph osd purge 2 --force --yes-i-really-mean-it" works and clean all the bas status.

*[root@mostha1 ~]# cephadm shell *Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e * * *[ceph: root@mostha1 /]# ceph osd purge 2 --force --yes-i-really-mean-it * purged osd.2 * * *[ceph: root@mostha1 /]# ceph osd tree* ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT PRI-AFF -1         1.72823  root default -5         0.45477      host dean  0    hdd  0.22739          osd.0         up   1.00000 1.00000  4    hdd  0.22739          osd.4         up   1.00000 1.00000 -9         0.22739      host ekman  6    hdd  0.22739          osd.6         up   1.00000 1.00000 -7         0.45479      host mostha1  5    hdd  0.45479          osd.5         up   1.00000 1.00000 -3         0.59128      host mostha2  1    hdd  0.22739          osd.1         up   1.00000 1.00000  3    hdd  0.36389          osd.3         up   1.00000 1.00000 * * *[ceph: root@mostha1 /]# lsblk* NAME MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT sda 8:0    1 232.9G  0 disk |-sda1 8:1    1   3.9G  0 part /rootfs/boot |-sda2 8:2    1   3.9G  0 part [SWAP] `-sda3 8:3    1   225G  0 part |-al8vg-rootvol 253:0    0  48.8G  0 lvm  /rootfs |-al8vg-homevol 253:2    0   9.8G  0 lvm  /rootfs/home |-al8vg-tmpvol 253:3    0   9.8G  0 lvm  /rootfs/tmp `-al8vg-varvol 253:4    0  19.8G  0 lvm  /rootfs/var sdb 8:16   1 465.8G  0 disk `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 253:1    0 465.8G  0 lvm sdc 8:32   1 232.9G  0 disk

"cephadm ceph-volume inventory" returns nothing:

*[root@mostha1 ~]# cephadm ceph-volume inventory ** *Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e

Device Path               Size         Device nodes rotates available Model name

[root@mostha1 ~]#

But running the same command within cephadm 15.2.17 works:

*[root@mostha1 ~]# cephadm --image 93146564743f ceph-volume inventory* Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c

Device Path               Size         rotates available Model name /dev/sdc                  232.83 GB    True    True SAMSUNG HE253GJ /dev/sda                  232.83 GB    True    False SAMSUNG HE253GJ /dev/sdb                  465.76 GB    True    False     WDC WD5003ABYX-1

[root@mostha1 ~]#

*[root@mostha1 ~]# podman images -a** *REPOSITORY                        TAG         IMAGE ID CREATED        SIZE quay.io/ceph/ceph                 v16.2.14    f13d80acdbb5 2 weeks ago    1.21 GB quay.io/ceph/ceph                 v15.2.17    93146564743f 14 months ago  1.24 GB ....

Patrick

Le 11/10/2023 à 15:14, Eugen Block a écrit :

...

Your response is a bit confusing since it seems to be mixed up with the previous answer. So you still need to remove the OSD properly, so purge it from the crush tree:

ceph osd purge 2 --force --yes-i-really-mean-it (only in a test cluster!)

If everything is clean (OSD has been removed, disk has been zapped, lsblk shows no LVs for that disk) you can check the inventory:

cephadm ceph-volume inventory

Please also add the output of 'ceph orch ls osd --export'.

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Hi Eugen,

- the OS is Alma Linux 8 with latests updates.

- this morning I've worked with ceph-volume but it ends with a strange final state. I was connected on host mostha1 where /dev/sdc was not reconized. These are the steps followed based on the ceph-volume documentation I've read: [root@mostha1 ~]# cephadm shell [ceph: root@mostha1 /]# ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring [ceph: root@mostha1 /]# ceph-volume lvm prepare --bluestore --data /dev/sdc

Now lsblk command shows sdc as an osd: .... sdb 8:16   1 465.8G  0 disk `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 253:1    0 465.8G  0 lvm sdc 8:32   1 232.9G  0 disk `-ceph--b27d7a07--278d--4ee2--b84e--53256ef8de4c-osd--block--45c8e92c--caf9--4fe7--9a42--7b45a0794632 253:5    0 232.8G  0 lvm

Then I've tried to activate this osd but it fails as in podman I have not access to systemctl:

[ceph: root@mostha1 /]# ceph-volume lvm activate 2 45c8e92c-caf9-4fe7-9a42-7b45a0794632 ..... Running command: /usr/bin/systemctl start ceph-osd@2  stderr: Failed to connect to bus: No such file or directory -->  RuntimeError: command returned non-zero exit status: 1 [ceph: root@mostha1 /]# ceph osd tree

And now I have now I have a strange status for this osd.2:

[ceph: root@mostha1 /]# ceph osd tree ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT PRI-AFF -1         1.72823  root default -5         0.45477      host dean  0    hdd  0.22739          osd.0         up   1.00000 1.00000  4    hdd  0.22739          osd.4         up   1.00000 1.00000 -9         0.22739      host ekman  6    hdd  0.22739          osd.6         up   1.00000 1.00000 -7         0.45479      host mostha1  5    hdd  0.45479          osd.5         up   1.00000 1.00000 -3         0.59128      host mostha2  1    hdd  0.22739          osd.1         up   1.00000 1.00000  3    hdd  0.36389          osd.3         up   1.00000 1.00000  2               0  osd.2               down         0 1.00000

I've tried to destroy the osd as you suggest but even if the command returns no error I still have this osd even if "lsblk" do not show any more /dev/sdc as a ceph osd device.

*[ceph: root@mostha1 /]# ceph-volume lvm zap --destroy /dev/sdc** *--> Zapping: /dev/sdc --> Zapping lvm member /dev/sdc. lv_path is /dev/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c/osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632 --> Unmounting /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-2  stderr: umount: /var/lib/ceph/osd/ceph-2 unmounted Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c/osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632 bs=1M count=10 conv=fsync  stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.575633 s, 18.2 MB/s --> Only 1 LV left in VG, will proceed to destroy volume group ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c  stderr: Removing ceph--b27d7a07--278d--4ee2--b84e--53256ef8de4c-osd--block--45c8e92c--caf9--4fe7--9a42--7b45a0794632 (253:1)  stderr: Releasing logical volume "osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632"  stderr: Archiving volume group "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" metadata (seqno 5).  stdout: Logical volume "osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632" successfully removed.  stderr: Removing physical volume "/dev/sdc" from volume group "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c"  stdout: Volume group "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" successfully removed  stderr: Creating volume group backup "/etc/lvm/backup/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" (seqno 6). Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/sdc  stdout: Labels on physical volume "/dev/sdc" successfully wiped. Running command: /usr/bin/dd if=/dev/zero of=/dev/sdc bs=1M count=10 conv=fsync  stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.590652 s, 17.8 MB/s *--> Zapping successful for: <Raw Device: /dev/sdc>* * * *[ceph: root@mostha1 /]# ceph osd tree** *ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT PRI-AFF -1         1.72823  root default -5         0.45477      host dean  0    hdd  0.22739          osd.0         up   1.00000 1.00000  4    hdd  0.22739          osd.4         up   1.00000 1.00000 -9         0.22739      host ekman  6    hdd  0.22739          osd.6         up   1.00000 1.00000 -7         0.45479      host mostha1  5    hdd  0.45479          osd.5         up   1.00000 1.00000 -3         0.59128      host mostha2  1    hdd  0.22739          osd.1         up   1.00000 1.00000  3    hdd  0.36389          osd.3         up   1.00000 1.00000  2               0  osd.2               down         0 1.00000 * * *[ceph: root@mostha1 /]# lsblk** *NAME MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT sda 8:0    1 232.9G  0 disk |-sda1 8:1    1   3.9G  0 part /rootfs/boot |-sda2 8:2    1   3.9G  0 part [SWAP] `-sda3 8:3    1   225G  0 part |-al8vg-rootvol 253:0    0  48.8G  0 lvm  /rootfs |-al8vg-homevol 253:3    0   9.8G  0 lvm  /rootfs/home |-al8vg-tmpvol 253:4    0   9.8G  0 lvm  /rootfs/tmp `-al8vg-varvol 253:5    0  19.8G  0 lvm  /rootfs/var sdb 8:16   1 465.8G  0 disk `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 253:2    0 465.8G  0 lvm *sdc *

Patrick Le 11/10/2023 à 11:00, Eugen Block a écrit : > Hi, > > just wondering if 'ceph-volume lvm zap --destroy /dev/sdc' would > help here. From your previous output you didn't specify the > --destroy flag. > Which cephadm version is installed on the host? Did you also > upgrade the OS when moving to Pacific? (Sorry if I missed that. > > > Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>: > >> Le 02/10/2023 à 18:22, Patrick Bégou a écrit : >>> Hi all, >>> >>> still stuck with this problem. >>> >>> I've deployed octopus and all my HDD have been setup as osd. >>> Fine. >>> I've upgraded to pacific and 2 osd have failed. They have been >>> automatically removed and upgrade finishes. Cluster Health is >>> finaly OK, no data loss. >>> >>> But now I cannot re-add these osd with pacific (I had previous >>> troubles on these old HDDs, lost one osd in octopus and was >>> able to reset and re-add it). >>> >>> I've tried manually to add the first osd on the node where it >>> is located, following >>> https://docs.ceph.com/en/pacific/rados/operations/bluestore-migration/ >>> (not sure it's the best idea...) but it fails too. This node >>> was the one used for deploying the cluster. >>> >>> [ceph: root@mostha1 /]# *ceph-volume lvm zap /dev/sdc* >>> --> Zapping: /dev/sdc >>> --> --destroy was not specified, but zapping a whole device >>> will remove the partition table >>> Running command: /usr/bin/dd if=/dev/zero of=/dev/sdc bs=1M >>> count=10 conv=fsync >>>  stderr: 10+0 records in >>> 10+0 records out >>> 10485760 bytes (10 MB, 10 MiB) copied, 0.663425 s, 15.8 MB/s >>> --> Zapping successful for: <Raw Device: /dev/sdc> >>> >>> >>> [ceph: root@mostha1 /]# *ceph-volume lvm create --bluestore >>> --data /dev/sdc* >>> Running command: /usr/bin/ceph-authtool --gen-print-key >>> Running command: /usr/bin/ceph --cluster ceph --name >>> client.bootstrap-osd --keyring >>> /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new >>> 9f1eb8ee-41e6-4350-ad73-1be21234ec7c >>>  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 auth: >>> unable to find a keyring on >>> /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or >>> directory >>>  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 >>> AuthRegistry(0x7fb4e405c4d8) no keyring found at >>> /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx >>>  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 auth: >>> unable to find a keyring on >>> /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or >>> directory >>>  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 >>> AuthRegistry(0x7fb4e40601d0) no keyring found at >>> /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx >>>  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 auth: >>> unable to find a keyring on >>> /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or >>> directory >>>  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 >>> AuthRegistry(0x7fb4eb8bee90) no keyring found at >>> /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx >>>  stderr: 2023-10-02T16:09:29.858+0000 7fb4e965c700 -1 >>> monclient(hunting): handle_auth_bad_method server >>> allowed_methods [2] but i only support [1] >>>  stderr: 2023-10-02T16:09:29.858+0000 7fb4e9e5d700 -1 >>> monclient(hunting): handle_auth_bad_method server >>> allowed_methods [2] but i only support [1] >>>  stderr: 2023-10-02T16:09:29.858+0000 7fb4e8e5b700 -1 >>> monclient(hunting): handle_auth_bad_method server >>> allowed_methods [2] but i only support [1] >>>  stderr: 2023-10-02T16:09:29.858+0000 7fb4eb8c0700 -1 >>> monclient: authenticate NOTE: no keyring found; disabled cephx >>> authentication >>>  stderr: [errno 13] RADOS permission denied (error connecting >>> to the cluster) >>> -->  RuntimeError: Unable to create a new OSD id >>> >>> Any idea of what is wrong ? >>> >>> Thanks >>> >>> Patrick >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> >> I'm still trying to understand what can be wrong or how to >> debug this situation where Ceph cannot see the devices. >> >> The device :dev/sdc exists: >> >>    [root@mostha1 ~]# cephadm shell lsmcli ldl >>    Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c >>    Using recent ceph image >> quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e >>    Path     | SCSI VPD 0x83    | Link Type | Serial Number | >> Health >>    Status ceph osd purge 2 --force --yes-i-really-mean-it >> ------------------------------------------------------------------------- >>    /dev/sda | 50024e92039e4f1c | PATA/SATA | S2B5J90ZA10142 | Good >>    /dev/sdb | 50014ee0ad5953c9 | PATA/SATA | WD-WMAYP0982329 | >> Good >>    /dev/sdc | 50024e920387fa2c | PATA/SATA | S2B5J90ZA02494 | Good >> >> But I cannot do anything with it: >> >>    [root@mostha1 ~]# cephadm shell ceph orch device zap >>    mostha1.legi.grenoble-inp.fr /dev/sdc --force >>    Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c >>    Using recent ceph image >> quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e >>    Error EINVAL: Device path '/dev/sdc' not found on host >>    'mostha1.legi.grenoble-inp.fr' >> >> Since I moved from octopus to Pacific. >> >> Patrick >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Good catch, and I found the thread I had in my mind, it was this exact one. :-D Anyway, can you share the ceph-volume.log from the working and the not working attempt? I tried to look for something significant in the pacific release notes for 16.2.11, and there were some changes to ceph-volume, but I'm not sure what it could be. Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

I've ran additional tests with Pacific releases and with "ceph-volume inventory" things went wrong with the first v16.11 release (v16.2.11-20230125)

=================== Ceph v16.2.10-20220920 =======================

Device Path               Size         rotates available Model name /dev/sdc                  232.83 GB    True    True      SAMSUNG HE253GJ /dev/sda                  232.83 GB    True    False     SAMSUNG HE253GJ /dev/sdb                  465.76 GB    True    False     WDC WD5003ABYX-1

=================== Ceph v16.2.11-20230125 =======================

Device Path               Size         Device nodes    rotates available Model name

May be this could help to see what has changed ?

Patrick

Le 11/10/2023 à 17:38, Eugen Block a écrit :

...

That's really strange. Just out of curiosity, have you tried Quincy (and/or Reef) as well? I don't recall what inventory does in the background exactly, I believe Adam King mentioned that in some thread, maybe that can help here. I'll search for that thread tomorrow.

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Hi Eugen,

[root@mostha1 ~]# rpm -q cephadm cephadm-16.2.14-0.el8.noarch

Log associated to the

2023-10-11 16:16:02,167 7f820515fb80 DEBUG -------------------------------------------------------------------------------- cephadm ['gather-facts'] 2023-10-11 16:16:02,208 7f820515fb80 DEBUG /bin/podman: 4.4.1 2023-10-11 16:16:02,313 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,317 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,322 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,326 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,329 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,333 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:04,474 7ff2a5c08b80 DEBUG -------------------------------------------------------------------------------- cephadm ['ceph-volume', 'inventory'] 2023-10-11 16:16:04,516 7ff2a5c08b80 DEBUG /usr/bin/podman: 4.4.1 2023-10-11 16:16:04,520 7ff2a5c08b80 DEBUG Using default config: /etc/ceph/ceph.conf 2023-10-11 16:16:04,573 7ff2a5c08b80 DEBUG /usr/bin/podman: 0d28d71358d7,445.8MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 2084faaf4d54,13.27MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 61073c53805d,512.7MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 6b9f0b72d668,361.1MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 7493a28808ad,163.7MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: a89672a3accf,59.22MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: b45271cc9726,54.24MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: e00ec13ab138,707.3MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,35.55MB / 50.32GB 2023-10-11 16:16:04,630 7ff2a5c08b80 DEBUG /usr/bin/podman: 0d28d71358d7,1.28% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 2084faaf4d54,0.00% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 61073c53805d,1.19% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 6b9f0b72d668,1.03% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 7493a28808ad,0.78% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: a89672a3accf,0.11% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: b45271cc9726,1.35% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: e00ec13ab138,0.43% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,0.02% 2023-10-11 16:16:04,634 7ff2a5c08b80 INFO Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c 2023-10-11 16:16:04,691 7ff2a5c08b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:16:04,692 7ff2a5c08b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:c08064dde4bba4e72a1f55d90ca32df9ef5aafab82efe2e0a0722444a5aaacca 2023-10-11 16:16:04,692 7ff2a5c08b80 DEBUG /usr/bin/podman: docker.io/ceph/ceph@sha256:056637972a107df4096f10951e4216b21fcd8ae0b9fb4552e628d35df3f61139 2023-10-11 16:16:04,694 7ff2a5c08b80 INFO Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:16:05,094 7ff2a5c08b80 DEBUG stat: 167 167 2023-10-11 16:16:05,903 7ff2a5c08b80 DEBUG Acquiring lock 140679815723776 on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:16:05,903 7ff2a5c08b80 DEBUG Lock 140679815723776 acquired on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:16:05,929 7ff2a5c08b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:05,933 7ff2a5c08b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:06,700 7ff2a5c08b80 DEBUG /usr/bin/podman: 2023-10-11 16:16:06,701 7ff2a5c08b80 DEBUG /usr/bin/podman: Device Path               Size         Device nodes    rotates available Model name

I have only one version of cephadm in /var/lib/ceph/{fsid} : [root@mostha1 ~]# ls -lrt /var/lib/ceph/250f9864-0142-11ee-8e5f-00266cf8869c/cephadm* -rw-r--r-- 1 root root 350889 28 sept. 16:39 /var/lib/ceph/250f9864-0142-11ee-8e5f-00266cf8869c/cephadm.f6868821c084cd9740b59c7c5eb59f0dd47f6e3b1e6fecb542cb44134ace8d78

Running " python3 /var/lib/ceph/250f9864-0142-11ee-8e5f-00266cf8869c/cephadm.f6868821c084cd9740b59c7c5eb59f0dd47f6e3b1e6fecb542cb44134ace8d78 ceph-volume inventory" give the same output and the same log (execpt the valu of the lock):

2023-10-11 16:21:35,965 7f467cf31b80 DEBUG -------------------------------------------------------------------------------- cephadm ['ceph-volume', 'inventory'] 2023-10-11 16:21:36,009 7f467cf31b80 DEBUG /usr/bin/podman: 4.4.1 2023-10-11 16:21:36,012 7f467cf31b80 DEBUG Using default config: /etc/ceph/ceph.conf 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 0d28d71358d7,452.1MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 2084faaf4d54,13.27MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 61073c53805d,513.6MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 6b9f0b72d668,322.4MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 7493a28808ad,164MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: a89672a3accf,58.5MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: b45271cc9726,54.69MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: e00ec13ab138,707.1MB / 50.32GB 2023-10-11 16:21:36,068 7f467cf31b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,36.28MB / 50.32GB 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 0d28d71358d7,1.27% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 2084faaf4d54,0.00% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 61073c53805d,1.16% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 6b9f0b72d668,1.02% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 7493a28808ad,0.78% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: a89672a3accf,0.11% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: b45271cc9726,1.35% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: e00ec13ab138,0.41% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,0.02% 2023-10-11 16:21:36,128 7f467cf31b80 INFO Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c 2023-10-11 16:21:36,186 7f467cf31b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:21:36,187 7f467cf31b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:c08064dde4bba4e72a1f55d90ca32df9ef5aafab82efe2e0a0722444a5aaacca 2023-10-11 16:21:36,187 7f467cf31b80 DEBUG /usr/bin/podman: docker.io/ceph/ceph@sha256:056637972a107df4096f10951e4216b21fcd8ae0b9fb4552e628d35df3f61139 2023-10-11 16:21:36,189 7f467cf31b80 INFO Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:21:36,549 7f467cf31b80 DEBUG stat: 167 167 2023-10-11 16:21:36,942 7f467cf31b80 DEBUG Acquiring lock 139940396923424 on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:21:36,942 7f467cf31b80 DEBUG Lock 139940396923424 acquired on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:21:36,969 7f467cf31b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:21:36,972 7f467cf31b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:21:37,749 7f467cf31b80 DEBUG /usr/bin/podman: 2023-10-11 16:21:37,750 7f467cf31b80 DEBUG /usr/bin/podman: Device Path               Size         Device nodes    rotates available Model name

Patrick

Le 11/10/2023 à 15:59, Eugen Block a écrit :

...

Can you check which cephadm version is installed on the host? And then please add (only the relevant) output from the cephadm.log when you run the inventory (without the --image <octopus>). Sometimes the version mismatch on the host and the one the orchestrator uses can cause some disruptions. You could try the same with the latest cephadm you have in /var/lib/ceph/${fsid}/ (ls -lrt /var/lib/ceph/${fsid}/cephadm.*). I mentioned that in this thread [1]. So you could try the following:

$ chmod +x /var/lib/ceph/{fsid}/cephadm.{latest}

$ python3 /var/lib/ceph/{fsid}/cephadm.{latest} ceph-volume inventory

Does the output differ? Paste the relevant cephadm.log from that attempt as well.

[1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LASBJCSPFGDY...

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Hi Eugen,

first many thanks for the time spent on this problem.

"ceph osd purge 2 --force --yes-i-really-mean-it" works and clean all the bas status.

*[root@mostha1 ~]# cephadm shell *Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e * * *[ceph: root@mostha1 /]# ceph osd purge 2 --force --yes-i-really-mean-it * purged osd.2 * * *[ceph: root@mostha1 /]# ceph osd tree* ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT PRI-AFF -1         1.72823  root default -5         0.45477      host dean  0    hdd  0.22739          osd.0         up   1.00000 1.00000  4    hdd  0.22739          osd.4         up   1.00000 1.00000 -9         0.22739      host ekman  6    hdd  0.22739          osd.6         up   1.00000 1.00000 -7         0.45479      host mostha1  5    hdd  0.45479          osd.5         up   1.00000 1.00000 -3         0.59128      host mostha2  1    hdd  0.22739          osd.1         up   1.00000 1.00000  3    hdd  0.36389          osd.3         up   1.00000 1.00000 * * *[ceph: root@mostha1 /]# lsblk* NAME MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT sda 8:0    1 232.9G  0 disk |-sda1 8:1    1   3.9G  0 part /rootfs/boot |-sda2 8:2    1   3.9G  0 part [SWAP] `-sda3 8:3    1   225G  0 part |-al8vg-rootvol 253:0    0  48.8G  0 lvm  /rootfs |-al8vg-homevol 253:2    0   9.8G  0 lvm  /rootfs/home |-al8vg-tmpvol 253:3    0   9.8G  0 lvm  /rootfs/tmp `-al8vg-varvol 253:4    0  19.8G  0 lvm  /rootfs/var sdb 8:16   1 465.8G  0 disk `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 253:1    0 465.8G  0 lvm sdc 8:32   1 232.9G  0 disk

"cephadm ceph-volume inventory" returns nothing:

*[root@mostha1 ~]# cephadm ceph-volume inventory ** *Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e

Device Path               Size         Device nodes rotates available Model name

[root@mostha1 ~]#

But running the same command within cephadm 15.2.17 works:

*[root@mostha1 ~]# cephadm --image 93146564743f ceph-volume inventory* Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c

Device Path               Size         rotates available Model name /dev/sdc                  232.83 GB    True    True SAMSUNG HE253GJ /dev/sda                  232.83 GB    True    False SAMSUNG HE253GJ /dev/sdb                  465.76 GB    True    False     WDC WD5003ABYX-1

[root@mostha1 ~]#

*[root@mostha1 ~]# podman images -a** *REPOSITORY                        TAG         IMAGE ID CREATED        SIZE quay.io/ceph/ceph                 v16.2.14    f13d80acdbb5 2 weeks ago    1.21 GB quay.io/ceph/ceph                 v15.2.17    93146564743f 14 months ago  1.24 GB ....

Patrick

Le 11/10/2023 à 15:14, Eugen Block a écrit :

...

Your response is a bit confusing since it seems to be mixed up with the previous answer. So you still need to remove the OSD properly, so purge it from the crush tree:

ceph osd purge 2 --force --yes-i-really-mean-it (only in a test cluster!)

If everything is clean (OSD has been removed, disk has been zapped, lsblk shows no LVs for that disk) you can check the inventory:

cephadm ceph-volume inventory

Please also add the output of 'ceph orch ls osd --export'.

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

> Hi Eugen, > > - the OS is Alma Linux 8 with latests updates. > > - this morning I've worked with ceph-volume but it ends with a > strange final state. I was connected on host mostha1 where > /dev/sdc was not reconized. These are the steps followed based > on the ceph-volume documentation I've read: > [root@mostha1 ~]# cephadm shell > [ceph: root@mostha1 /]# ceph auth get client.bootstrap-osd > > /var/lib/ceph/bootstrap-osd/ceph.keyring > [ceph: root@mostha1 /]# ceph-volume lvm prepare --bluestore > --data /dev/sdc > > Now lsblk command shows sdc as an osd: > .... > sdb 8:16   1 465.8G  0 disk > `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 253:1    0 465.8G  0 > lvm > sdc 8:32   1 232.9G  0 disk > `-ceph--b27d7a07--278d--4ee2--b84e--53256ef8de4c-osd--block--45c8e92c--caf9--4fe7--9a42--7b45a0794632 253:5    0 232.8G  0 > lvm > > Then I've tried to activate this osd but it fails as in podman > I have not access to systemctl: > > [ceph: root@mostha1 /]# ceph-volume lvm activate 2 > 45c8e92c-caf9-4fe7-9a42-7b45a0794632 > ..... > Running command: /usr/bin/systemctl start ceph-osd@2 >  stderr: Failed to connect to bus: No such file or directory > -->  RuntimeError: command returned non-zero exit status: 1 > [ceph: root@mostha1 /]# ceph osd tree > > And now I have now I have a strange status for this osd.2: > > [ceph: root@mostha1 /]# ceph osd tree > ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT PRI-AFF > -1         1.72823  root default > -5         0.45477      host dean >  0    hdd  0.22739          osd.0         up   1.00000 1.00000 >  4    hdd  0.22739          osd.4         up   1.00000 1.00000 > -9         0.22739      host ekman >  6    hdd  0.22739          osd.6         up   1.00000 1.00000 > -7         0.45479      host mostha1 >  5    hdd  0.45479          osd.5         up   1.00000 1.00000 > -3         0.59128      host mostha2 >  1    hdd  0.22739          osd.1         up   1.00000 1.00000 >  3    hdd  0.36389          osd.3         up   1.00000 1.00000 >  2               0  osd.2               down         0 1.00000 > > I've tried to destroy the osd as you suggest but even if the > command returns no error I still have this osd even if "lsblk" > do not show any more /dev/sdc as a ceph osd device. > > *[ceph: root@mostha1 /]# ceph-volume lvm zap --destroy /dev/sdc** > *--> Zapping: /dev/sdc > --> Zapping lvm member /dev/sdc. lv_path is > /dev/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c/osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632 > --> Unmounting /var/lib/ceph/osd/ceph-2 > Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-2 >  stderr: umount: /var/lib/ceph/osd/ceph-2 unmounted > Running command: /usr/bin/dd if=/dev/zero > of=/dev/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c/osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632 bs=1M count=10 > conv=fsync >  stderr: 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.575633 s, 18.2 MB/s > --> Only 1 LV left in VG, will proceed to destroy volume group > ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c > Running command: nsenter --mount=/rootfs/proc/1/ns/mnt > --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net > --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f > ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c >  stderr: Removing > ceph--b27d7a07--278d--4ee2--b84e--53256ef8de4c-osd--block--45c8e92c--caf9--4fe7--9a42--7b45a0794632 > (253:1) >  stderr: Releasing logical volume > "osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632" >  stderr: Archiving volume group > "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" metadata (seqno 5). >  stdout: Logical volume > "osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632" successfully > removed. >  stderr: Removing physical volume "/dev/sdc" from volume group > "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" >  stdout: Volume group > "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" successfully removed >  stderr: Creating volume group backup > "/etc/lvm/backup/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" > (seqno 6). > Running command: nsenter --mount=/rootfs/proc/1/ns/mnt > --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net > --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/sdc >  stdout: Labels on physical volume "/dev/sdc" successfully wiped. > Running command: /usr/bin/dd if=/dev/zero of=/dev/sdc bs=1M > count=10 conv=fsync >  stderr: 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.590652 s, 17.8 MB/s > *--> Zapping successful for: <Raw Device: /dev/sdc>* > * > * > *[ceph: root@mostha1 /]# ceph osd tree** > *ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT PRI-AFF > -1         1.72823  root default > -5         0.45477      host dean >  0    hdd  0.22739          osd.0         up   1.00000 1.00000 >  4    hdd  0.22739          osd.4         up   1.00000 1.00000 > -9         0.22739      host ekman >  6    hdd  0.22739          osd.6         up   1.00000 1.00000 > -7         0.45479      host mostha1 >  5    hdd  0.45479          osd.5         up   1.00000 1.00000 > -3         0.59128      host mostha2 >  1    hdd  0.22739          osd.1         up   1.00000 1.00000 >  3    hdd  0.36389          osd.3         up   1.00000 1.00000 >  2               0  osd.2               down         0 1.00000 > * > * > *[ceph: root@mostha1 /]# lsblk** > *NAME MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT > sda 8:0    1 232.9G  0 disk > |-sda1 8:1    1   3.9G  0 part /rootfs/boot > |-sda2 8:2    1   3.9G  0 part [SWAP] > `-sda3 8:3    1   225G  0 part > |-al8vg-rootvol 253:0    0  48.8G  0 lvm  /rootfs > |-al8vg-homevol 253:3    0   9.8G  0 lvm  /rootfs/home > |-al8vg-tmpvol 253:4    0   9.8G  0 lvm  /rootfs/tmp > `-al8vg-varvol 253:5    0  19.8G  0 lvm  /rootfs/var > sdb 8:16   1 465.8G  0 disk > `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 253:2    0 465.8G  0 > lvm > *sdc * > > Patrick > Le 11/10/2023 à 11:00, Eugen Block a écrit : >> Hi, >> >> just wondering if 'ceph-volume lvm zap --destroy /dev/sdc' >> would help here. From your previous output you didn't specify >> the --destroy flag. >> Which cephadm version is installed on the host? Did you also >> upgrade the OS when moving to Pacific? (Sorry if I missed that. >> >> >> Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>: >> >>> Le 02/10/2023 à 18:22, Patrick Bégou a écrit : >>>> Hi all, >>>> >>>> still stuck with this problem. >>>> >>>> I've deployed octopus and all my HDD have been setup as osd. Fine. >>>> I've upgraded to pacific and 2 osd have failed. They have >>>> been automatically removed and upgrade finishes. Cluster >>>> Health is finaly OK, no data loss. >>>> >>>> But now I cannot re-add these osd with pacific (I had >>>> previous troubles on these old HDDs, lost one osd in >>>> octopus and was able to reset and re-add it). >>>> >>>> I've tried manually to add the first osd on the node where >>>> it is located, following >>>> https://docs.ceph.com/en/pacific/rados/operations/bluestore-migration/ (not sure it's the best idea...) but it fails too. This node was the one used for deploying the >>>> cluster. >>>> >>>> [ceph: root@mostha1 /]# *ceph-volume lvm zap /dev/sdc* >>>> --> Zapping: /dev/sdc >>>> --> --destroy was not specified, but zapping a whole device >>>> will remove the partition table >>>> Running command: /usr/bin/dd if=/dev/zero of=/dev/sdc bs=1M >>>> count=10 conv=fsync >>>>  stderr: 10+0 records in >>>> 10+0 records out >>>> 10485760 bytes (10 MB, 10 MiB) copied, 0.663425 s, 15.8 MB/s >>>> --> Zapping successful for: <Raw Device: /dev/sdc> >>>> >>>> >>>> [ceph: root@mostha1 /]# *ceph-volume lvm create --bluestore >>>> --data /dev/sdc* >>>> Running command: /usr/bin/ceph-authtool --gen-print-key >>>> Running command: /usr/bin/ceph --cluster ceph --name >>>> client.bootstrap-osd --keyring >>>> /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new >>>> 9f1eb8ee-41e6-4350-ad73-1be21234ec7c >>>>  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 auth: >>>> unable to find a keyring on >>>> /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file >>>> or directory >>>>  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 >>>> AuthRegistry(0x7fb4e405c4d8) no keyring found at >>>> /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx >>>>  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 auth: >>>> unable to find a keyring on >>>> /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file >>>> or directory >>>>  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 >>>> AuthRegistry(0x7fb4e40601d0) no keyring found at >>>> /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx >>>>  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 auth: >>>> unable to find a keyring on >>>> /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file >>>> or directory >>>>  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 >>>> AuthRegistry(0x7fb4eb8bee90) no keyring found at >>>> /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx >>>>  stderr: 2023-10-02T16:09:29.858+0000 7fb4e965c700 -1 >>>> monclient(hunting): handle_auth_bad_method server >>>> allowed_methods [2] but i only support [1] >>>>  stderr: 2023-10-02T16:09:29.858+0000 7fb4e9e5d700 -1 >>>> monclient(hunting): handle_auth_bad_method server >>>> allowed_methods [2] but i only support [1] >>>>  stderr: 2023-10-02T16:09:29.858+0000 7fb4e8e5b700 -1 >>>> monclient(hunting): handle_auth_bad_method server >>>> allowed_methods [2] but i only support [1] >>>>  stderr: 2023-10-02T16:09:29.858+0000 7fb4eb8c0700 -1 >>>> monclient: authenticate NOTE: no keyring found; disabled >>>> cephx authentication >>>>  stderr: [errno 13] RADOS permission denied (error >>>> connecting to the cluster) >>>> -->  RuntimeError: Unable to create a new OSD id >>>> >>>> Any idea of what is wrong ? >>>> >>>> Thanks >>>> >>>> Patrick >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> >>> >>> I'm still trying to understand what can be wrong or how to >>> debug this situation where Ceph cannot see the devices. >>> >>> The device :dev/sdc exists: >>> >>>    [root@mostha1 ~]# cephadm shell lsmcli ldl >>>    Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c >>>    Using recent ceph image >>> quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e    Path     | SCSI VPD 0x83    | Link Type | Serial Number | >>> Health >>>    Status ceph osd purge 2 --force --yes-i-really-mean-it >>> -------------------------------------------------------------------------    /dev/sda | 50024e92039e4f1c | PATA/SATA | S2B5J90ZA10142 | >>> Good >>>    /dev/sdb | 50014ee0ad5953c9 | PATA/SATA | WD-WMAYP0982329 | Good >>>    /dev/sdc | 50024e920387fa2c | PATA/SATA | S2B5J90ZA02494 | Good >>> >>> But I cannot do anything with it: >>> >>>    [root@mostha1 ~]# cephadm shell ceph orch device zap >>>    mostha1.legi.grenoble-inp.fr /dev/sdc --force >>>    Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c >>>    Using recent ceph image >>> quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e    Error EINVAL: Device path '/dev/sdc' not found on >>> host >>>    'mostha1.legi.grenoble-inp.fr' >>> >>> Since I moved from octopus to Pacific. >>> >>> Patrick >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

There are no attachments. Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

Hi Eugen,

You will find in attachment cephadm.log and cepĥ-volume.log. Each contains the outputs for the 2 versions.  v16.2.10-20220920 is really more verbose or v16.2.11-20230125 does not execute all the detection process

Patrick

Le 12/10/2023 à 09:34, Eugen Block a écrit :

...

Good catch, and I found the thread I had in my mind, it was this exact one. :-D Anyway, can you share the ceph-volume.log from the working and the not working attempt? I tried to look for something significant in the pacific release notes for 16.2.11, and there were some changes to ceph-volume, but I'm not sure what it could be.

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

I've ran additional tests with Pacific releases and with "ceph-volume inventory" things went wrong with the first v16.11 release (v16.2.11-20230125)

=================== Ceph v16.2.10-20220920 =======================

Device Path               Size         rotates available Model name /dev/sdc                  232.83 GB    True    True      SAMSUNG HE253GJ /dev/sda                  232.83 GB    True    False     SAMSUNG HE253GJ /dev/sdb                  465.76 GB    True    False     WDC WD5003ABYX-1

=================== Ceph v16.2.11-20230125 =======================

Device Path               Size         Device nodes    rotates available Model name

May be this could help to see what has changed ?

Patrick

Le 11/10/2023 à 17:38, Eugen Block a écrit :

...

That's really strange. Just out of curiosity, have you tried Quincy (and/or Reef) as well? I don't recall what inventory does in the background exactly, I believe Adam King mentioned that in some thread, maybe that can help here. I'll search for that thread tomorrow.

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Hi Eugen,

[root@mostha1 ~]# rpm -q cephadm cephadm-16.2.14-0.el8.noarch

Log associated to the

2023-10-11 16:16:02,167 7f820515fb80 DEBUG -------------------------------------------------------------------------------- cephadm ['gather-facts'] 2023-10-11 16:16:02,208 7f820515fb80 DEBUG /bin/podman: 4.4.1 2023-10-11 16:16:02,313 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,317 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,322 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,326 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,329 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,333 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:04,474 7ff2a5c08b80 DEBUG -------------------------------------------------------------------------------- cephadm ['ceph-volume', 'inventory'] 2023-10-11 16:16:04,516 7ff2a5c08b80 DEBUG /usr/bin/podman: 4.4.1 2023-10-11 16:16:04,520 7ff2a5c08b80 DEBUG Using default config: /etc/ceph/ceph.conf 2023-10-11 16:16:04,573 7ff2a5c08b80 DEBUG /usr/bin/podman: 0d28d71358d7,445.8MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 2084faaf4d54,13.27MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 61073c53805d,512.7MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 6b9f0b72d668,361.1MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 7493a28808ad,163.7MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: a89672a3accf,59.22MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: b45271cc9726,54.24MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: e00ec13ab138,707.3MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,35.55MB / 50.32GB 2023-10-11 16:16:04,630 7ff2a5c08b80 DEBUG /usr/bin/podman: 0d28d71358d7,1.28% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 2084faaf4d54,0.00% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 61073c53805d,1.19% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 6b9f0b72d668,1.03% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 7493a28808ad,0.78% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: a89672a3accf,0.11% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: b45271cc9726,1.35% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: e00ec13ab138,0.43% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,0.02% 2023-10-11 16:16:04,634 7ff2a5c08b80 INFO Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c 2023-10-11 16:16:04,691 7ff2a5c08b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:16:04,692 7ff2a5c08b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:c08064dde4bba4e72a1f55d90ca32df9ef5aafab82efe2e0a0722444a5aaacca 2023-10-11 16:16:04,692 7ff2a5c08b80 DEBUG /usr/bin/podman: docker.io/ceph/ceph@sha256:056637972a107df4096f10951e4216b21fcd8ae0b9fb4552e628d35df3f61139 2023-10-11 16:16:04,694 7ff2a5c08b80 INFO Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:16:05,094 7ff2a5c08b80 DEBUG stat: 167 167 2023-10-11 16:16:05,903 7ff2a5c08b80 DEBUG Acquiring lock 140679815723776 on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:16:05,903 7ff2a5c08b80 DEBUG Lock 140679815723776 acquired on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:16:05,929 7ff2a5c08b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:05,933 7ff2a5c08b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:06,700 7ff2a5c08b80 DEBUG /usr/bin/podman: 2023-10-11 16:16:06,701 7ff2a5c08b80 DEBUG /usr/bin/podman: Device Path               Size         Device nodes rotates available Model name

I have only one version of cephadm in /var/lib/ceph/{fsid} : [root@mostha1 ~]# ls -lrt /var/lib/ceph/250f9864-0142-11ee-8e5f-00266cf8869c/cephadm* -rw-r--r-- 1 root root 350889 28 sept. 16:39 /var/lib/ceph/250f9864-0142-11ee-8e5f-00266cf8869c/cephadm.f6868821c084cd9740b59c7c5eb59f0dd47f6e3b1e6fecb542cb44134ace8d78

Running " python3 /var/lib/ceph/250f9864-0142-11ee-8e5f-00266cf8869c/cephadm.f6868821c084cd9740b59c7c5eb59f0dd47f6e3b1e6fecb542cb44134ace8d78 ceph-volume inventory" give the same output and the same log (execpt the valu of the lock):

2023-10-11 16:21:35,965 7f467cf31b80 DEBUG -------------------------------------------------------------------------------- cephadm ['ceph-volume', 'inventory'] 2023-10-11 16:21:36,009 7f467cf31b80 DEBUG /usr/bin/podman: 4.4.1 2023-10-11 16:21:36,012 7f467cf31b80 DEBUG Using default config: /etc/ceph/ceph.conf 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 0d28d71358d7,452.1MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 2084faaf4d54,13.27MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 61073c53805d,513.6MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 6b9f0b72d668,322.4MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 7493a28808ad,164MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: a89672a3accf,58.5MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: b45271cc9726,54.69MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: e00ec13ab138,707.1MB / 50.32GB 2023-10-11 16:21:36,068 7f467cf31b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,36.28MB / 50.32GB 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 0d28d71358d7,1.27% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 2084faaf4d54,0.00% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 61073c53805d,1.16% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 6b9f0b72d668,1.02% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 7493a28808ad,0.78% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: a89672a3accf,0.11% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: b45271cc9726,1.35% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: e00ec13ab138,0.41% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,0.02% 2023-10-11 16:21:36,128 7f467cf31b80 INFO Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c 2023-10-11 16:21:36,186 7f467cf31b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:21:36,187 7f467cf31b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:c08064dde4bba4e72a1f55d90ca32df9ef5aafab82efe2e0a0722444a5aaacca 2023-10-11 16:21:36,187 7f467cf31b80 DEBUG /usr/bin/podman: docker.io/ceph/ceph@sha256:056637972a107df4096f10951e4216b21fcd8ae0b9fb4552e628d35df3f61139 2023-10-11 16:21:36,189 7f467cf31b80 INFO Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:21:36,549 7f467cf31b80 DEBUG stat: 167 167 2023-10-11 16:21:36,942 7f467cf31b80 DEBUG Acquiring lock 139940396923424 on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:21:36,942 7f467cf31b80 DEBUG Lock 139940396923424 acquired on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:21:36,969 7f467cf31b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:21:36,972 7f467cf31b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:21:37,749 7f467cf31b80 DEBUG /usr/bin/podman: 2023-10-11 16:21:37,750 7f467cf31b80 DEBUG /usr/bin/podman: Device Path               Size         Device nodes rotates available Model name

Patrick

Le 11/10/2023 à 15:59, Eugen Block a écrit :

...

Can you check which cephadm version is installed on the host? And then please add (only the relevant) output from the cephadm.log when you run the inventory (without the --image <octopus>). Sometimes the version mismatch on the host and the one the orchestrator uses can cause some disruptions. You could try the same with the latest cephadm you have in /var/lib/ceph/${fsid}/ (ls -lrt /var/lib/ceph/${fsid}/cephadm.*). I mentioned that in this thread [1]. So you could try the following:

$ chmod +x /var/lib/ceph/{fsid}/cephadm.{latest}

$ python3 /var/lib/ceph/{fsid}/cephadm.{latest} ceph-volume inventory

Does the output differ? Paste the relevant cephadm.log from that attempt as well.

[1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LASBJCSPFGDY...

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

> Hi Eugen, > > first many thanks for the time spent on this problem. > > "ceph osd purge 2 --force --yes-i-really-mean-it" works and > clean all the bas status. > > *[root@mostha1 ~]# cephadm shell > *Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c > Using recent ceph image > quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e > * > * > *[ceph: root@mostha1 /]# ceph osd purge 2 --force > --yes-i-really-mean-it * > purged osd.2 > * > * > *[ceph: root@mostha1 /]# ceph osd tree* > ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT PRI-AFF > -1         1.72823  root default > -5         0.45477      host dean >  0    hdd  0.22739          osd.0         up   1.00000 1.00000 >  4    hdd  0.22739          osd.4         up   1.00000 1.00000 > -9         0.22739      host ekman >  6    hdd  0.22739          osd.6         up   1.00000 1.00000 > -7         0.45479      host mostha1 >  5    hdd  0.45479          osd.5         up   1.00000 1.00000 > -3         0.59128      host mostha2 >  1    hdd  0.22739          osd.1         up   1.00000 1.00000 >  3    hdd  0.36389          osd.3         up   1.00000 1.00000 > * > * > *[ceph: root@mostha1 /]# lsblk* > NAME MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT > sda 8:0    1 232.9G  0 disk > |-sda1 8:1    1   3.9G  0 part /rootfs/boot > |-sda2 8:2    1   3.9G  0 part [SWAP] > `-sda3 8:3    1   225G  0 part > |-al8vg-rootvol 253:0    0  48.8G  0 lvm  /rootfs > |-al8vg-homevol 253:2    0   9.8G  0 lvm  /rootfs/home > |-al8vg-tmpvol 253:3    0   9.8G  0 lvm  /rootfs/tmp > `-al8vg-varvol 253:4    0  19.8G  0 lvm  /rootfs/var > sdb 8:16   1 465.8G  0 disk > `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 253:1    0 465.8G  0 > lvm > sdc 8:32   1 232.9G  0 disk > > "cephadm ceph-volume inventory" returns nothing: > > *[root@mostha1 ~]# cephadm ceph-volume inventory ** > *Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c > Using recent ceph image > quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e > > Device Path               Size         Device nodes rotates > available Model name > > [root@mostha1 ~]# > > But running the same command within cephadm 15.2.17 works: > > *[root@mostha1 ~]# cephadm --image 93146564743f ceph-volume inventory* > Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c > > Device Path               Size         rotates available Model name > /dev/sdc                  232.83 GB    True    True SAMSUNG HE253GJ > /dev/sda                  232.83 GB    True    False SAMSUNG HE253GJ > /dev/sdb                  465.76 GB    True    False WDC WD5003ABYX-1 > > [root@mostha1 ~]# > > *[root@mostha1 ~]# podman images -a** > *REPOSITORY                        TAG         IMAGE ID > CREATED        SIZE > quay.io/ceph/ceph                 v16.2.14 f13d80acdbb5 2 > weeks ago    1.21 GB > quay.io/ceph/ceph                 v15.2.17 93146564743f 14 > months ago  1.24 GB > .... > > > Patrick > > Le 11/10/2023 à 15:14, Eugen Block a écrit : >> Your response is a bit confusing since it seems to be mixed >> up with the previous answer. So you still need to remove the >> OSD properly, so purge it from the crush tree: >> >> ceph osd purge 2 --force --yes-i-really-mean-it (only in a >> test cluster!) >> >> If everything is clean (OSD has been removed, disk has been >> zapped, lsblk shows no LVs for that disk) you can check the >> inventory: >> >> cephadm ceph-volume inventory >> >> Please also add the output of 'ceph orch ls osd --export'. >> >> Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>: >> >>> Hi Eugen, >>> >>> - the OS is Alma Linux 8 with latests updates. >>> >>> - this morning I've worked with ceph-volume but it ends with >>> a strange final state. I was connected on host mostha1 where >>> /dev/sdc was not reconized. These are the steps followed >>> based on the ceph-volume documentation I've read: >>> [root@mostha1 ~]# cephadm shell >>> [ceph: root@mostha1 /]# ceph auth get client.bootstrap-osd > >>> /var/lib/ceph/bootstrap-osd/ceph.keyring >>> [ceph: root@mostha1 /]# ceph-volume lvm prepare --bluestore >>> --data /dev/sdc >>> >>> Now lsblk command shows sdc as an osd: >>> .... >>> sdb 8:16   1 465.8G  0 disk >>> `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 253:1    0 465.8G  0 >>> lvm >>> sdc 8:32   1 232.9G  0 disk >>> `-ceph--b27d7a07--278d--4ee2--b84e--53256ef8de4c-osd--block--45c8e92c--caf9--4fe7--9a42--7b45a0794632 253:5    0 232.8G  0 >>> lvm >>> >>> Then I've tried to activate this osd but it fails as in >>> podman I have not access to systemctl: >>> >>> [ceph: root@mostha1 /]# ceph-volume lvm activate 2 >>> 45c8e92c-caf9-4fe7-9a42-7b45a0794632 >>> ..... >>> Running command: /usr/bin/systemctl start ceph-osd@2 >>>  stderr: Failed to connect to bus: No such file or directory >>> -->  RuntimeError: command returned non-zero exit status: 1 >>> [ceph: root@mostha1 /]# ceph osd tree >>> >>> And now I have now I have a strange status for this osd.2: >>> >>> [ceph: root@mostha1 /]# ceph osd tree >>> ID  CLASS  WEIGHT   TYPE NAME         STATUS REWEIGHT PRI-AFF >>> -1         1.72823  root default >>> -5         0.45477      host dean >>>  0    hdd  0.22739          osd.0         up 1.00000 1.00000 >>>  4    hdd  0.22739          osd.4         up 1.00000 1.00000 >>> -9         0.22739      host ekman >>>  6    hdd  0.22739          osd.6         up 1.00000 1.00000 >>> -7         0.45479      host mostha1 >>>  5    hdd  0.45479          osd.5         up 1.00000 1.00000 >>> -3         0.59128      host mostha2 >>>  1    hdd  0.22739          osd.1         up 1.00000 1.00000 >>>  3    hdd  0.36389          osd.3         up 1.00000 1.00000 >>>  2               0  osd.2               down 0 1.00000 >>> >>> I've tried to destroy the osd as you suggest but even if the >>> command returns no error I still have this osd even if >>> "lsblk" do not show any more /dev/sdc as a ceph osd device. >>> >>> *[ceph: root@mostha1 /]# ceph-volume lvm zap --destroy /dev/sdc** >>> *--> Zapping: /dev/sdc >>> --> Zapping lvm member /dev/sdc. lv_path is >>> /dev/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c/osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632 >>> --> Unmounting /var/lib/ceph/osd/ceph-2 >>> Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-2 >>>  stderr: umount: /var/lib/ceph/osd/ceph-2 unmounted >>> Running command: /usr/bin/dd if=/dev/zero >>> of=/dev/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c/osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632 bs=1M count=10 >>> conv=fsync >>>  stderr: 10+0 records in >>> 10+0 records out >>> 10485760 bytes (10 MB, 10 MiB) copied, 0.575633 s, 18.2 MB/s >>> --> Only 1 LV left in VG, will proceed to destroy volume >>> group ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c >>> Running command: nsenter --mount=/rootfs/proc/1/ns/mnt >>> --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net >>> --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f >>> ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c >>>  stderr: Removing >>> ceph--b27d7a07--278d--4ee2--b84e--53256ef8de4c-osd--block--45c8e92c--caf9--4fe7--9a42--7b45a0794632 >>> (253:1) >>>  stderr: Releasing logical volume >>> "osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632" >>>  stderr: Archiving volume group >>> "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" metadata (seqno >>> 5). >>>  stdout: Logical volume >>> "osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632" >>> successfully removed. >>>  stderr: Removing physical volume "/dev/sdc" from volume >>> group "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" >>>  stdout: Volume group >>> "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" successfully >>> removed >>>  stderr: Creating volume group backup >>> "/etc/lvm/backup/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" >>> (seqno 6). >>> Running command: nsenter --mount=/rootfs/proc/1/ns/mnt >>> --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net >>> --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/sdc >>>  stdout: Labels on physical volume "/dev/sdc" successfully wiped. >>> Running command: /usr/bin/dd if=/dev/zero of=/dev/sdc bs=1M >>> count=10 conv=fsync >>>  stderr: 10+0 records in >>> 10+0 records out >>> 10485760 bytes (10 MB, 10 MiB) copied, 0.590652 s, 17.8 MB/s >>> *--> Zapping successful for: <Raw Device: /dev/sdc>* >>> * >>> * >>> *[ceph: root@mostha1 /]# ceph osd tree** >>> *ID  CLASS  WEIGHT   TYPE NAME         STATUS REWEIGHT PRI-AFF >>> -1         1.72823  root default >>> -5         0.45477      host dean >>>  0    hdd  0.22739          osd.0         up 1.00000 1.00000 >>>  4    hdd  0.22739          osd.4         up 1.00000 1.00000 >>> -9         0.22739      host ekman >>>  6    hdd  0.22739          osd.6         up 1.00000 1.00000 >>> -7         0.45479      host mostha1 >>>  5    hdd  0.45479          osd.5         up 1.00000 1.00000 >>> -3         0.59128      host mostha2 >>>  1    hdd  0.22739          osd.1         up 1.00000 1.00000 >>>  3    hdd  0.36389          osd.3         up 1.00000 1.00000 >>>  2               0  osd.2               down 0 1.00000 >>> * >>> * >>> *[ceph: root@mostha1 /]# lsblk** >>> *NAME MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT >>> sda 8:0    1 232.9G  0 disk >>> |-sda1 8:1    1   3.9G  0 part /rootfs/boot >>> |-sda2 8:2    1   3.9G  0 part [SWAP] >>> `-sda3 8:3    1   225G  0 part >>> |-al8vg-rootvol 253:0    0  48.8G  0 lvm  /rootfs >>> |-al8vg-homevol 253:3    0   9.8G  0 lvm /rootfs/home >>> |-al8vg-tmpvol 253:4    0   9.8G  0 lvm  /rootfs/tmp >>> `-al8vg-varvol 253:5    0  19.8G  0 lvm  /rootfs/var >>> sdb 8:16   1 465.8G  0 disk >>> `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 253:2    0 465.8G  0 >>> lvm >>> *sdc * >>> >>> Patrick >>> Le 11/10/2023 à 11:00, Eugen Block a écrit : >>>> Hi, >>>> >>>> just wondering if 'ceph-volume lvm zap --destroy /dev/sdc' >>>> would help here. From your previous output you didn't >>>> specify the --destroy flag. >>>> Which cephadm version is installed on the host? Did you >>>> also upgrade the OS when moving to Pacific? (Sorry if I >>>> missed that. >>>> >>>> >>>> Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>: >>>> >>>>> Le 02/10/2023 à 18:22, Patrick Bégou a écrit : >>>>>> Hi all, >>>>>> >>>>>> still stuck with this problem. >>>>>> >>>>>> I've deployed octopus and all my HDD have been setup as osd. Fine. >>>>>> I've upgraded to pacific and 2 osd have failed. They have >>>>>> been automatically removed and upgrade finishes. Cluster >>>>>> Health is finaly OK, no data loss. >>>>>> >>>>>> But now I cannot re-add these osd with pacific (I had >>>>>> previous troubles on these old HDDs, lost one osd in >>>>>> octopus and was able to reset and re-add it). >>>>>> >>>>>> I've tried manually to add the first osd on the node >>>>>> where it is located, following >>>>>> https://docs.ceph.com/en/pacific/rados/operations/bluestore-migration/ (not sure it's the best idea...) but it fails too. This node was the one used for deploying the >>>>>> cluster. >>>>>> >>>>>> [ceph: root@mostha1 /]# *ceph-volume lvm zap /dev/sdc* >>>>>> --> Zapping: /dev/sdc >>>>>> --> --destroy was not specified, but zapping a whole >>>>>> device will remove the partition table >>>>>> Running command: /usr/bin/dd if=/dev/zero of=/dev/sdc >>>>>> bs=1M count=10 conv=fsync >>>>>>  stderr: 10+0 records in >>>>>> 10+0 records out >>>>>> 10485760 bytes (10 MB, 10 MiB) copied, 0.663425 s, 15.8 MB/s >>>>>> --> Zapping successful for: <Raw Device: /dev/sdc> >>>>>> >>>>>> >>>>>> [ceph: root@mostha1 /]# *ceph-volume lvm create >>>>>> --bluestore --data /dev/sdc* >>>>>> Running command: /usr/bin/ceph-authtool --gen-print-key >>>>>> Running command: /usr/bin/ceph --cluster ceph --name >>>>>> client.bootstrap-osd --keyring >>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new >>>>>> 9f1eb8ee-41e6-4350-ad73-1be21234ec7c >>>>>>  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 >>>>>> auth: unable to find a keyring on >>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such >>>>>> file or directory >>>>>>  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 >>>>>> AuthRegistry(0x7fb4e405c4d8) no keyring found at >>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx >>>>>>  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 >>>>>> auth: unable to find a keyring on >>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such >>>>>> file or directory >>>>>>  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 >>>>>> AuthRegistry(0x7fb4e40601d0) no keyring found at >>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx >>>>>>  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 >>>>>> auth: unable to find a keyring on >>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such >>>>>> file or directory >>>>>>  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 >>>>>> AuthRegistry(0x7fb4eb8bee90) no keyring found at >>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx >>>>>>  stderr: 2023-10-02T16:09:29.858+0000 7fb4e965c700 -1 >>>>>> monclient(hunting): handle_auth_bad_method server >>>>>> allowed_methods [2] but i only support [1] >>>>>>  stderr: 2023-10-02T16:09:29.858+0000 7fb4e9e5d700 -1 >>>>>> monclient(hunting): handle_auth_bad_method server >>>>>> allowed_methods [2] but i only support [1] >>>>>>  stderr: 2023-10-02T16:09:29.858+0000 7fb4e8e5b700 -1 >>>>>> monclient(hunting): handle_auth_bad_method server >>>>>> allowed_methods [2] but i only support [1] >>>>>>  stderr: 2023-10-02T16:09:29.858+0000 7fb4eb8c0700 -1 >>>>>> monclient: authenticate NOTE: no keyring found; disabled >>>>>> cephx authentication >>>>>>  stderr: [errno 13] RADOS permission denied (error >>>>>> connecting to the cluster) >>>>>> -->  RuntimeError: Unable to create a new OSD id >>>>>> >>>>>> Any idea of what is wrong ? >>>>>> >>>>>> Thanks >>>>>> >>>>>> Patrick >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>> >>>>> >>>>> I'm still trying to understand what can be wrong or how to >>>>> debug this situation where Ceph cannot see the devices. >>>>> >>>>> The device :dev/sdc exists: >>>>> >>>>>    [root@mostha1 ~]# cephadm shell lsmcli ldl >>>>>    Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c >>>>>    Using recent ceph image >>>>> quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e    Path     | SCSI VPD 0x83    | Link Type | Serial Number | >>>>> Health >>>>>    Status ceph osd purge 2 --force --yes-i-really-mean-it >>>>> -------------------------------------------------------------------------    /dev/sda | 50024e92039e4f1c | PATA/SATA | S2B5J90ZA10142 | >>>>> Good >>>>>    /dev/sdb | 50014ee0ad5953c9 | PATA/SATA | WD-WMAYP0982329 | Good >>>>>    /dev/sdc | 50024e920387fa2c | PATA/SATA | S2B5J90ZA02494 | Good >>>>> >>>>> But I cannot do anything with it: >>>>> >>>>>    [root@mostha1 ~]# cephadm shell ceph orch device zap >>>>>    mostha1.legi.grenoble-inp.fr /dev/sdc --force >>>>>    Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c >>>>>    Using recent ceph image >>>>> quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e    Error EINVAL: Device path '/dev/sdc' not found on >>>>> host >>>>>    'mostha1.legi.grenoble-inp.fr' >>>>> >>>>> Since I moved from octopus to Pacific. >>>>> >>>>> Patrick >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

The server has enough available storage: [root@mostha1 log]# df -h Sys. de fichiers          Taille Utilisé Dispo Uti% Monté sur devtmpfs                     24G       0   24G   0% /dev tmpfs                        24G     84K   24G   1% /dev/shm tmpfs                        24G    195M   24G   1% /run tmpfs                        24G       0   24G   0% /sys/fs/cgroup /dev/mapper/al8vg-rootvol    49G    6,5G   43G  14% / /dev/sda1                   3,8G    412M  3,2G  12% /boot /dev/mapper/al8vg-varvol     20G    9,7G   11G  49% /var /dev/mapper/al8vg-tmpvol    9,8G    103M  9,7G   2% /tmp /dev/mapper/al8vg-homevol   9,8G    103M  9,7G   2% /home tmpfs                       4,7G       0  4,7G   0% /run/user/0 overlay                      20G    9,7G   11G  49% /var/lib/containers/storage/overlay/b8769720357497ebdbf68768753da154b3d63cfbef254036441af60a91649127/merged overlay                      20G    9,7G   11G  49% /var/lib/containers/storage/overlay/2eed15daec130da50530621740025655ecd961e1b1855f35922f03561960d999/merged overlay                      20G    9,7G   11G  49% /var/lib/containers/storage/overlay/4d0b4f0b4063cce3f983beda844440bac78dd3b5f30379d2eb96daefef8ddfaf/merged overlay                      20G    9,7G   11G  49% /var/lib/containers/storage/overlay/129c5d3e070f80f17a79c1f172b60c2fc0f30a84b51b07ea207dc5868cd1d7f0/merged overlay                      20G    9,7G   11G  49% /var/lib/containers/storage/overlay/c41d6bdaf941d16fd80326ef5dae6a02524d3f41bcb64cb29bda2bd5816fee9a/merged overlay                      20G    9,7G   11G  49% /var/lib/containers/storage/overlay/1b6c1c893e7ed2c128378bdf2af408f3a834f3453a0505ac042099d6f484dc9b/merged overlay                      20G    9,7G   11G  49% /var/lib/containers/storage/overlay/962e5c1380a60e9a54ac29eccb71667f13a5f9047b2ee98e6303a5fea613162f/merged overlay                      20G    9,7G   11G  49% /var/lib/containers/storage/overlay/3578d0f5a70afce839017dec888908dead82fb50f90834e5b040e9fd2ada9fba/merged overlay                      20G    9,7G   11G  49% /var/lib/containers/storage/overlay/7d9c35751388325c3da54f03981770aa49599a657c2dfe3ba9527884864f177d/merged When I was testing different versions, I removed tested images each time with "podman rmi" for i in v16.2.10-20220920 v16.2.11-20230125 v16.2.11-20230209 v16.2.11-20230316; do echo "=================== Ceph $i =======================" cephadm --image quay.io/ceph/ceph:$i ceph-volume inventory id=$(podman images |grep " $i "|cut -c 46-59) podman rmi $id done |tee trace.ceph16.2.txt I do not now how to investigate, may be with a "git bisect" between the 2 releases to catch the faulty commit in a podman container context. I'm not so familiar with containers and ceph. Patrick Le 13/10/2023 à 09:18, Eugen Block a écrit :

Trying to resend with the attachment. I can't really find anything suspicious, ceph-volume (16.2.11) does recognize /dev/sdc though:

[2023-10-12 08:58:14,135][ceph_volume.process][INFO  ] stdout NAME="sdc" KNAME="sdc" PKNAME="" MAJ:MIN="8:32" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="1" MODEL="SAMSUNG HE253GJ " SIZE="232.9G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="1" SCHED="mq-deadline" TYPE="disk" DISC-ALN="0" DISC-GRAN="0B" DISC-MAX="0B" DISC-ZERO="0" PKNAME="" PARTLABEL="" [2023-10-12 08:58:14,139][ceph_volume.util.system][INFO  ] Executable pvs found on the host, will use /sbin/pvs [2023-10-12 08:58:14,140][ceph_volume.process][INFO  ] Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvs --noheadings --readonly --units=b --nosuffix --separator=";" -o pv_name,vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size

But apparently it just stops after that. I already tried to find a debug log-level for ceph-volume but it's not applicable to all subcommands. The cephadm.log also just stops without even finishing the "copying blob", which makes me wonder if it actually pulls the entire image? I assume you have enough free disk space (otherwise I would expect a message "failed to pull target image"), do you see any other warnings in syslog or something? Or are the logs incomplete? Maybe someone else finds any clues in the logs...

Regards, Eugen

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Hi Eugen,

You will find in attachment cephadm.log and cepĥ-volume.log. Each contains the outputs for the 2 versions.  v16.2.10-20220920 is really more verbose or v16.2.11-20230125 does not execute all the detection process

Patrick

Le 12/10/2023 à 09:34, Eugen Block a écrit :

...

Good catch, and I found the thread I had in my mind, it was this exact one. :-D Anyway, can you share the ceph-volume.log from the working and the not working attempt? I tried to look for something significant in the pacific release notes for 16.2.11, and there were some changes to ceph-volume, but I'm not sure what it could be.

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

I've ran additional tests with Pacific releases and with "ceph-volume inventory" things went wrong with the first v16.11 release (v16.2.11-20230125)

=================== Ceph v16.2.10-20220920 =======================

Device Path               Size         rotates available Model name /dev/sdc                  232.83 GB    True    True SAMSUNG HE253GJ /dev/sda                  232.83 GB    True    False SAMSUNG HE253GJ /dev/sdb                  465.76 GB    True    False     WDC WD5003ABYX-1

=================== Ceph v16.2.11-20230125 =======================

Device Path               Size         Device nodes rotates available Model name

May be this could help to see what has changed ?

Patrick

Le 11/10/2023 à 17:38, Eugen Block a écrit :

...

That's really strange. Just out of curiosity, have you tried Quincy (and/or Reef) as well? I don't recall what inventory does in the background exactly, I believe Adam King mentioned that in some thread, maybe that can help here. I'll search for that thread tomorrow.

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Hi Eugen,

[root@mostha1 ~]# rpm -q cephadm cephadm-16.2.14-0.el8.noarch

Log associated to the

2023-10-11 16:16:02,167 7f820515fb80 DEBUG -------------------------------------------------------------------------------- cephadm ['gather-facts'] 2023-10-11 16:16:02,208 7f820515fb80 DEBUG /bin/podman: 4.4.1 2023-10-11 16:16:02,313 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,317 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,322 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,326 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,329 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,333 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:04,474 7ff2a5c08b80 DEBUG -------------------------------------------------------------------------------- cephadm ['ceph-volume', 'inventory'] 2023-10-11 16:16:04,516 7ff2a5c08b80 DEBUG /usr/bin/podman: 4.4.1 2023-10-11 16:16:04,520 7ff2a5c08b80 DEBUG Using default config: /etc/ceph/ceph.conf 2023-10-11 16:16:04,573 7ff2a5c08b80 DEBUG /usr/bin/podman: 0d28d71358d7,445.8MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 2084faaf4d54,13.27MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 61073c53805d,512.7MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 6b9f0b72d668,361.1MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 7493a28808ad,163.7MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: a89672a3accf,59.22MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: b45271cc9726,54.24MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: e00ec13ab138,707.3MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,35.55MB / 50.32GB 2023-10-11 16:16:04,630 7ff2a5c08b80 DEBUG /usr/bin/podman: 0d28d71358d7,1.28% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 2084faaf4d54,0.00% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 61073c53805d,1.19% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 6b9f0b72d668,1.03% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 7493a28808ad,0.78% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: a89672a3accf,0.11% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: b45271cc9726,1.35% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: e00ec13ab138,0.43% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,0.02% 2023-10-11 16:16:04,634 7ff2a5c08b80 INFO Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c 2023-10-11 16:16:04,691 7ff2a5c08b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:16:04,692 7ff2a5c08b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:c08064dde4bba4e72a1f55d90ca32df9ef5aafab82efe2e0a0722444a5aaacca 2023-10-11 16:16:04,692 7ff2a5c08b80 DEBUG /usr/bin/podman: docker.io/ceph/ceph@sha256:056637972a107df4096f10951e4216b21fcd8ae0b9fb4552e628d35df3f61139 2023-10-11 16:16:04,694 7ff2a5c08b80 INFO Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:16:05,094 7ff2a5c08b80 DEBUG stat: 167 167 2023-10-11 16:16:05,903 7ff2a5c08b80 DEBUG Acquiring lock 140679815723776 on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:16:05,903 7ff2a5c08b80 DEBUG Lock 140679815723776 acquired on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:16:05,929 7ff2a5c08b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:05,933 7ff2a5c08b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:06,700 7ff2a5c08b80 DEBUG /usr/bin/podman: 2023-10-11 16:16:06,701 7ff2a5c08b80 DEBUG /usr/bin/podman: Device Path               Size Device nodes rotates available Model name

I have only one version of cephadm in /var/lib/ceph/{fsid} : [root@mostha1 ~]# ls -lrt /var/lib/ceph/250f9864-0142-11ee-8e5f-00266cf8869c/cephadm* -rw-r--r-- 1 root root 350889 28 sept. 16:39 /var/lib/ceph/250f9864-0142-11ee-8e5f-00266cf8869c/cephadm.f6868821c084cd9740b59c7c5eb59f0dd47f6e3b1e6fecb542cb44134ace8d78

Running " python3 /var/lib/ceph/250f9864-0142-11ee-8e5f-00266cf8869c/cephadm.f6868821c084cd9740b59c7c5eb59f0dd47f6e3b1e6fecb542cb44134ace8d78 ceph-volume inventory" give the same output and the same log (execpt the valu of the lock):

2023-10-11 16:21:35,965 7f467cf31b80 DEBUG -------------------------------------------------------------------------------- cephadm ['ceph-volume', 'inventory'] 2023-10-11 16:21:36,009 7f467cf31b80 DEBUG /usr/bin/podman: 4.4.1 2023-10-11 16:21:36,012 7f467cf31b80 DEBUG Using default config: /etc/ceph/ceph.conf 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 0d28d71358d7,452.1MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 2084faaf4d54,13.27MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 61073c53805d,513.6MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 6b9f0b72d668,322.4MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 7493a28808ad,164MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: a89672a3accf,58.5MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: b45271cc9726,54.69MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: e00ec13ab138,707.1MB / 50.32GB 2023-10-11 16:21:36,068 7f467cf31b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,36.28MB / 50.32GB 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 0d28d71358d7,1.27% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 2084faaf4d54,0.00% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 61073c53805d,1.16% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 6b9f0b72d668,1.02% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 7493a28808ad,0.78% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: a89672a3accf,0.11% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: b45271cc9726,1.35% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: e00ec13ab138,0.41% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,0.02% 2023-10-11 16:21:36,128 7f467cf31b80 INFO Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c 2023-10-11 16:21:36,186 7f467cf31b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:21:36,187 7f467cf31b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:c08064dde4bba4e72a1f55d90ca32df9ef5aafab82efe2e0a0722444a5aaacca 2023-10-11 16:21:36,187 7f467cf31b80 DEBUG /usr/bin/podman: docker.io/ceph/ceph@sha256:056637972a107df4096f10951e4216b21fcd8ae0b9fb4552e628d35df3f61139 2023-10-11 16:21:36,189 7f467cf31b80 INFO Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:21:36,549 7f467cf31b80 DEBUG stat: 167 167 2023-10-11 16:21:36,942 7f467cf31b80 DEBUG Acquiring lock 139940396923424 on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:21:36,942 7f467cf31b80 DEBUG Lock 139940396923424 acquired on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:21:36,969 7f467cf31b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:21:36,972 7f467cf31b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:21:37,749 7f467cf31b80 DEBUG /usr/bin/podman: 2023-10-11 16:21:37,750 7f467cf31b80 DEBUG /usr/bin/podman: Device Path               Size Device nodes rotates available Model name

Patrick

Le 11/10/2023 à 15:59, Eugen Block a écrit : > Can you check which cephadm version is installed on the host? > And then please add (only the relevant) output from the > cephadm.log when you run the inventory (without the --image > <octopus>). Sometimes the version mismatch on the host and the > one the orchestrator uses can cause some disruptions. You could > try the same with the latest cephadm you have in > /var/lib/ceph/${fsid}/ (ls -lrt > /var/lib/ceph/${fsid}/cephadm.*). I mentioned that in this > thread [1]. So you could try the following: > > $ chmod +x /var/lib/ceph/{fsid}/cephadm.{latest} > > $ python3 /var/lib/ceph/{fsid}/cephadm.{latest} ceph-volume > inventory > > Does the output differ? Paste the relevant cephadm.log from that > attempt as well. > > [1] > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LASBJCSPFGDY... > > Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>: > >> Hi Eugen, >> >> first many thanks for the time spent on this problem. >> >> "ceph osd purge 2 --force --yes-i-really-mean-it" works and >> clean all the bas status. >> >> *[root@mostha1 ~]# cephadm shell >> *Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c >> Using recent ceph image >> quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e >> * >> * >> *[ceph: root@mostha1 /]# ceph osd purge 2 --force >> --yes-i-really-mean-it * >> purged osd.2 >> * >> * >> *[ceph: root@mostha1 /]# ceph osd tree* >> ID  CLASS  WEIGHT   TYPE NAME         STATUS REWEIGHT PRI-AFF >> -1         1.72823  root default >> -5         0.45477      host dean >>  0    hdd  0.22739          osd.0         up 1.00000 1.00000 >>  4    hdd  0.22739          osd.4         up 1.00000 1.00000 >> -9         0.22739      host ekman >>  6    hdd  0.22739          osd.6         up 1.00000 1.00000 >> -7         0.45479      host mostha1 >>  5    hdd  0.45479          osd.5         up 1.00000 1.00000 >> -3         0.59128      host mostha2 >>  1    hdd  0.22739          osd.1         up 1.00000 1.00000 >>  3    hdd  0.36389          osd.3         up 1.00000 1.00000 >> * >> * >> *[ceph: root@mostha1 /]# lsblk* >> NAME MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT >> sda 8:0    1 232.9G  0 disk >> |-sda1 8:1    1   3.9G  0 part /rootfs/boot >> |-sda2 8:2    1   3.9G  0 part [SWAP] >> `-sda3 8:3    1   225G  0 part >> |-al8vg-rootvol 253:0    0  48.8G  0 lvm  /rootfs >> |-al8vg-homevol 253:2    0   9.8G  0 lvm /rootfs/home >> |-al8vg-tmpvol 253:3    0   9.8G  0 lvm  /rootfs/tmp >> `-al8vg-varvol 253:4    0  19.8G  0 lvm  /rootfs/var >> sdb 8:16   1 465.8G  0 disk >> `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 >> 253:1    0 465.8G  0 lvm >> sdc 8:32   1 232.9G  0 disk >> >> "cephadm ceph-volume inventory" returns nothing: >> >> *[root@mostha1 ~]# cephadm ceph-volume inventory ** >> *Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c >> Using recent ceph image >> quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e >> >> Device Path               Size         Device nodes rotates >> available Model name >> >> [root@mostha1 ~]# >> >> But running the same command within cephadm 15.2.17 works: >> >> *[root@mostha1 ~]# cephadm --image 93146564743f ceph-volume >> inventory* >> Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c >> >> Device Path               Size         rotates available Model >> name >> /dev/sdc                  232.83 GB    True    True SAMSUNG >> HE253GJ >> /dev/sda                  232.83 GB    True    False SAMSUNG >> HE253GJ >> /dev/sdb                  465.76 GB    True    False WDC >> WD5003ABYX-1 >> >> [root@mostha1 ~]# >> >> *[root@mostha1 ~]# podman images -a** >> *REPOSITORY                        TAG         IMAGE ID >> CREATED        SIZE >> quay.io/ceph/ceph                 v16.2.14 f13d80acdbb5 2 weeks >> ago    1.21 GB >> quay.io/ceph/ceph                 v15.2.17 93146564743f 14 >> months ago  1.24 GB >> .... >> >> >> Patrick >> >> Le 11/10/2023 à 15:14, Eugen Block a écrit : >>> Your response is a bit confusing since it seems to be mixed up >>> with the previous answer. So you still need to remove the OSD >>> properly, so purge it from the crush tree: >>> >>> ceph osd purge 2 --force --yes-i-really-mean-it (only in a >>> test cluster!) >>> >>> If everything is clean (OSD has been removed, disk has been >>> zapped, lsblk shows no LVs for that disk) you can check the >>> inventory: >>> >>> cephadm ceph-volume inventory >>> >>> Please also add the output of 'ceph orch ls osd --export'. >>> >>> Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>: >>> >>>> Hi Eugen, >>>> >>>> - the OS is Alma Linux 8 with latests updates. >>>> >>>> - this morning I've worked with ceph-volume but it ends with >>>> a strange final state. I was connected on host mostha1 where >>>> /dev/sdc was not reconized. These are the steps followed >>>> based on the ceph-volume documentation I've read: >>>> [root@mostha1 ~]# cephadm shell >>>> [ceph: root@mostha1 /]# ceph auth get client.bootstrap-osd > >>>> /var/lib/ceph/bootstrap-osd/ceph.keyring >>>> [ceph: root@mostha1 /]# ceph-volume lvm prepare --bluestore >>>> --data /dev/sdc >>>> >>>> Now lsblk command shows sdc as an osd: >>>> .... >>>> sdb 8:16   1 465.8G  0 disk >>>> `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 >>>> 253:1    0 465.8G  0 lvm >>>> sdc 8:32   1 232.9G  0 disk >>>> `-ceph--b27d7a07--278d--4ee2--b84e--53256ef8de4c-osd--block--45c8e92c--caf9--4fe7--9a42--7b45a0794632 >>>> 253:5    0 232.8G  0 lvm >>>> >>>> Then I've tried to activate this osd but it fails as in >>>> podman I have not access to systemctl: >>>> >>>> [ceph: root@mostha1 /]# ceph-volume lvm activate 2 >>>> 45c8e92c-caf9-4fe7-9a42-7b45a0794632 >>>> ..... >>>> Running command: /usr/bin/systemctl start ceph-osd@2 >>>>  stderr: Failed to connect to bus: No such file or directory >>>> -->  RuntimeError: command returned non-zero exit status: 1 >>>> [ceph: root@mostha1 /]# ceph osd tree >>>> >>>> And now I have now I have a strange status for this osd.2: >>>> >>>> [ceph: root@mostha1 /]# ceph osd tree >>>> ID  CLASS  WEIGHT   TYPE NAME         STATUS REWEIGHT PRI-AFF >>>> -1         1.72823  root default >>>> -5         0.45477      host dean >>>>  0    hdd  0.22739          osd.0         up 1.00000 1.00000 >>>>  4    hdd  0.22739          osd.4         up 1.00000 1.00000 >>>> -9         0.22739      host ekman >>>>  6    hdd  0.22739          osd.6         up 1.00000 1.00000 >>>> -7         0.45479      host mostha1 >>>>  5    hdd  0.45479          osd.5         up 1.00000 1.00000 >>>> -3         0.59128      host mostha2 >>>>  1    hdd  0.22739          osd.1         up 1.00000 1.00000 >>>>  3    hdd  0.36389          osd.3         up 1.00000 1.00000 >>>>  2               0  osd.2               down 0 1.00000 >>>> >>>> I've tried to destroy the osd as you suggest but even if the >>>> command returns no error I still have this osd even if >>>> "lsblk" do not show any more /dev/sdc as a ceph osd device. >>>> >>>> *[ceph: root@mostha1 /]# ceph-volume lvm zap --destroy >>>> /dev/sdc** >>>> *--> Zapping: /dev/sdc >>>> --> Zapping lvm member /dev/sdc. lv_path is >>>> /dev/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c/osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632 >>>> --> Unmounting /var/lib/ceph/osd/ceph-2 >>>> Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-2 >>>>  stderr: umount: /var/lib/ceph/osd/ceph-2 unmounted >>>> Running command: /usr/bin/dd if=/dev/zero >>>> of=/dev/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c/osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632 >>>> bs=1M count=10 conv=fsync >>>>  stderr: 10+0 records in >>>> 10+0 records out >>>> 10485760 bytes (10 MB, 10 MiB) copied, 0.575633 s, 18.2 MB/s >>>> --> Only 1 LV left in VG, will proceed to destroy volume >>>> group ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c >>>> Running command: nsenter --mount=/rootfs/proc/1/ns/mnt >>>> --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net >>>> --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f >>>> ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c >>>>  stderr: Removing >>>> ceph--b27d7a07--278d--4ee2--b84e--53256ef8de4c-osd--block--45c8e92c--caf9--4fe7--9a42--7b45a0794632 >>>> (253:1) >>>>  stderr: Releasing logical volume >>>> "osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632" >>>>  stderr: Archiving volume group >>>> "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" metadata (seqno 5). >>>>  stdout: Logical volume >>>> "osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632" successfully >>>> removed. >>>>  stderr: Removing physical volume "/dev/sdc" from volume >>>> group "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" >>>>  stdout: Volume group >>>> "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" successfully removed >>>>  stderr: Creating volume group backup >>>> "/etc/lvm/backup/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" >>>> (seqno 6). >>>> Running command: nsenter --mount=/rootfs/proc/1/ns/mnt >>>> --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net >>>> --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/sdc >>>>  stdout: Labels on physical volume "/dev/sdc" successfully >>>> wiped. >>>> Running command: /usr/bin/dd if=/dev/zero of=/dev/sdc bs=1M >>>> count=10 conv=fsync >>>>  stderr: 10+0 records in >>>> 10+0 records out >>>> 10485760 bytes (10 MB, 10 MiB) copied, 0.590652 s, 17.8 MB/s >>>> *--> Zapping successful for: <Raw Device: /dev/sdc>* >>>> * >>>> * >>>> *[ceph: root@mostha1 /]# ceph osd tree** >>>> *ID  CLASS  WEIGHT   TYPE NAME         STATUS REWEIGHT PRI-AFF >>>> -1         1.72823  root default >>>> -5         0.45477      host dean >>>>  0    hdd  0.22739          osd.0         up 1.00000 1.00000 >>>>  4    hdd  0.22739          osd.4         up 1.00000 1.00000 >>>> -9         0.22739      host ekman >>>>  6    hdd  0.22739          osd.6         up 1.00000 1.00000 >>>> -7         0.45479      host mostha1 >>>>  5    hdd  0.45479          osd.5         up 1.00000 1.00000 >>>> -3         0.59128      host mostha2 >>>>  1    hdd  0.22739          osd.1         up 1.00000 1.00000 >>>>  3    hdd  0.36389          osd.3         up 1.00000 1.00000 >>>>  2               0  osd.2               down 0 1.00000 >>>> * >>>> * >>>> *[ceph: root@mostha1 /]# lsblk** >>>> *NAME MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT >>>> sda 8:0    1 232.9G  0 disk >>>> |-sda1 8:1    1   3.9G  0 part /rootfs/boot >>>> |-sda2 8:2    1   3.9G  0 part [SWAP] >>>> `-sda3 8:3    1   225G  0 part >>>> |-al8vg-rootvol 253:0    0  48.8G  0 lvm /rootfs >>>> |-al8vg-homevol 253:3    0   9.8G  0 lvm /rootfs/home >>>> |-al8vg-tmpvol 253:4    0   9.8G  0 lvm /rootfs/tmp >>>> `-al8vg-varvol 253:5    0  19.8G  0 lvm /rootfs/var >>>> sdb 8:16   1 465.8G  0 disk >>>> `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 >>>> 253:2    0 465.8G  0 lvm >>>> *sdc * >>>> >>>> Patrick >>>> Le 11/10/2023 à 11:00, Eugen Block a écrit : >>>>> Hi, >>>>> >>>>> just wondering if 'ceph-volume lvm zap --destroy /dev/sdc' >>>>> would help here. From your previous output you didn't >>>>> specify the --destroy flag. >>>>> Which cephadm version is installed on the host? Did you also >>>>> upgrade the OS when moving to Pacific? (Sorry if I missed that. >>>>> >>>>> >>>>> Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>: >>>>> >>>>>> Le 02/10/2023 à 18:22, Patrick Bégou a écrit : >>>>>>> Hi all, >>>>>>> >>>>>>> still stuck with this problem. >>>>>>> >>>>>>> I've deployed octopus and all my HDD have been setup as >>>>>>> osd. Fine. >>>>>>> I've upgraded to pacific and 2 osd have failed. They have >>>>>>> been automatically removed and upgrade finishes. Cluster >>>>>>> Health is finaly OK, no data loss. >>>>>>> >>>>>>> But now I cannot re-add these osd with pacific (I had >>>>>>> previous troubles on these old HDDs, lost one osd in >>>>>>> octopus and was able to reset and re-add it). >>>>>>> >>>>>>> I've tried manually to add the first osd on the node where >>>>>>> it is located, following >>>>>>> https://docs.ceph.com/en/pacific/rados/operations/bluestore-migration/ >>>>>>> (not sure it's the best idea...) but it fails too. This >>>>>>> node was the one used for deploying the cluster. >>>>>>> >>>>>>> [ceph: root@mostha1 /]# *ceph-volume lvm zap /dev/sdc* >>>>>>> --> Zapping: /dev/sdc >>>>>>> --> --destroy was not specified, but zapping a whole >>>>>>> device will remove the partition table >>>>>>> Running command: /usr/bin/dd if=/dev/zero of=/dev/sdc >>>>>>> bs=1M count=10 conv=fsync >>>>>>>  stderr: 10+0 records in >>>>>>> 10+0 records out >>>>>>> 10485760 bytes (10 MB, 10 MiB) copied, 0.663425 s, 15.8 MB/s >>>>>>> --> Zapping successful for: <Raw Device: /dev/sdc> >>>>>>> >>>>>>> >>>>>>> [ceph: root@mostha1 /]# *ceph-volume lvm create >>>>>>> --bluestore --data /dev/sdc* >>>>>>> Running command: /usr/bin/ceph-authtool --gen-print-key >>>>>>> Running command: /usr/bin/ceph --cluster ceph --name >>>>>>> client.bootstrap-osd --keyring >>>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new >>>>>>> 9f1eb8ee-41e6-4350-ad73-1be21234ec7c >>>>>>>  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 >>>>>>> auth: unable to find a keyring on >>>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file >>>>>>> or directory >>>>>>>  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 >>>>>>> AuthRegistry(0x7fb4e405c4d8) no keyring found at >>>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx >>>>>>>  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 >>>>>>> auth: unable to find a keyring on >>>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file >>>>>>> or directory >>>>>>>  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 >>>>>>> AuthRegistry(0x7fb4e40601d0) no keyring found at >>>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx >>>>>>>  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 >>>>>>> auth: unable to find a keyring on >>>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file >>>>>>> or directory >>>>>>>  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 >>>>>>> AuthRegistry(0x7fb4eb8bee90) no keyring found at >>>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx >>>>>>>  stderr: 2023-10-02T16:09:29.858+0000 7fb4e965c700 -1 >>>>>>> monclient(hunting): handle_auth_bad_method server >>>>>>> allowed_methods [2] but i only support [1] >>>>>>>  stderr: 2023-10-02T16:09:29.858+0000 7fb4e9e5d700 -1 >>>>>>> monclient(hunting): handle_auth_bad_method server >>>>>>> allowed_methods [2] but i only support [1] >>>>>>>  stderr: 2023-10-02T16:09:29.858+0000 7fb4e8e5b700 -1 >>>>>>> monclient(hunting): handle_auth_bad_method server >>>>>>> allowed_methods [2] but i only support [1] >>>>>>>  stderr: 2023-10-02T16:09:29.858+0000 7fb4eb8c0700 -1 >>>>>>> monclient: authenticate NOTE: no keyring found; disabled >>>>>>> cephx authentication >>>>>>>  stderr: [errno 13] RADOS permission denied (error >>>>>>> connecting to the cluster) >>>>>>> -->  RuntimeError: Unable to create a new OSD id >>>>>>> >>>>>>> Any idea of what is wrong ? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Patrick >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>> >>>>>> >>>>>> I'm still trying to understand what can be wrong or how to >>>>>> debug this situation where Ceph cannot see the devices. >>>>>> >>>>>> The device :dev/sdc exists: >>>>>> >>>>>>    [root@mostha1 ~]# cephadm shell lsmcli ldl >>>>>>    Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c >>>>>>    Using recent ceph image >>>>>> quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e >>>>>>    Path     | SCSI VPD 0x83    | Link Type | Serial Number >>>>>> | Health >>>>>>    Status ceph osd purge 2 --force --yes-i-really-mean-it >>>>>> ------------------------------------------------------------------------- >>>>>>    /dev/sda | 50024e92039e4f1c | PATA/SATA | S2B5J90ZA10142 >>>>>> | Good >>>>>>    /dev/sdb | 50014ee0ad5953c9 | PATA/SATA | >>>>>> WD-WMAYP0982329 | Good >>>>>>    /dev/sdc | 50024e920387fa2c | PATA/SATA | S2B5J90ZA02494 >>>>>> | Good >>>>>> >>>>>> But I cannot do anything with it: >>>>>> >>>>>>    [root@mostha1 ~]# cephadm shell ceph orch device zap >>>>>>    mostha1.legi.grenoble-inp.fr /dev/sdc --force >>>>>>    Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c >>>>>>    Using recent ceph image >>>>>> quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e >>>>>>    Error EINVAL: Device path '/dev/sdc' not found on host >>>>>>    'mostha1.legi.grenoble-inp.fr' >>>>>> >>>>>> Since I moved from octopus to Pacific. >>>>>> >>>>>> Patrick >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>> >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io

Hi Johan, So it is not O.S. related as you are running Debian and I am running Alma Linux. But I'm surprised why so few people meet this bug. Patrick Le 13/10/2023 à 17:38, Johan a écrit :

At home Im running a small cluster, Ceph v17.2.6, Debian 11 Bullseye.

I have recently added a new server to the cluster but face the same problem as Patrick, I can't add any HDD. Ceph doesn't recognise them.

I have run the same tests as Patrick, using Ceph v14-v18, and as Patrick showed the problem appears in Ceph v16.2.11-20230125

=================== Ceph v16.2.10-20220920 ======================= $ sudo cephadm --image quay.io/ceph/ceph:v16.2.10-20220920 ceph-volume inventory Inferring fsid 5592891c-30e4-11ed-b720-f02f741f58ac

Device Path               Size         rotates available Model name /dev/nvme0n1              931.51 GB    False   False     KINGSTON SNV2S1000G /dev/nvme1n1              931.51 GB    False   False     KINGSTON SNV2S1000G /dev/sda                  3.64 TB      True    False     WDC WD4003FFBX-6 /dev/sdb                  5.46 TB      True    False     WDC WD6003FFBX-6 /dev/sdc                  7.28 TB      True    False ST8000NE001-2M71 /dev/sdd                  7.28 TB      True    False     WDC WD8003FFBX-6

=================== Ceph v16.2.11-20230125 ======================= $ sudo cephadm --image quay.io/ceph/ceph:v16.2.11-20230125 ceph-volume inventory Inferring fsid 5592891c-30e4-11ed-b720-f02f741f58ac

Device Path               Size         Device nodes    rotates available Model name /dev/md0                  9.30 GB      nvme1n1p2,nvme0n1p2 False False /dev/md1                  59.57 GB     nvme0n1p3,nvme1n1p3 False False /dev/md2                  279.27 GB    nvme1n1p4,nvme0n1p4 False False /dev/nvme0n1              931.51 GB    nvme0n1         False False KINGSTON SNV2S1000G /dev/nvme1n1              931.51 GB    nvme1n1         False False KINGSTON SNV2S1000G

/Johan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

The problem appears in v16.2.11-20230125. I have no insight into the different commits. /Johan Den 2023-10-16 kl. 08:25, skrev 544463199@qq.com:

I encountered a similar problem on ceph17.2.5, could you found which commit caused it? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

hi johan， The OS that I use is centos 8.3，The feedback result of the ceph-volume inventory command in ceph17.2.5 is empty, but ceph orch daemon add osd can add osd. Hope it helps you.

Hi all, I'm trying to catch the faulty commit. I'm able to build Ceph from the git repo in a fresh podman container but at this time, the lsblk command returns nothing in my container. In ceph containers lsblk works So something is wrong with launching my podman container (or different from launching ceph containers) and I cannot find what. Any help about this step ? Thanks Patrick Le 13/10/2023 à 09:18, Eugen Block a écrit :

Trying to resend with the attachment. I can't really find anything suspicious, ceph-volume (16.2.11) does recognize /dev/sdc though:

[2023-10-12 08:58:14,135][ceph_volume.process][INFO  ] stdout NAME="sdc" KNAME="sdc" PKNAME="" MAJ:MIN="8:32" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="1" MODEL="SAMSUNG HE253GJ " SIZE="232.9G" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="1" SCHED="mq-deadline" TYPE="disk" DISC-ALN="0" DISC-GRAN="0B" DISC-MAX="0B" DISC-ZERO="0" PKNAME="" PARTLABEL="" [2023-10-12 08:58:14,139][ceph_volume.util.system][INFO  ] Executable pvs found on the host, will use /sbin/pvs [2023-10-12 08:58:14,140][ceph_volume.process][INFO  ] Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvs --noheadings --readonly --units=b --nosuffix --separator=";" -o pv_name,vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size

But apparently it just stops after that. I already tried to find a debug log-level for ceph-volume but it's not applicable to all subcommands. The cephadm.log also just stops without even finishing the "copying blob", which makes me wonder if it actually pulls the entire image? I assume you have enough free disk space (otherwise I would expect a message "failed to pull target image"), do you see any other warnings in syslog or something? Or are the logs incomplete? Maybe someone else finds any clues in the logs...

Regards, Eugen

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Hi Eugen,

You will find in attachment cephadm.log and cepĥ-volume.log. Each contains the outputs for the 2 versions.  v16.2.10-20220920 is really more verbose or v16.2.11-20230125 does not execute all the detection process

Patrick

Le 12/10/2023 à 09:34, Eugen Block a écrit :

...

Good catch, and I found the thread I had in my mind, it was this exact one. :-D Anyway, can you share the ceph-volume.log from the working and the not working attempt? I tried to look for something significant in the pacific release notes for 16.2.11, and there were some changes to ceph-volume, but I'm not sure what it could be.

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

I've ran additional tests with Pacific releases and with "ceph-volume inventory" things went wrong with the first v16.11 release (v16.2.11-20230125)

=================== Ceph v16.2.10-20220920 =======================

Device Path               Size         rotates available Model name /dev/sdc                  232.83 GB    True    True SAMSUNG HE253GJ /dev/sda                  232.83 GB    True    False SAMSUNG HE253GJ /dev/sdb                  465.76 GB    True    False     WDC WD5003ABYX-1

=================== Ceph v16.2.11-20230125 =======================

Device Path               Size         Device nodes rotates available Model name

May be this could help to see what has changed ?

Patrick

Le 11/10/2023 à 17:38, Eugen Block a écrit :

...

That's really strange. Just out of curiosity, have you tried Quincy (and/or Reef) as well? I don't recall what inventory does in the background exactly, I believe Adam King mentioned that in some thread, maybe that can help here. I'll search for that thread tomorrow.

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Hi Eugen,

[root@mostha1 ~]# rpm -q cephadm cephadm-16.2.14-0.el8.noarch

Log associated to the

2023-10-11 16:16:02,167 7f820515fb80 DEBUG -------------------------------------------------------------------------------- cephadm ['gather-facts'] 2023-10-11 16:16:02,208 7f820515fb80 DEBUG /bin/podman: 4.4.1 2023-10-11 16:16:02,313 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,317 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,322 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,326 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,329 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:02,333 7f820515fb80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:04,474 7ff2a5c08b80 DEBUG -------------------------------------------------------------------------------- cephadm ['ceph-volume', 'inventory'] 2023-10-11 16:16:04,516 7ff2a5c08b80 DEBUG /usr/bin/podman: 4.4.1 2023-10-11 16:16:04,520 7ff2a5c08b80 DEBUG Using default config: /etc/ceph/ceph.conf 2023-10-11 16:16:04,573 7ff2a5c08b80 DEBUG /usr/bin/podman: 0d28d71358d7,445.8MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 2084faaf4d54,13.27MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 61073c53805d,512.7MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 6b9f0b72d668,361.1MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: 7493a28808ad,163.7MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: a89672a3accf,59.22MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: b45271cc9726,54.24MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: e00ec13ab138,707.3MB / 50.32GB 2023-10-11 16:16:04,574 7ff2a5c08b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,35.55MB / 50.32GB 2023-10-11 16:16:04,630 7ff2a5c08b80 DEBUG /usr/bin/podman: 0d28d71358d7,1.28% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 2084faaf4d54,0.00% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 61073c53805d,1.19% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 6b9f0b72d668,1.03% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: 7493a28808ad,0.78% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: a89672a3accf,0.11% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: b45271cc9726,1.35% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: e00ec13ab138,0.43% 2023-10-11 16:16:04,631 7ff2a5c08b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,0.02% 2023-10-11 16:16:04,634 7ff2a5c08b80 INFO Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c 2023-10-11 16:16:04,691 7ff2a5c08b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:16:04,692 7ff2a5c08b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:c08064dde4bba4e72a1f55d90ca32df9ef5aafab82efe2e0a0722444a5aaacca 2023-10-11 16:16:04,692 7ff2a5c08b80 DEBUG /usr/bin/podman: docker.io/ceph/ceph@sha256:056637972a107df4096f10951e4216b21fcd8ae0b9fb4552e628d35df3f61139 2023-10-11 16:16:04,694 7ff2a5c08b80 INFO Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:16:05,094 7ff2a5c08b80 DEBUG stat: 167 167 2023-10-11 16:16:05,903 7ff2a5c08b80 DEBUG Acquiring lock 140679815723776 on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:16:05,903 7ff2a5c08b80 DEBUG Lock 140679815723776 acquired on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:16:05,929 7ff2a5c08b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:05,933 7ff2a5c08b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:16:06,700 7ff2a5c08b80 DEBUG /usr/bin/podman: 2023-10-11 16:16:06,701 7ff2a5c08b80 DEBUG /usr/bin/podman: Device Path               Size Device nodes rotates available Model name

I have only one version of cephadm in /var/lib/ceph/{fsid} : [root@mostha1 ~]# ls -lrt /var/lib/ceph/250f9864-0142-11ee-8e5f-00266cf8869c/cephadm* -rw-r--r-- 1 root root 350889 28 sept. 16:39 /var/lib/ceph/250f9864-0142-11ee-8e5f-00266cf8869c/cephadm.f6868821c084cd9740b59c7c5eb59f0dd47f6e3b1e6fecb542cb44134ace8d78

Running " python3 /var/lib/ceph/250f9864-0142-11ee-8e5f-00266cf8869c/cephadm.f6868821c084cd9740b59c7c5eb59f0dd47f6e3b1e6fecb542cb44134ace8d78 ceph-volume inventory" give the same output and the same log (execpt the valu of the lock):

2023-10-11 16:21:35,965 7f467cf31b80 DEBUG -------------------------------------------------------------------------------- cephadm ['ceph-volume', 'inventory'] 2023-10-11 16:21:36,009 7f467cf31b80 DEBUG /usr/bin/podman: 4.4.1 2023-10-11 16:21:36,012 7f467cf31b80 DEBUG Using default config: /etc/ceph/ceph.conf 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 0d28d71358d7,452.1MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 2084faaf4d54,13.27MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 61073c53805d,513.6MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 6b9f0b72d668,322.4MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: 7493a28808ad,164MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: a89672a3accf,58.5MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: b45271cc9726,54.69MB / 50.32GB 2023-10-11 16:21:36,067 7f467cf31b80 DEBUG /usr/bin/podman: e00ec13ab138,707.1MB / 50.32GB 2023-10-11 16:21:36,068 7f467cf31b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,36.28MB / 50.32GB 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 0d28d71358d7,1.27% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 2084faaf4d54,0.00% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 61073c53805d,1.16% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 6b9f0b72d668,1.02% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: 7493a28808ad,0.78% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: a89672a3accf,0.11% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: b45271cc9726,1.35% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: e00ec13ab138,0.41% 2023-10-11 16:21:36,125 7f467cf31b80 DEBUG /usr/bin/podman: fcb1e1a6b08d,0.02% 2023-10-11 16:21:36,128 7f467cf31b80 INFO Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c 2023-10-11 16:21:36,186 7f467cf31b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:21:36,187 7f467cf31b80 DEBUG /usr/bin/podman: quay.io/ceph/ceph@sha256:c08064dde4bba4e72a1f55d90ca32df9ef5aafab82efe2e0a0722444a5aaacca 2023-10-11 16:21:36,187 7f467cf31b80 DEBUG /usr/bin/podman: docker.io/ceph/ceph@sha256:056637972a107df4096f10951e4216b21fcd8ae0b9fb4552e628d35df3f61139 2023-10-11 16:21:36,189 7f467cf31b80 INFO Using recent ceph image quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e 2023-10-11 16:21:36,549 7f467cf31b80 DEBUG stat: 167 167 2023-10-11 16:21:36,942 7f467cf31b80 DEBUG Acquiring lock 139940396923424 on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:21:36,942 7f467cf31b80 DEBUG Lock 139940396923424 acquired on /run/cephadm/250f9864-0142-11ee-8e5f-00266cf8869c.lock 2023-10-11 16:21:36,969 7f467cf31b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:21:36,972 7f467cf31b80 DEBUG sestatus: SELinux status:                 disabled 2023-10-11 16:21:37,749 7f467cf31b80 DEBUG /usr/bin/podman: 2023-10-11 16:21:37,750 7f467cf31b80 DEBUG /usr/bin/podman: Device Path               Size Device nodes rotates available Model name

Patrick

Le 11/10/2023 à 15:59, Eugen Block a écrit : > Can you check which cephadm version is installed on the host? > And then please add (only the relevant) output from the > cephadm.log when you run the inventory (without the --image > <octopus>). Sometimes the version mismatch on the host and the > one the orchestrator uses can cause some disruptions. You could > try the same with the latest cephadm you have in > /var/lib/ceph/${fsid}/ (ls -lrt > /var/lib/ceph/${fsid}/cephadm.*). I mentioned that in this > thread [1]. So you could try the following: > > $ chmod +x /var/lib/ceph/{fsid}/cephadm.{latest} > > $ python3 /var/lib/ceph/{fsid}/cephadm.{latest} ceph-volume > inventory > > Does the output differ? Paste the relevant cephadm.log from that > attempt as well. > > [1] > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LASBJCSPFGDY... > > Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>: > >> Hi Eugen, >> >> first many thanks for the time spent on this problem. >> >> "ceph osd purge 2 --force --yes-i-really-mean-it" works and >> clean all the bas status. >> >> *[root@mostha1 ~]# cephadm shell >> *Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c >> Using recent ceph image >> quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e >> * >> * >> *[ceph: root@mostha1 /]# ceph osd purge 2 --force >> --yes-i-really-mean-it * >> purged osd.2 >> * >> * >> *[ceph: root@mostha1 /]# ceph osd tree* >> ID  CLASS  WEIGHT   TYPE NAME         STATUS REWEIGHT PRI-AFF >> -1         1.72823  root default >> -5         0.45477      host dean >>  0    hdd  0.22739          osd.0         up 1.00000 1.00000 >>  4    hdd  0.22739          osd.4         up 1.00000 1.00000 >> -9         0.22739      host ekman >>  6    hdd  0.22739          osd.6         up 1.00000 1.00000 >> -7         0.45479      host mostha1 >>  5    hdd  0.45479          osd.5         up 1.00000 1.00000 >> -3         0.59128      host mostha2 >>  1    hdd  0.22739          osd.1         up 1.00000 1.00000 >>  3    hdd  0.36389          osd.3         up 1.00000 1.00000 >> * >> * >> *[ceph: root@mostha1 /]# lsblk* >> NAME MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT >> sda 8:0    1 232.9G  0 disk >> |-sda1 8:1    1   3.9G  0 part /rootfs/boot >> |-sda2 8:2    1   3.9G  0 part [SWAP] >> `-sda3 8:3    1   225G  0 part >> |-al8vg-rootvol 253:0    0  48.8G  0 lvm  /rootfs >> |-al8vg-homevol 253:2    0   9.8G  0 lvm /rootfs/home >> |-al8vg-tmpvol 253:3    0   9.8G  0 lvm  /rootfs/tmp >> `-al8vg-varvol 253:4    0  19.8G  0 lvm  /rootfs/var >> sdb 8:16   1 465.8G  0 disk >> `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 >> 253:1    0 465.8G  0 lvm >> sdc 8:32   1 232.9G  0 disk >> >> "cephadm ceph-volume inventory" returns nothing: >> >> *[root@mostha1 ~]# cephadm ceph-volume inventory ** >> *Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c >> Using recent ceph image >> quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e >> >> Device Path               Size         Device nodes rotates >> available Model name >> >> [root@mostha1 ~]# >> >> But running the same command within cephadm 15.2.17 works: >> >> *[root@mostha1 ~]# cephadm --image 93146564743f ceph-volume >> inventory* >> Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c >> >> Device Path               Size         rotates available Model >> name >> /dev/sdc                  232.83 GB    True    True SAMSUNG >> HE253GJ >> /dev/sda                  232.83 GB    True    False SAMSUNG >> HE253GJ >> /dev/sdb                  465.76 GB    True    False WDC >> WD5003ABYX-1 >> >> [root@mostha1 ~]# >> >> *[root@mostha1 ~]# podman images -a** >> *REPOSITORY                        TAG         IMAGE ID >> CREATED        SIZE >> quay.io/ceph/ceph                 v16.2.14 f13d80acdbb5 2 weeks >> ago    1.21 GB >> quay.io/ceph/ceph                 v15.2.17 93146564743f 14 >> months ago  1.24 GB >> .... >> >> >> Patrick >> >> Le 11/10/2023 à 15:14, Eugen Block a écrit : >>> Your response is a bit confusing since it seems to be mixed up >>> with the previous answer. So you still need to remove the OSD >>> properly, so purge it from the crush tree: >>> >>> ceph osd purge 2 --force --yes-i-really-mean-it (only in a >>> test cluster!) >>> >>> If everything is clean (OSD has been removed, disk has been >>> zapped, lsblk shows no LVs for that disk) you can check the >>> inventory: >>> >>> cephadm ceph-volume inventory >>> >>> Please also add the output of 'ceph orch ls osd --export'. >>> >>> Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>: >>> >>>> Hi Eugen, >>>> >>>> - the OS is Alma Linux 8 with latests updates. >>>> >>>> - this morning I've worked with ceph-volume but it ends with >>>> a strange final state. I was connected on host mostha1 where >>>> /dev/sdc was not reconized. These are the steps followed >>>> based on the ceph-volume documentation I've read: >>>> [root@mostha1 ~]# cephadm shell >>>> [ceph: root@mostha1 /]# ceph auth get client.bootstrap-osd > >>>> /var/lib/ceph/bootstrap-osd/ceph.keyring >>>> [ceph: root@mostha1 /]# ceph-volume lvm prepare --bluestore >>>> --data /dev/sdc >>>> >>>> Now lsblk command shows sdc as an osd: >>>> .... >>>> sdb 8:16   1 465.8G  0 disk >>>> `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 >>>> 253:1    0 465.8G  0 lvm >>>> sdc 8:32   1 232.9G  0 disk >>>> `-ceph--b27d7a07--278d--4ee2--b84e--53256ef8de4c-osd--block--45c8e92c--caf9--4fe7--9a42--7b45a0794632 >>>> 253:5    0 232.8G  0 lvm >>>> >>>> Then I've tried to activate this osd but it fails as in >>>> podman I have not access to systemctl: >>>> >>>> [ceph: root@mostha1 /]# ceph-volume lvm activate 2 >>>> 45c8e92c-caf9-4fe7-9a42-7b45a0794632 >>>> ..... >>>> Running command: /usr/bin/systemctl start ceph-osd@2 >>>>  stderr: Failed to connect to bus: No such file or directory >>>> -->  RuntimeError: command returned non-zero exit status: 1 >>>> [ceph: root@mostha1 /]# ceph osd tree >>>> >>>> And now I have now I have a strange status for this osd.2: >>>> >>>> [ceph: root@mostha1 /]# ceph osd tree >>>> ID  CLASS  WEIGHT   TYPE NAME         STATUS REWEIGHT PRI-AFF >>>> -1         1.72823  root default >>>> -5         0.45477      host dean >>>>  0    hdd  0.22739          osd.0         up 1.00000 1.00000 >>>>  4    hdd  0.22739          osd.4         up 1.00000 1.00000 >>>> -9         0.22739      host ekman >>>>  6    hdd  0.22739          osd.6         up 1.00000 1.00000 >>>> -7         0.45479      host mostha1 >>>>  5    hdd  0.45479          osd.5         up 1.00000 1.00000 >>>> -3         0.59128      host mostha2 >>>>  1    hdd  0.22739          osd.1         up 1.00000 1.00000 >>>>  3    hdd  0.36389          osd.3         up 1.00000 1.00000 >>>>  2               0  osd.2               down 0 1.00000 >>>> >>>> I've tried to destroy the osd as you suggest but even if the >>>> command returns no error I still have this osd even if >>>> "lsblk" do not show any more /dev/sdc as a ceph osd device. >>>> >>>> *[ceph: root@mostha1 /]# ceph-volume lvm zap --destroy >>>> /dev/sdc** >>>> *--> Zapping: /dev/sdc >>>> --> Zapping lvm member /dev/sdc. lv_path is >>>> /dev/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c/osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632 >>>> --> Unmounting /var/lib/ceph/osd/ceph-2 >>>> Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-2 >>>>  stderr: umount: /var/lib/ceph/osd/ceph-2 unmounted >>>> Running command: /usr/bin/dd if=/dev/zero >>>> of=/dev/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c/osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632 >>>> bs=1M count=10 conv=fsync >>>>  stderr: 10+0 records in >>>> 10+0 records out >>>> 10485760 bytes (10 MB, 10 MiB) copied, 0.575633 s, 18.2 MB/s >>>> --> Only 1 LV left in VG, will proceed to destroy volume >>>> group ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c >>>> Running command: nsenter --mount=/rootfs/proc/1/ns/mnt >>>> --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net >>>> --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f >>>> ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c >>>>  stderr: Removing >>>> ceph--b27d7a07--278d--4ee2--b84e--53256ef8de4c-osd--block--45c8e92c--caf9--4fe7--9a42--7b45a0794632 >>>> (253:1) >>>>  stderr: Releasing logical volume >>>> "osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632" >>>>  stderr: Archiving volume group >>>> "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" metadata (seqno 5). >>>>  stdout: Logical volume >>>> "osd-block-45c8e92c-caf9-4fe7-9a42-7b45a0794632" successfully >>>> removed. >>>>  stderr: Removing physical volume "/dev/sdc" from volume >>>> group "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" >>>>  stdout: Volume group >>>> "ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" successfully removed >>>>  stderr: Creating volume group backup >>>> "/etc/lvm/backup/ceph-b27d7a07-278d-4ee2-b84e-53256ef8de4c" >>>> (seqno 6). >>>> Running command: nsenter --mount=/rootfs/proc/1/ns/mnt >>>> --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net >>>> --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/sdc >>>>  stdout: Labels on physical volume "/dev/sdc" successfully >>>> wiped. >>>> Running command: /usr/bin/dd if=/dev/zero of=/dev/sdc bs=1M >>>> count=10 conv=fsync >>>>  stderr: 10+0 records in >>>> 10+0 records out >>>> 10485760 bytes (10 MB, 10 MiB) copied, 0.590652 s, 17.8 MB/s >>>> *--> Zapping successful for: <Raw Device: /dev/sdc>* >>>> * >>>> * >>>> *[ceph: root@mostha1 /]# ceph osd tree** >>>> *ID  CLASS  WEIGHT   TYPE NAME         STATUS REWEIGHT PRI-AFF >>>> -1         1.72823  root default >>>> -5         0.45477      host dean >>>>  0    hdd  0.22739          osd.0         up 1.00000 1.00000 >>>>  4    hdd  0.22739          osd.4         up 1.00000 1.00000 >>>> -9         0.22739      host ekman >>>>  6    hdd  0.22739          osd.6         up 1.00000 1.00000 >>>> -7         0.45479      host mostha1 >>>>  5    hdd  0.45479          osd.5         up 1.00000 1.00000 >>>> -3         0.59128      host mostha2 >>>>  1    hdd  0.22739          osd.1         up 1.00000 1.00000 >>>>  3    hdd  0.36389          osd.3         up 1.00000 1.00000 >>>>  2               0  osd.2               down 0 1.00000 >>>> * >>>> * >>>> *[ceph: root@mostha1 /]# lsblk** >>>> *NAME MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT >>>> sda 8:0    1 232.9G  0 disk >>>> |-sda1 8:1    1   3.9G  0 part /rootfs/boot >>>> |-sda2 8:2    1   3.9G  0 part [SWAP] >>>> `-sda3 8:3    1   225G  0 part >>>> |-al8vg-rootvol 253:0    0  48.8G  0 lvm /rootfs >>>> |-al8vg-homevol 253:3    0   9.8G  0 lvm /rootfs/home >>>> |-al8vg-tmpvol 253:4    0   9.8G  0 lvm /rootfs/tmp >>>> `-al8vg-varvol 253:5    0  19.8G  0 lvm /rootfs/var >>>> sdb 8:16   1 465.8G  0 disk >>>> `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 >>>> 253:2    0 465.8G  0 lvm >>>> *sdc * >>>> >>>> Patrick >>>> Le 11/10/2023 à 11:00, Eugen Block a écrit : >>>>> Hi, >>>>> >>>>> just wondering if 'ceph-volume lvm zap --destroy /dev/sdc' >>>>> would help here. From your previous output you didn't >>>>> specify the --destroy flag. >>>>> Which cephadm version is installed on the host? Did you also >>>>> upgrade the OS when moving to Pacific? (Sorry if I missed that. >>>>> >>>>> >>>>> Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>: >>>>> >>>>>> Le 02/10/2023 à 18:22, Patrick Bégou a écrit : >>>>>>> Hi all, >>>>>>> >>>>>>> still stuck with this problem. >>>>>>> >>>>>>> I've deployed octopus and all my HDD have been setup as >>>>>>> osd. Fine. >>>>>>> I've upgraded to pacific and 2 osd have failed. They have >>>>>>> been automatically removed and upgrade finishes. Cluster >>>>>>> Health is finaly OK, no data loss. >>>>>>> >>>>>>> But now I cannot re-add these osd with pacific (I had >>>>>>> previous troubles on these old HDDs, lost one osd in >>>>>>> octopus and was able to reset and re-add it). >>>>>>> >>>>>>> I've tried manually to add the first osd on the node where >>>>>>> it is located, following >>>>>>> https://docs.ceph.com/en/pacific/rados/operations/bluestore-migration/ >>>>>>> (not sure it's the best idea...) but it fails too. This >>>>>>> node was the one used for deploying the cluster. >>>>>>> >>>>>>> [ceph: root@mostha1 /]# *ceph-volume lvm zap /dev/sdc* >>>>>>> --> Zapping: /dev/sdc >>>>>>> --> --destroy was not specified, but zapping a whole >>>>>>> device will remove the partition table >>>>>>> Running command: /usr/bin/dd if=/dev/zero of=/dev/sdc >>>>>>> bs=1M count=10 conv=fsync >>>>>>>  stderr: 10+0 records in >>>>>>> 10+0 records out >>>>>>> 10485760 bytes (10 MB, 10 MiB) copied, 0.663425 s, 15.8 MB/s >>>>>>> --> Zapping successful for: <Raw Device: /dev/sdc> >>>>>>> >>>>>>> >>>>>>> [ceph: root@mostha1 /]# *ceph-volume lvm create >>>>>>> --bluestore --data /dev/sdc* >>>>>>> Running command: /usr/bin/ceph-authtool --gen-print-key >>>>>>> Running command: /usr/bin/ceph --cluster ceph --name >>>>>>> client.bootstrap-osd --keyring >>>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new >>>>>>> 9f1eb8ee-41e6-4350-ad73-1be21234ec7c >>>>>>>  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 >>>>>>> auth: unable to find a keyring on >>>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file >>>>>>> or directory >>>>>>>  stderr: 2023-10-02T16:09:29.855+0000 7fb4eb8c0700 -1 >>>>>>> AuthRegistry(0x7fb4e405c4d8) no keyring found at >>>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx >>>>>>>  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 >>>>>>> auth: unable to find a keyring on >>>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file >>>>>>> or directory >>>>>>>  stderr: 2023-10-02T16:09:29.856+0000 7fb4eb8c0700 -1 >>>>>>> AuthRegistry(0x7fb4e40601d0) no keyring found at >>>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx >>>>>>>  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 >>>>>>> auth: unable to find a keyring on >>>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file >>>>>>> or directory >>>>>>>  stderr: 2023-10-02T16:09:29.857+0000 7fb4eb8c0700 -1 >>>>>>> AuthRegistry(0x7fb4eb8bee90) no keyring found at >>>>>>> /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx >>>>>>>  stderr: 2023-10-02T16:09:29.858+0000 7fb4e965c700 -1 >>>>>>> monclient(hunting): handle_auth_bad_method server >>>>>>> allowed_methods [2] but i only support [1] >>>>>>>  stderr: 2023-10-02T16:09:29.858+0000 7fb4e9e5d700 -1 >>>>>>> monclient(hunting): handle_auth_bad_method server >>>>>>> allowed_methods [2] but i only support [1] >>>>>>>  stderr: 2023-10-02T16:09:29.858+0000 7fb4e8e5b700 -1 >>>>>>> monclient(hunting): handle_auth_bad_method server >>>>>>> allowed_methods [2] but i only support [1] >>>>>>>  stderr: 2023-10-02T16:09:29.858+0000 7fb4eb8c0700 -1 >>>>>>> monclient: authenticate NOTE: no keyring found; disabled >>>>>>> cephx authentication >>>>>>>  stderr: [errno 13] RADOS permission denied (error >>>>>>> connecting to the cluster) >>>>>>> -->  RuntimeError: Unable to create a new OSD id >>>>>>> >>>>>>> Any idea of what is wrong ? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Patrick >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>> >>>>>> >>>>>> I'm still trying to understand what can be wrong or how to >>>>>> debug this situation where Ceph cannot see the devices. >>>>>> >>>>>> The device :dev/sdc exists: >>>>>> >>>>>>    [root@mostha1 ~]# cephadm shell lsmcli ldl >>>>>>    Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c >>>>>>    Using recent ceph image >>>>>> quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e >>>>>>    Path     | SCSI VPD 0x83    | Link Type | Serial Number >>>>>> | Health >>>>>>    Status ceph osd purge 2 --force --yes-i-really-mean-it >>>>>> ------------------------------------------------------------------------- >>>>>>    /dev/sda | 50024e92039e4f1c | PATA/SATA | S2B5J90ZA10142 >>>>>> | Good >>>>>>    /dev/sdb | 50014ee0ad5953c9 | PATA/SATA | >>>>>> WD-WMAYP0982329 | Good >>>>>>    /dev/sdc | 50024e920387fa2c | PATA/SATA | S2B5J90ZA02494 >>>>>> | Good >>>>>> >>>>>> But I cannot do anything with it: >>>>>> >>>>>>    [root@mostha1 ~]# cephadm shell ceph orch device zap >>>>>>    mostha1.legi.grenoble-inp.fr /dev/sdc --force >>>>>>    Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c >>>>>>    Using recent ceph image >>>>>> quay.io/ceph/ceph@sha256:f30bf50755d7087f47c6223e6a921caf5b12e86401b3d49220230c84a8302a1e >>>>>>    Error EINVAL: Device path '/dev/sdc' not found on host >>>>>>    'mostha1.legi.grenoble-inp.fr' >>>>>> >>>>>> Since I moved from octopus to Pacific. >>>>>> >>>>>> Patrick >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>> >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

I think you can try to roll back this part of the python code and wait for your good news :)

It seems that the only way to modify the code is manually ...

Hi Eugen, Yes Eugen, all the devices /dev/sd[abc] have the removable flag set to 1. May be because they are hot-swappable hard drives. I have contacted the commit author Zack Cerza and he asked me for some additional tests too this morning. I add him in copy to this mail. Patrick Le 24/10/2023 à 12:57, Eugen Block a écrit :

Hi,

just to confirm, could you check that the disk which is *not* discovered by 16.2.11 has a "removable" flag?

cat /sys/block/sdX/removable

I could reproduce it as well on a test machine with a USB thumb drive (live distro) which is excluded in 16.2.11 but is shown in 16.2.10. Although I'm not a developer I tried to understand what changes were made in https://github.com/ceph/ceph/pull/46375/files#diff-330f9319b0fe352dff0486f66... and there's this line:

...

if get_file_contents(os.path.join(_sys_block_path, dev, 'removable')) == "1":    continue

The thumb drive is removable, of course, apparently that is filtered here.

Regards, Eugen

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Le 23/10/2023 à 03:04, 544463199@qq.com a écrit :

...

I think you can try to roll back this part of the python code and wait for your good news :)

Not so easy 😕

[root@e9865d9a7f41 ceph]# git revert 4fc6bc394dffaf3ad375ff29cbb0a3eb9e4dbefc Auto-merging src/ceph-volume/ceph_volume/tests/util/test_device.py CONFLICT (content): Merge conflict in src/ceph-volume/ceph_volume/tests/util/test_device.py Auto-merging src/ceph-volume/ceph_volume/util/device.py CONFLICT (content): Merge conflict in src/ceph-volume/ceph_volume/util/device.py Auto-merging src/ceph-volume/ceph_volume/util/disk.py CONFLICT (content): Merge conflict in src/ceph-volume/ceph_volume/util/disk.py error: could not revert 4fc6bc394df... ceph-volume: Optionally consume loop devices

Patrick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

That's correct - it's the removable flag that's causing the disks to be excluded. I actually just merged this PR last week: https://github.com/ceph/ceph/pull/49954 One of the changes it made was to enable removable (but not USB) devices, as there are vendors that report hot-swappable drives as removable. Patrick, it looks like this may resolve your issue as well. On Tue, Oct 24, 2023 at 5:57 AM Eugen Block <eblock@nde.ag> wrote:

Hi,

...

May be because they are hot-swappable hard drives.

yes, that's my assumption as well.

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Hi Eugen,

Yes Eugen, all the devices /dev/sd[abc] have the removable flag set to 1. May be because they are hot-swappable hard drives.

I have contacted the commit author Zack Cerza and he asked me for some additional tests too this morning. I add him in copy to this mail.

Patrick

Le 24/10/2023 à 12:57, Eugen Block a écrit :

...

Hi,

just to confirm, could you check that the disk which is *not* discovered by 16.2.11 has a "removable" flag?

cat /sys/block/sdX/removable

I could reproduce it as well on a test machine with a USB thumb drive (live distro) which is excluded in 16.2.11 but is shown in 16.2.10. Although I'm not a developer I tried to understand what changes were made in https://github.com/ceph/ceph/pull/46375/files#diff-330f9319b0fe352dff0486f66... and there's this line:

...

if get_file_contents(os.path.join(_sys_block_path, dev, 'removable')) == "1": continue

The thumb drive is removable, of course, apparently that is filtered here.

Regards, Eugen

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Le 23/10/2023 à 03:04, 544463199@qq.com a écrit :

...

I think you can try to roll back this part of the python code and wait for your good news :)

Not so easy 😕

[root@e9865d9a7f41 ceph]# git revert 4fc6bc394dffaf3ad375ff29cbb0a3eb9e4dbefc Auto-merging src/ceph-volume/ceph_volume/tests/util/test_device.py CONFLICT (content): Merge conflict in src/ceph-volume/ceph_volume/tests/util/test_device.py Auto-merging src/ceph-volume/ceph_volume/util/device.py CONFLICT (content): Merge conflict in src/ceph-volume/ceph_volume/util/device.py Auto-merging src/ceph-volume/ceph_volume/util/disk.py CONFLICT (content): Merge conflict in src/ceph-volume/ceph_volume/util/disk.py error: could not revert 4fc6bc394df... ceph-volume: Optionally consume loop devices

Patrick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi all, First of all I apologize if I've not done things correctly but these are some tests results. 1) I've compiled the main branch in a fresh podman container (Alma Linux 8) and installed. Successfull! 2) I have done a copy of the /etc/ceph directory of the host (member of the ceph cluster in Pacific 16.2.14) in this container (good or bad idea ?) 3) "ceph-volume inventory" works but with some error messages: [root@74285dcfa91f etc]# ceph-volume inventory  stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected.  stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected.  stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected.  stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected.  stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected. Device Path               Size         Device nodes    rotates available Model name /dev/sdc                  232.83 GB    sdc             True True      SAMSUNG HE253GJ /dev/sda                  232.83 GB    sda             True False     SAMSUNG HE253GJ /dev/sdb                  465.76 GB    sdb             True False     WDC WD5003ABYX-1 4) ceph version show: [root@74285dcfa91f etc]# ceph -v ceph version 18.0.0-6846-g2706ecac4a9 (2706ecac4a90447420904e42d6e0445134dff2be) reef (dev) 5) lsblk works (container launched with "--privileged" flag) [root@74285dcfa91f etc]# lsblk NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT sda      8:0    1 232.9G  0 disk |-sda1   8:1        3.9G  0 part |-sda2   8:2    1   3.9G  0 part [SWAP] `-sda3   8:3    1   225G  0 part sdb      8:16   1 465.8G  0 disk sdc      8:32   1 232.9G  0 disk But some commands do not works (my setup or ceph ?) [root@74285dcfa91f etc]# ceph orch device zap mostha1.legi.grenoble-inp.fr /dev/sdc --force Error EINVAL: Device path '/dev/sdc' not found on host 'mostha1.legi.grenoble-inp.fr' [root@74285dcfa91f etc]# [root@74285dcfa91f etc]# ceph orch device ls [root@74285dcfa91f etc]# Patrick Le 24/10/2023 à 22:43, Zack Cerza a écrit :

That's correct - it's the removable flag that's causing the disks to be excluded.

I actually just merged this PR last week: https://github.com/ceph/ceph/pull/49954

One of the changes it made was to enable removable (but not USB) devices, as there are vendors that report hot-swappable drives as removable. Patrick, it looks like this may resolve your issue as well.

On Tue, Oct 24, 2023 at 5:57 AM Eugen Block <eblock@nde.ag> wrote:

...

Hi,

...

May be because they are hot-swappable hard drives. yes, that's my assumption as well.

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Hi Eugen,

Yes Eugen, all the devices /dev/sd[abc] have the removable flag set to 1. May be because they are hot-swappable hard drives.

I have contacted the commit author Zack Cerza and he asked me for some additional tests too this morning. I add him in copy to this mail.

Patrick

Le 24/10/2023 à 12:57, Eugen Block a écrit :

...

Hi,

just to confirm, could you check that the disk which is *not* discovered by 16.2.11 has a "removable" flag?

cat /sys/block/sdX/removable

I could reproduce it as well on a test machine with a USB thumb drive (live distro) which is excluded in 16.2.11 but is shown in 16.2.10. Although I'm not a developer I tried to understand what changes were made in https://github.com/ceph/ceph/pull/46375/files#diff-330f9319b0fe352dff0486f66... and there's this line:

...

if get_file_contents(os.path.join(_sys_block_path, dev, 'removable')) == "1": continue The thumb drive is removable, of course, apparently that is filtered here.

Regards, Eugen

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Le 23/10/2023 à 03:04, 544463199@qq.com a écrit :

...

I think you can try to roll back this part of the python code and wait for your good news :)

Not so easy 😕

[root@e9865d9a7f41 ceph]# git revert 4fc6bc394dffaf3ad375ff29cbb0a3eb9e4dbefc Auto-merging src/ceph-volume/ceph_volume/tests/util/test_device.py CONFLICT (content): Merge conflict in src/ceph-volume/ceph_volume/tests/util/test_device.py Auto-merging src/ceph-volume/ceph_volume/util/device.py CONFLICT (content): Merge conflict in src/ceph-volume/ceph_volume/util/device.py Auto-merging src/ceph-volume/ceph_volume/util/disk.py CONFLICT (content): Merge conflict in src/ceph-volume/ceph_volume/util/disk.py error: could not revert 4fc6bc394df... ceph-volume: Optionally consume loop devices

Patrick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Some tests: If in Nautilus 16.2.14 in /usr/lib/python3.6/site-packages/ceph_volume/util/disk.py I disable lines 804 and 805     804         if get_file_contents(os.path.join(_sys_block_path, dev, 'removable')) == "1":     805             continue the command "ceph-volume inventory" works as in Octopus or Nautilus < 16.2.11: [ceph: root@mostha1 /]# ceph-volume inventory Device Path               Size         Device nodes    rotates available Model name /dev/sdc                  232.83 GB    sdc             True True      SAMSUNG HE253GJ /dev/sda                  232.83 GB    sda             True False     SAMSUNG HE253GJ /dev/sdb                  465.76 GB    sdb             True False     WDC WD5003ABYX-1 but 1) "ceph orch device ls" still returns nothing. 2) I cannot zap the /dev/sdc device: [ceph: root@mostha1 /]# ceph orch device zap mostha1.legi.grenoble-inp.fr /dev/sdc --force Error EINVAL: Device path '/dev/sdc' not found on host 'mostha1.legi.grenoble-inp.fr' 3) I cannot manualy add the sdc device as an osd: [ceph: root@mostha1 /]# ceph orch daemon add osd mostha1.legi.grenoble-inp.fr:/dev/sdc Created no osd(s) on host mostha1.legi.grenoble-inp.fr; already created? Even is the device is present and unused: [ceph: root@mostha1 /]# lsblk NAME MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT sda 8:0    1 232.9G  0 disk |-sda1 8:1    1   3.9G  0 part /rootfs/boot |-sda2 8:2    1   3.9G  0 part [SWAP] `-sda3 8:3    1   225G  0 part |-al8vg-rootvol 253:0    0  48.8G  0 lvm  /rootfs |-al8vg-homevol 253:2    0   9.8G  0 lvm  /rootfs/home |-al8vg-tmpvol 253:3    0   9.8G  0 lvm  /rootfs/tmp `-al8vg-varvol 253:4    0  79.8G  0 lvm  /rootfs/var sdb 8:16   1 465.8G  0 disk `-ceph--08827fdc--136e--4070--97e9--e5e8b3970766-osd--block--7dec1808--d6f4--4f90--ac74--75a4346e1df5 253:1    0 465.8G  0 lvm sdc 8:32   1 232.9G  0 disk Patrick Le 24/10/2023 à 13:38, Patrick Begou a écrit :

Hi Eugen,

Yes Eugen, all the devices /dev/sd[abc] have the removable flag set to 1. May be because they are hot-swappable hard drives.

I have contacted the commit author Zack Cerza and he asked me for some additional tests too this morning. I add him in copy to this mail.

Patrick

Le 24/10/2023 à 12:57, Eugen Block a écrit :

...

Hi,

just to confirm, could you check that the disk which is *not* discovered by 16.2.11 has a "removable" flag?

cat /sys/block/sdX/removable

I could reproduce it as well on a test machine with a USB thumb drive (live distro) which is excluded in 16.2.11 but is shown in 16.2.10. Although I'm not a developer I tried to understand what changes were made in https://github.com/ceph/ceph/pull/46375/files#diff-330f9319b0fe352dff0486f66... and there's this line:

...

if get_file_contents(os.path.join(_sys_block_path, dev, 'removable')) == "1":    continue

The thumb drive is removable, of course, apparently that is filtered here.

Regards, Eugen

Zitat von Patrick Begou <Patrick.Begou@univ-grenoble-alpes.fr>:

...

Le 23/10/2023 à 03:04, 544463199@qq.com a écrit :

...

I think you can try to roll back this part of the python code and wait for your good news :)

Not so easy 😕

[root@e9865d9a7f41 ceph]# git revert 4fc6bc394dffaf3ad375ff29cbb0a3eb9e4dbefc Auto-merging src/ceph-volume/ceph_volume/tests/util/test_device.py CONFLICT (content): Merge conflict in src/ceph-volume/ceph_volume/tests/util/test_device.py Auto-merging src/ceph-volume/ceph_volume/util/device.py CONFLICT (content): Merge conflict in src/ceph-volume/ceph_volume/util/device.py Auto-merging src/ceph-volume/ceph_volume/util/disk.py CONFLICT (content): Merge conflict in src/ceph-volume/ceph_volume/util/disk.py error: could not revert 4fc6bc394df... ceph-volume: Optionally consume loop devices

Patrick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io