12 May
2023
12 May
'23
10:45 p.m.
Hi everyone
I'm new to CEPH, just a french 4 days training session with Octopus on
VMs that convince me to build my first cluster.
At this time I have 4 old identical nodes for testing with 3 HDDs each,
2 network interfaces and running Alma Linux8 (el8). I try to replay the
training session but it fails, breaking the web interface because of
some problems with podman 4.2 not compatible with Octopus.
So I try to deploy Pacific with cephadm tool on my first node (mostha1)
(to enable testing also an upgrade later).
dnf -y install
https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noar…
monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }')
cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \
--initial-dashboard-user admceph \
--allow-fqdn-hostname --cluster-network 10.1.0.0/16
This was sucessfull.
But running "*c**eph orch device ls*" do not show any HDD even if I have
/dev/sda (used by the OS), /dev/sdb and /dev/sdc
The web interface shows a row capacity which is an aggregate of the
sizes of the 3 HDDs for the node.
I've also tried to reset /dev/sdb but cephadm do not see it:
[ceph: root@mostha1 /]# ceph orch device zap
mostha1.legi.grenoble-inp.fr /dev/sdb --force
Error EINVAL: Device path '/dev/sdb' not found on host
'mostha1.legi.grenoble-inp.fr'
On my first attempt with octopus, I was able to list the available HDD
with this command line. Before moving to Pacific, the OS on this node
has been reinstalled from scratch.
Any advices for a CEPH beginner ?
Thanks
Patrick
13 May
13 May
1:19 a.m.
New subject: [EXTERNAL] [Pacific] ceph orch device ls do not returns any HDD
I don’t quite understand why that zap would not work. But, here’s where I’d start.
1. cephadm check-host
* Run this on each of your hosts to make sure cephadm, podman and all other
prerequisites are installed and recognized
2. ceph orch ls
* This should show at least a mon, mgr, and osd spec deployed
3. ceph orch ls osd –export
* This will show the OSD placement service specifications that orchestrator uses to
identify devices to deploy as OSDs
4. ceph orch host ls
* This will list the hosts that have been added to orchestrator’s inventory, and
what labels are applied which correlate to the service placement labels
5. ceph log last cephadm
* This will show you what orchestrator has been trying to do, and how it may be
failing
Also, it’s never un-helpful to have a look at “ceph -s” and “ceph health detail”,
particularly for any people trying to help you without access to your systems.
Best of luck,
Josh Beaman
From: Patrick Begou <Patrick.Begou(a)univ-grenoble-alpes.fr>
Date: Friday, May 12, 2023 at 10:45 AM
To: ceph-users <ceph-users(a)ceph.io>
Subject: [EXTERNAL] [ceph-users] [Pacific] ceph orch device ls do not returns any HDD
Hi everyone
I'm new to CEPH, just a french 4 days training session with Octopus on
VMs that convince me to build my first cluster.
At this time I have 4 old identical nodes for testing with 3 HDDs each,
2 network interfaces and running Alma Linux8 (el8). I try to replay the
training session but it fails, breaking the web interface because of
some problems with podman 4.2 not compatible with Octopus.
So I try to deploy Pacific with cephadm tool on my first node (mostha1)
(to enable testing also an upgrade later).
dnf -y install
https://urldefense.com/v3/__https://download.ceph.com/rpm-16.2.13/el8/noarc…
monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }')
cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \
--initial-dashboard-user admceph \
--allow-fqdn-hostname --cluster-network 10.1.0.0/16
This was sucessfull.
But running "*c**eph orch device ls*" do not show any HDD even if I have
/dev/sda (used by the OS), /dev/sdb and /dev/sdc
The web interface shows a row capacity which is an aggregate of the
sizes of the 3 HDDs for the node.
I've also tried to reset /dev/sdb but cephadm do not see it:
[ceph: root@mostha1 /]# ceph orch device zap
mostha1.legi.grenoble-inp.fr /dev/sdb --force
Error EINVAL: Device path '/dev/sdb' not found on host
'mostha1.legi.grenoble-inp.fr'
On my first attempt with octopus, I was able to list the available HDD
with this command line. Before moving to Pacific, the OS on this node
has been reinstalled from scratch.
Any advices for a CEPH beginner ?
Thanks
Patrick
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:22 a.m.
New subject: [EXTERNAL] [Pacific] ceph orch device ls do not returns any HDD
Hi Joshua and thanks for this quick reply.
At this step I have only one node. I was checking what ceph was
returning with different commands on this host before adding new hosts.
Just to compare with my first Octopus install. As this hardware is for
testing only, it remains easy for me to break everything and reinstall
again.
[root@mostha1 ~]# cephadm check-host
podman (/usr/bin/podman) version 4.2.0 is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
[ceph: root@mostha1 /]# ceph -s
cluster:
id: 4b7a6504-f0be-11ed-be1a-00266cf8869c
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 3
services:
mon: 1 daemons, quorum mostha1.legi.grenoble-inp.fr (age 5h)
mgr: mostha1.legi.grenoble-inp.fr.hogwuz(active, since 5h)
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
[ceph: root@mostha1 /]# ceph orch ls
NAME PORTS RUNNING REFRESHED AGE PLACEMENT
alertmanager ?:9093,9094 1/1 6m ago 6h count:1
crash 1/1 6m ago 6h *
grafana ?:3000 1/1 6m ago 6h count:1
mgr 1/2 6m ago 6h count:2
mon 1/5 6m ago 6h count:5
node-exporter ?:9100 1/1 6m ago 6h *
prometheus ?:9095 1/1 6m ago 6h count:1
[ceph: root@mostha1 /]# ceph orch ls osd -export
No services reported
[ceph: root@mostha1 /]# ceph orch host ls
HOST ADDR LABELS STATUS
mostha1.legi.grenoble-inp.fr 194.254.66.34 _admin
1 hosts in cluster
[ceph: root@mostha1 /]# ceph log last cephadm
...
2023-05-12T15:19:58.754655+0000
mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1876 : cephadm
[INF] Zap device mostha1.legi.grenoble-inp.fr:/dev/sdb
2023-05-12T15:19:58.756639+0000
mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1877 : cephadm
[ERR] Device path '/dev/sdb' not found on host
'mostha1.legi.grenoble-inp.fr'
Traceback (most recent call last):
File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125,
in wrapper
return OrchResult(f(*args, **kwargs))
File "/usr/share/ceph/mgr/cephadm/module.py", line 2275, in
zap_device
f"Device path '{path}' not found on host '{host}'")
orchestrator._interface.OrchestratorError: Device path '/dev/sdb'
not found on host 'mostha1.legi.grenoble-inp.fr'
....
[ceph: root@mostha1 /]# ls -l /dev/sdb
brw-rw---- 1 root disk 8, 16 May 12 15:16 /dev/sdb
[ceph: root@mostha1 /]# lsblk /dev/sdb
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 1 465.8G 0 disk
`-sdb1 8:17 1 465.8G 0 part
I have crated a full partition on /dev/sdb (for testing) and /dev/sdc
has no partition table (removed).
But all seams fine with these commands.
Patrick
Le 12/05/2023 à 20:19, Beaman, Joshua a écrit :
I don’t quite understand why that zap would not work. But, here’s
where I’d start.
1. cephadm check-host
1. Run this on each of your hosts to make sure cephadm, podman
and all other prerequisites are installed and recognized
2. ceph orch ls
1. This should show at least a mon, mgr, and osd spec deployed
3. ceph orch ls osd –export
1. This will show the OSD placement service specifications that
orchestrator uses to identify devices to deploy as OSDs
4. ceph orch host ls
1. This will list the hosts that have been added to
orchestrator’s inventory, and what labels are applied which
correlate to the service placement labels
5. ceph log last cephadm
1. This will show you what orchestrator has been trying to do,
and how it may be failing
Also, it’s never un-helpful to have a look at “ceph -s” and “ceph
health detail”, particularly for any people trying to help you without
access to your systems.
Best of luck,
Josh Beaman
*From: *Patrick Begou <Patrick.Begou(a)univ-grenoble-alpes.fr>
*Date: *Friday, May 12, 2023 at 10:45 AM
*To: *ceph-users <ceph-users(a)ceph.io>
*Subject: *[EXTERNAL] [ceph-users] [Pacific] ceph orch device ls do
not returns any HDD
Hi everyone
I'm new to CEPH, just a french 4 days training session with Octopus on
VMs that convince me to build my first cluster.
At this time I have 4 old identical nodes for testing with 3 HDDs each,
2 network interfaces and running Alma Linux8 (el8). I try to replay the
training session but it fails, breaking the web interface because of
some problems with podman 4.2 not compatible with Octopus.
So I try to deploy Pacific with cephadm tool on my first node (mostha1)
(to enable testing also an upgrade later).
dnf -y install
https://urldefense.com/v3/__https://download.ceph.com/rpm-16.2.13/el8/noarc…
<https://urldefense.com/v3/__https:/download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noarch.rpm__;!!CQl3mcHX2A!H9cwNCJyKXYQ4BbGA3gwHHRitjOS4lBCZT9wlnBZ-8IDue0MvdcPD8Dnv5yQCZw_eA4BNDYaEq1eouKQcQO7HshgdUJ0SJ-EgLfaBGBmCQ$>
monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }')
cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \
--initial-dashboard-user admceph \
--allow-fqdn-hostname --cluster-network 10.1.0.0/16
This was sucessfull.
But running "*c**eph orch device ls*" do not show any HDD even if I have
/dev/sda (used by the OS), /dev/sdb and /dev/sdc
The web interface shows a row capacity which is an aggregate of the
sizes of the 3 HDDs for the node.
I've also tried to reset /dev/sdb but cephadm do not see it:
[ceph: root@mostha1 /]# ceph orch device zap
mostha1.legi.grenoble-inp.fr /dev/sdb --force
Error EINVAL: Device path '/dev/sdb' not found on host
'mostha1.legi.grenoble-inp.fr'
On my first attempt with octopus, I was able to list the available HDD
with this command line. Before moving to Pacific, the OS on this node
has been reinstalled from scratch.
Any advices for a CEPH beginner ?
Thanks
Patrick
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:38 a.m.
New subject: [EXTERNAL] [Pacific] ceph orch device ls do not returns any HDD
The most significant point I see there, is you have no OSD service spec to tell
orchestrator how to deploy OSDs. The easiest fix for that would be “ceph orch apply osd
--all-available-devices”
This will create a simple spec that should work for a test environment. Most likely it
will collocate the block, block.db, and WAL all on the same device. Not ideal for prod
environments, but fine for practice and testing.
The other command I should have had you try is “cephadm ceph-volume inventory”. That
should show you the devices available for OSD deployment, and hopefully matches up to what
your “lsblk” shows. If you need to zap HDDs and orchestrator is still not seeing them,
you can try “cephadm ceph-volume lvm zap /dev/sdb”
Thank you,
Josh Beaman
From: Patrick Begou <Patrick.Begou(a)univ-grenoble-alpes.fr>
Date: Friday, May 12, 2023 at 2:22 PM
To: Beaman, Joshua <Joshua_Beaman(a)comcast.com>om>, ceph-users
<ceph-users(a)ceph.io>
Subject: Re: [EXTERNAL] [ceph-users] [Pacific] ceph orch device ls do not returns any HDD
Hi Joshua and thanks for this quick reply.
At this step I have only one node. I was checking what ceph was returning with different
commands on this host before adding new hosts. Just to compare with my first Octopus
install. As this hardware is for testing only, it remains easy for me to break everything
and reinstall again.
[root@mostha1 ~]# cephadm check-host
podman (/usr/bin/podman) version 4.2.0 is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
[ceph: root@mostha1 /]# ceph -s
cluster:
id: 4b7a6504-f0be-11ed-be1a-00266cf8869c
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 3
services:
mon: 1 daemons, quorum mostha1.legi.grenoble-inp.fr (age 5h)
mgr: mostha1.legi.grenoble-inp.fr.hogwuz(active, since 5h)
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
[ceph: root@mostha1 /]# ceph orch ls
NAME PORTS RUNNING REFRESHED AGE PLACEMENT
alertmanager ?:9093,9094 1/1 6m ago 6h count:1
crash 1/1 6m ago 6h *
grafana ?:3000 1/1 6m ago 6h count:1
mgr 1/2 6m ago 6h count:2
mon 1/5 6m ago 6h count:5
node-exporter ?:9100 1/1 6m ago 6h *
prometheus ?:9095 1/1 6m ago 6h count:1
[ceph: root@mostha1 /]# ceph orch ls osd -export
No services reported
[ceph: root@mostha1 /]# ceph orch host ls
HOST ADDR LABELS STATUS
mostha1.legi.grenoble-inp.fr 194.254.66.34 _admin
1 hosts in cluster
[ceph: root@mostha1 /]# ceph log last cephadm
...
2023-05-12T15:19:58.754655+0000 mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1876 :
cephadm [INF] Zap device mostha1.legi.grenoble-inp.fr:/dev/sdb
2023-05-12T15:19:58.756639+0000 mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1877 :
cephadm [ERR] Device path '/dev/sdb' not found on host
'mostha1.legi.grenoble-inp.fr'
Traceback (most recent call last):
File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper
return OrchResult(f(*args, **kwargs))
File "/usr/share/ceph/mgr/cephadm/module.py", line 2275, in zap_device
f"Device path '{path}' not found on host '{host}'")
orchestrator._interface.OrchestratorError: Device path '/dev/sdb' not found on
host 'mostha1.legi.grenoble-inp.fr'
....
[ceph: root@mostha1 /]# ls -l /dev/sdb
brw-rw---- 1 root disk 8, 16 May 12 15:16 /dev/sdb
[ceph: root@mostha1 /]# lsblk /dev/sdb
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 1 465.8G 0 disk
`-sdb1 8:17 1 465.8G 0 part
I have crated a full partition on /dev/sdb (for testing) and /dev/sdc has no partition
table (removed).
But all seams fine with these commands.
Patrick
Le 12/05/2023 à 20:19, Beaman, Joshua a écrit :
I don’t quite understand why that zap would not work. But, here’s where I’d start.
1. cephadm check-host
* Run this on each of your hosts to make sure cephadm, podman and all other
prerequisites are installed and recognized
1. ceph orch ls
* This should show at least a mon, mgr, and osd spec deployed
1. ceph orch ls osd –export
* This will show the OSD placement service specifications that orchestrator uses to
identify devices to deploy as OSDs
1. ceph orch host ls
* This will list the hosts that have been added to orchestrator’s inventory, and
what labels are applied which correlate to the service placement labels
1. ceph log last cephadm
* This will show you what orchestrator has been trying to do, and how it may be
failing
Also, it’s never un-helpful to have a look at “ceph -s” and “ceph health detail”,
particularly for any people trying to help you without access to your systems.
Best of luck,
Josh Beaman
From: Patrick Begou
<Patrick.Begou@univ-grenoble-alpes.fr><mailto:Patrick.Begou@univ-grenoble-alpes.fr>
Date: Friday, May 12, 2023 at 10:45 AM
To: ceph-users <ceph-users@ceph.io><mailto:ceph-users@ceph.io>
Subject: [EXTERNAL] [ceph-users] [Pacific] ceph orch device ls do not returns any HDD
Hi everyone
I'm new to CEPH, just a french 4 days training session with Octopus on
VMs that convince me to build my first cluster.
At this time I have 4 old identical nodes for testing with 3 HDDs each,
2 network interfaces and running Alma Linux8 (el8). I try to replay the
training session but it fails, breaking the web interface because of
some problems with podman 4.2 not compatible with Octopus.
So I try to deploy Pacific with cephadm tool on my first node (mostha1)
(to enable testing also an upgrade later).
dnf -y install
https://urldefense.com/v3/__https://download.ceph.com/rpm-16.2.13/el8/noarc…
monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }')
cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \
--initial-dashboard-user admceph \
--allow-fqdn-hostname --cluster-network 10.1.0.0/16
This was sucessfull.
But running "*c**eph orch device ls*" do not show any HDD even if I have
/dev/sda (used by the OS), /dev/sdb and /dev/sdc
The web interface shows a row capacity which is an aggregate of the
sizes of the 3 HDDs for the node.
I've also tried to reset /dev/sdb but cephadm do not see it:
[ceph: root@mostha1 /]# ceph orch device zap
mostha1.legi.grenoble-inp.fr /dev/sdb --force
Error EINVAL: Device path '/dev/sdb' not found on host
'mostha1.legi.grenoble-inp.fr'
On my first attempt with octopus, I was able to list the available HDD
with this command line. Before moving to Pacific, the OS on this node
has been reinstalled from scratch.
Any advices for a CEPH beginner ?
Thanks
Patrick
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
3:32 p.m.
New subject: [EXTERNAL] [Pacific] ceph orch device ls do not returns any HDD
Hi Joshua,
I've tried these commands but it looks like CEPH is unable to see and
configure these HDDs.
[root@mostha1 ~]# cephadm ceph-volume inventory
Inferring fsid 4b7a6504-f0be-11ed-be1a-00266cf8869c
Using recent ceph image
quay.io/ceph/ceph@sha256:e6919776f0ff8331a8e9c4b18d36c5e9eed31e1a80da62ae8454e42d10e95544
Device Path Size Device nodes rotates
available Model name
[root@mostha1 ~]# cephadm shell
[ceph: root@mostha1 /]# ceph orch apply osd --all-available-devices
Scheduled osd.all-available-devices update...
[ceph: root@mostha1 /]# ceph orch device ls[ceph: root@mostha1 /]#
ceph-volume lvm zap /dev/sdb
--> Zapping: /dev/sdb
--> --destroy was not specified, but zapping a whole device will
remove the partition table
Running command: /usr/bin/dd if=/dev/zero of=/dev/sdb bs=1M count=10
conv=fsync
stderr: 10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.10039 s, 104 MB/s
--> Zapping successful for: <Raw Device: /dev/sdb>
I can check that /dev/sdb1 has been erased, so previous command is
successful
[ceph: root@mostha1 ceph]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 232.9G 0 disk
|-sda1 8:1 1 3.9G 0 part /rootfs/boot
|-sda2 8:2 1 78.1G 0 part
| `-osvg-rootvol 253:0 0 48.8G 0 lvm /rootfs
|-sda3 8:3 1 3.9G 0 part [SWAP]
`-sda4 8:4 1 146.9G 0 part
|-secretvg-homevol 253:1 0 9.8G 0 lvm /rootfs/home
|-secretvg-tmpvol 253:2 0 9.8G 0 lvm /rootfs/tmp
`-secretvg-varvol 253:3 0 9.8G 0 lvm /rootfs/var
sdb 8:16 1 465.8G 0 disk
sdc 8:32 1 232.9G 0 disk
But still no visible HDD:
[ceph: root@mostha1 ceph]# ceph orch apply osd --all-available-devices
Scheduled osd.all-available-devices update...
[ceph: root@mostha1 ceph]# ceph orch device ls
[ceph: root@mostha1 ceph]#
May be I have done something bad at install time as in the container
I've unintentionally run:
dnf -y install
https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noar…
(an awful copy/paste launching the command). Can this break The
container ? I do not know what should be available as ceph packages in
the container to remove properly this install (no dnf.log file in the
container)
Patrick
Le 12/05/2023 à 21:38, Beaman, Joshua a écrit :
The most significant point I see there, is you have no
OSD service
spec to tell orchestrator how to deploy OSDs. The easiest fix for
that would be “cephorchapplyosd--all-available-devices”
This will create a simple spec that should work for a test
environment. Most likely it will collocate the block, block.db, and
WAL all on the same device. Not ideal for prod environments, but fine
for practice and testing.
The other command I should have had you try is “cephadm ceph-volume
inventory”. That should show you the devices available for OSD
deployment, and hopefully matches up to what your “lsblk” shows. If
you need to zap HDDs and orchestrator is still not seeing them, you
can try “cephadm ceph-volume lvm zap /dev/sdb”
Thank you,
Josh Beaman
*From: *Patrick Begou <Patrick.Begou(a)univ-grenoble-alpes.fr>
*Date: *Friday, May 12, 2023 at 2:22 PM
*To: *Beaman, Joshua <Joshua_Beaman(a)comcast.com>om>, ceph-users
<ceph-users(a)ceph.io>
*Subject: *Re: [EXTERNAL] [ceph-users] [Pacific] ceph orch device ls
do not returns any HDD
Hi Joshua and thanks for this quick reply.
At this step I have only one node. I was checking what ceph was
returning with different commands on this host before adding new
hosts. Just to compare with my first Octopus install. As this hardware
is for testing only, it remains easy for me to break everything and
reinstall again.
[root@mostha1 ~]# cephadm check-host
podman (/usr/bin/podman) version 4.2.0 is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
[ceph: root@mostha1 /]# ceph -s
cluster:
id: 4b7a6504-f0be-11ed-be1a-00266cf8869c
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 3
services:
mon: 1 daemons, quorum mostha1.legi.grenoble-inp.fr (age 5h)
mgr: mostha1.legi.grenoble-inp.fr.hogwuz(active, since 5h)
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
[ceph: root@mostha1 /]# ceph orch ls
NAME PORTS RUNNING REFRESHED AGE PLACEMENT
alertmanager ?:9093,9094 1/1 6m ago 6h count:1
crash 1/1 6m ago 6h *
grafana ?:3000 1/1 6m ago 6h count:1
mgr 1/2 6m ago 6h count:2
mon 1/5 6m ago 6h count:5
node-exporter ?:9100 1/1 6m ago 6h *
prometheus ?:9095 1/1 6m ago 6h count:1
[ceph: root@mostha1 /]# ceph orch ls osd -export
No services reported
[ceph: root@mostha1 /]# ceph orch host ls
HOST ADDR LABELS STATUS
mostha1.legi.grenoble-inp.fr 194.254.66.34 _admin
1 hosts in cluster
[ceph: root@mostha1 /]# ceph log last cephadm
...
2023-05-12T15:19:58.754655+0000
mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1876 : cephadm
[INF] Zap device mostha1.legi.grenoble-inp.fr:/dev/sdb
2023-05-12T15:19:58.756639+0000
mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1877 : cephadm
[ERR] Device path '/dev/sdb' not found on host
'mostha1.legi.grenoble-inp.fr'
Traceback (most recent call last):
File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125,
in wrapper
return OrchResult(f(*args, **kwargs))
File "/usr/share/ceph/mgr/cephadm/module.py", line 2275, in
zap_device
f"Device path '{path}' not found on host '{host}'")
orchestrator._interface.OrchestratorError: Device path '/dev/sdb'
not found on host 'mostha1.legi.grenoble-inp.fr'
....
[ceph: root@mostha1 /]# ls -l /dev/sdb
brw-rw---- 1 root disk 8, 16 May 12 15:16 /dev/sdb
[ceph: root@mostha1 /]# lsblk /dev/sdb
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 1 465.8G 0 disk
`-sdb1 8:17 1 465.8G 0 part
I have crated a full partition on /dev/sdb (for testing) and /dev/sdc
has no partition table (removed).
But all seams fine with these commands.
Patrick
Le 12/05/2023 à 20:19, Beaman, Joshua a écrit :
I don’t quite understand why that zap would not work. But, here’s
where I’d start.
1. cephadm check-host
1. Run this on each of your hosts to make sure cephadm,
podman and all other prerequisites are installed and
recognized
2. ceph orch ls
1. This should show at least a mon, mgr, and osd spec deployed
3. ceph orch ls osd –export
1. This will show the OSD placement service specifications
that orchestrator uses to identify devices to deploy as OSDs
4. ceph orch host ls
1. This will list the hosts that have been added to
orchestrator’s inventory, and what labels are applied
which correlate to the service placement labels
5. ceph log last cephadm
1. This will show you what orchestrator has been trying to
do, and how it may be failing
Also, it’s never un-helpful to have a look at “ceph -s” and “ceph
health detail”, particularly for any people trying to help you
without access to your systems.
Best of luck,
Josh Beaman
*From: *Patrick Begou <Patrick.Begou(a)univ-grenoble-alpes.fr>
<mailto:Patrick.Begou@univ-grenoble-alpes.fr>
*Date: *Friday, May 12, 2023 at 10:45 AM
*To: *ceph-users <ceph-users(a)ceph.io> <mailto:ceph-users@ceph.io>
*Subject: *[EXTERNAL] [ceph-users] [Pacific] ceph orch device ls
do not returns any HDD
Hi everyone
I'm new to CEPH, just a french 4 days training session with
Octopus on
VMs that convince me to build my first cluster.
At this time I have 4 old identical nodes for testing with 3 HDDs
each,
2 network interfaces and running Alma Linux8 (el8). I try to
replay the
training session but it fails, breaking the web interface because of
some problems with podman 4.2 not compatible with Octopus.
So I try to deploy Pacific with cephadm tool on my first node
(mostha1)
(to enable testing also an upgrade later).
dnf -y install
https://urldefense.com/v3/__https://download.ceph.com/rpm-16.2.13/el8/noarc…
<https://urldefense.com/v3/__https:/download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noarch.rpm__;!!CQl3mcHX2A!H9cwNCJyKXYQ4BbGA3gwHHRitjOS4lBCZT9wlnBZ-8IDue0MvdcPD8Dnv5yQCZw_eA4BNDYaEq1eouKQcQO7HshgdUJ0SJ-EgLfaBGBmCQ$>
monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }')
cephadm bootstrap --mon-ip $monip --initial-dashboard-password
xxxxx \
--initial-dashboard-user admceph \
--allow-fqdn-hostname --cluster-network
10.1.0.0/16
This was sucessfull.
But running "*c**eph orch device ls*" do not show any HDD even if
I have
/dev/sda (used by the OS), /dev/sdb and /dev/sdc
The web interface shows a row capacity which is an aggregate of the
sizes of the 3 HDDs for the node.
I've also tried to reset /dev/sdb but cephadm do not see it:
[ceph: root@mostha1 /]# ceph orch device zap
mostha1.legi.grenoble-inp.fr /dev/sdb --force
Error EINVAL: Device path '/dev/sdb' not found on host
'mostha1.legi.grenoble-inp.fr'
On my first attempt with octopus, I was able to list the available
HDD
with this command line. Before moving to Pacific, the OS on this node
has been reinstalled from scratch.
Any advices for a CEPH beginner ?
Thanks
Patrick
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
16 May
16 May
1:31 a.m.
New subject: [EXTERNAL] [Pacific] ceph orch device ls do not returns any HDD
Patrick,
Sorry for delayed response. This seems to be the limit of assistance I’m capable of
providing. My deployments are all ubuntu and bootstrapped (or upgraded) according to this
starting doc:
https://docs.ceph.com/en/quincy/cephadm/install/#cephadm-deploying-new-clus…
It is very confusing to me that cephadm and ceph-volume are able to zap the device, but
cephadm ceph-volume inventory shows nothing. It’s even more perplexing to me, because on
my systems even the OS disks are listed as not available.
Maybe someone else here as an idea what’s going on.
One last difference that might be a place for you to investigate. I’m using docker, so
perhaps your podman installation is somehow limiting direct access to the disk devices?
Best of luck,
Josh Beaman
From: Patrick Begou <Patrick.Begou(a)univ-grenoble-alpes.fr>
Date: Saturday, May 13, 2023 at 3:33 AM
To: Beaman, Joshua <Joshua_Beaman(a)comcast.com>om>, ceph-users
<ceph-users(a)ceph.io>
Subject: Re: [EXTERNAL] [ceph-users] [Pacific] ceph orch device ls do not returns any HDD
Hi Joshua,
I've tried these commands but it looks like CEPH is unable to see and configure these
HDDs.
[root@mostha1 ~]# cephadm ceph-volume inventory
Inferring fsid 4b7a6504-f0be-11ed-be1a-00266cf8869c
Using recent ceph image
quay.io/ceph/ceph@sha256:e6919776f0ff8331a8e9c4b18d36c5e9eed31e1a80da62ae8454e42d10e95544
Device Path Size Device nodes rotates available Model name
[root@mostha1 ~]# cephadm shell
[ceph: root@mostha1 /]# ceph orch apply osd --all-available-devices
Scheduled osd.all-available-devices update...
[ceph: root@mostha1 /]# ceph orch device ls[ceph: root@mostha1 /]# ceph-volume lvm zap
/dev/sdb
--> Zapping: /dev/sdb
--> --destroy was not specified, but zapping a whole device will remove the partition
table
Running command: /usr/bin/dd if=/dev/zero of=/dev/sdb bs=1M count=10 conv=fsync
stderr: 10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.10039 s, 104 MB/s
--> Zapping successful for: <Raw Device: /dev/sdb>
I can check that /dev/sdb1 has been erased, so previous command is successful
[ceph: root@mostha1 ceph]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 232.9G 0 disk
|-sda1 8:1 1 3.9G 0 part /rootfs/boot
|-sda2 8:2 1 78.1G 0 part
| `-osvg-rootvol 253:0 0 48.8G 0 lvm /rootfs
|-sda3 8:3 1 3.9G 0 part [SWAP]
`-sda4 8:4 1 146.9G 0 part
|-secretvg-homevol 253:1 0 9.8G 0 lvm /rootfs/home
|-secretvg-tmpvol 253:2 0 9.8G 0 lvm /rootfs/tmp
`-secretvg-varvol 253:3 0 9.8G 0 lvm /rootfs/var
sdb 8:16 1 465.8G 0 disk
sdc 8:32 1 232.9G 0 disk
But still no visible HDD:
[ceph: root@mostha1 ceph]# ceph orch apply osd --all-available-devices
Scheduled osd.all-available-devices update...
[ceph: root@mostha1 ceph]# ceph orch device ls
[ceph: root@mostha1 ceph]#
May be I have done something bad at install time as in the container I've
unintentionally run:
dnf -y install
https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noar…
(an awful copy/paste launching the command). Can this break The container ? I do not know
what should be available as ceph packages in the container to remove properly this install
(no dnf.log file in the container)
Patrick
Le 12/05/2023 à 21:38, Beaman, Joshua a écrit :
The most significant point I see there, is you have no OSD service spec to tell
orchestrator how to deploy OSDs. The easiest fix for that would be “ceph orch apply osd
--all-available-devices”
This will create a simple spec that should work for a test environment. Most likely it
will collocate the block, block.db, and WAL all on the same device. Not ideal for prod
environments, but fine for practice and testing.
The other command I should have had you try is “cephadm ceph-volume inventory”. That
should show you the devices available for OSD deployment, and hopefully matches up to what
your “lsblk” shows. If you need to zap HDDs and orchestrator is still not seeing them,
you can try “cephadm ceph-volume lvm zap /dev/sdb”
Thank you,
Josh Beaman
From: Patrick Begou
<Patrick.Begou@univ-grenoble-alpes.fr><mailto:Patrick.Begou@univ-grenoble-alpes.fr>
Date: Friday, May 12, 2023 at 2:22 PM
To: Beaman, Joshua
<Joshua_Beaman@comcast.com><mailto:Joshua_Beaman@comcast.com>, ceph-users
<ceph-users@ceph.io><mailto:ceph-users@ceph.io>
Subject: Re: [EXTERNAL] [ceph-users] [Pacific] ceph orch device ls do not returns any HDD
Hi Joshua and thanks for this quick reply.
At this step I have only one node. I was checking what ceph was returning with different
commands on this host before adding new hosts. Just to compare with my first Octopus
install. As this hardware is for testing only, it remains easy for me to break everything
and reinstall again.
[root@mostha1 ~]# cephadm check-host
podman (/usr/bin/podman) version 4.2.0 is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
[ceph: root@mostha1 /]# ceph -s
cluster:
id: 4b7a6504-f0be-11ed-be1a-00266cf8869c
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 3
services:
mon: 1 daemons, quorum mostha1.legi.grenoble-inp.fr (age 5h)
mgr: mostha1.legi.grenoble-inp.fr.hogwuz(active, since 5h)
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
[ceph: root@mostha1 /]# ceph orch ls
NAME PORTS RUNNING REFRESHED AGE PLACEMENT
alertmanager ?:9093,9094 1/1 6m ago 6h count:1
crash 1/1 6m ago 6h *
grafana ?:3000 1/1 6m ago 6h count:1
mgr 1/2 6m ago 6h count:2
mon 1/5 6m ago 6h count:5
node-exporter ?:9100 1/1 6m ago 6h *
prometheus ?:9095 1/1 6m ago 6h count:1
[ceph: root@mostha1 /]# ceph orch ls osd -export
No services reported
[ceph: root@mostha1 /]# ceph orch host ls
HOST ADDR LABELS STATUS
mostha1.legi.grenoble-inp.fr 194.254.66.34 _admin
1 hosts in cluster
[ceph: root@mostha1 /]# ceph log last cephadm
...
2023-05-12T15:19:58.754655+0000 mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1876 :
cephadm [INF] Zap device mostha1.legi.grenoble-inp.fr:/dev/sdb
2023-05-12T15:19:58.756639+0000 mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1877 :
cephadm [ERR] Device path '/dev/sdb' not found on host
'mostha1.legi.grenoble-inp.fr'
Traceback (most recent call last):
File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper
return OrchResult(f(*args, **kwargs))
File "/usr/share/ceph/mgr/cephadm/module.py", line 2275, in zap_device
f"Device path '{path}' not found on host '{host}'")
orchestrator._interface.OrchestratorError: Device path '/dev/sdb' not found on
host 'mostha1.legi.grenoble-inp.fr'
....
[ceph: root@mostha1 /]# ls -l /dev/sdb
brw-rw---- 1 root disk 8, 16 May 12 15:16 /dev/sdb
[ceph: root@mostha1 /]# lsblk /dev/sdb
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 1 465.8G 0 disk
`-sdb1 8:17 1 465.8G 0 part
I have crated a full partition on /dev/sdb (for testing) and /dev/sdc has no partition
table (removed).
But all seams fine with these commands.
Patrick
Le 12/05/2023 à 20:19, Beaman, Joshua a écrit :
I don’t quite understand why that zap would not work. But, here’s where I’d start.
1. cephadm check-host
* Run this on each of your hosts to make sure cephadm, podman and all other
prerequisites are installed and recognized
1. ceph orch ls
* This should show at least a mon, mgr, and osd spec deployed
1. ceph orch ls osd –export
* This will show the OSD placement service specifications that orchestrator uses to
identify devices to deploy as OSDs
1. ceph orch host ls
* This will list the hosts that have been added to orchestrator’s inventory, and
what labels are applied which correlate to the service placement labels
1. ceph log last cephadm
* This will show you what orchestrator has been trying to do, and how it may be
failing
Also, it’s never un-helpful to have a look at “ceph -s” and “ceph health detail”,
particularly for any people trying to help you without access to your systems.
Best of luck,
Josh Beaman
From: Patrick Begou
<Patrick.Begou@univ-grenoble-alpes.fr><mailto:Patrick.Begou@univ-grenoble-alpes.fr>
Date: Friday, May 12, 2023 at 10:45 AM
To: ceph-users <ceph-users@ceph.io><mailto:ceph-users@ceph.io>
Subject: [EXTERNAL] [ceph-users] [Pacific] ceph orch device ls do not returns any HDD
Hi everyone
I'm new to CEPH, just a french 4 days training session with Octopus on
VMs that convince me to build my first cluster.
At this time I have 4 old identical nodes for testing with 3 HDDs each,
2 network interfaces and running Alma Linux8 (el8). I try to replay the
training session but it fails, breaking the web interface because of
some problems with podman 4.2 not compatible with Octopus.
So I try to deploy Pacific with cephadm tool on my first node (mostha1)
(to enable testing also an upgrade later).
dnf -y install
https://urldefense.com/v3/__https://download.ceph.com/rpm-16.2.13/el8/noarc…
monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }')
cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \
--initial-dashboard-user admceph \
--allow-fqdn-hostname --cluster-network 10.1.0.0/16
This was sucessfull.
But running "*c**eph orch device ls*" do not show any HDD even if I have
/dev/sda (used by the OS), /dev/sdb and /dev/sdc
The web interface shows a row capacity which is an aggregate of the
sizes of the 3 HDDs for the node.
I've also tried to reset /dev/sdb but cephadm do not see it:
[ceph: root@mostha1 /]# ceph orch device zap
mostha1.legi.grenoble-inp.fr /dev/sdb --force
Error EINVAL: Device path '/dev/sdb' not found on host
'mostha1.legi.grenoble-inp.fr'
On my first attempt with octopus, I was able to list the available HDD
with this command line. Before moving to Pacific, the OS on this node
has been reinstalled from scratch.
Any advices for a CEPH beginner ?
Thanks
Patrick
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
2:20 a.m.
New subject: [EXTERNAL] [Pacific] ceph orch device ls do not returns any HDD
As you've already seem to have figured out, "ceph orch device ls" is
populated with the results from "ceph-volume inventory". My best guess to
try and debug this would be to manually run "cephadm ceph-volume --
inventory" (the same as "cephadm ceph-volume inventory", I just like to
separate the ceph-volume command from cephadm itself with the " -- ") and
then check /var/log/ceph/<fsid>/ceph-volume.log from when you ran the
command onward to try and see why it isn't seeing your devices. For example
I can see a line like
[2023-05-15 19:11:58,048][ceph_volume.main][INFO ] Running command:
ceph-volume inventory
in there. Then if I look onward from there I can see it ran things like
lsblk -P -o
NAME,KNAME,PKNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL
as part of getting my device list. So if I was having issues I would try
running that directly and see what I got. Will note that ceph-volume on
certain more recent versions (not sure about octopus) runs commands through
nsenter, so you'd have to look past that part in the log lines to the
underlying command being used, typically something with lsblk, blkid,
udevadm, lvs, or pvs.
Also, if you want to see if it's an issue with a certain version of
ceph-volume, you can use different versions by passing the image flag to
cephadm. E.g.
cephadm --image quay.io/ceph/ceph:v17.2.6 ceph-volume -- inventory
would use the 17.2.6 version of ceph-volume for the inventory. It works by
running ceph-volume through the container, so you don't have to have to
worry about installing different packages to try them and it should pull
the container image on its own if it isn't on the machine already (but note
that means the command will take longer as it pulls the image the first
time).
On Sat, May 13, 2023 at 4:34 AM Patrick Begou <
Patrick.Begou(a)univ-grenoble-alpes.fr> wrote:
Hi Joshua,
I've tried these commands but it looks like CEPH is unable to see and
configure these HDDs.
[root@mostha1 ~]# cephadm ceph-volume inventory
Inferring fsid 4b7a6504-f0be-11ed-be1a-00266cf8869c
Using recent ceph image
quay.io/ceph/ceph@sha256:e6919776f0ff8331a8e9c4b18d36c5e9eed31e1a80da62ae8454e42d10e95544
Device Path Size Device nodes rotates
available Model name
[root@mostha1 ~]# cephadm shell
[ceph: root@mostha1 /]# ceph orch apply osd --all-available-devices
Scheduled osd.all-available-devices update...
[ceph: root@mostha1 /]# ceph orch device ls[ceph: root@mostha1 /]#
ceph-volume lvm zap /dev/sdb
--> Zapping: /dev/sdb
--> --destroy was not specified, but zapping a whole device will
remove the partition table
Running command: /usr/bin/dd if=/dev/zero of=/dev/sdb bs=1M count=10
conv=fsync
stderr: 10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.10039 s, 104 MB/s
--> Zapping successful for: <Raw Device: /dev/sdb>
I can check that /dev/sdb1 has been erased, so previous command is
successful
[ceph: root@mostha1 ceph]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 232.9G 0 disk
|-sda1 8:1 1 3.9G 0 part /rootfs/boot
|-sda2 8:2 1 78.1G 0 part
| `-osvg-rootvol 253:0 0 48.8G 0 lvm /rootfs
|-sda3 8:3 1 3.9G 0 part [SWAP]
`-sda4 8:4 1 146.9G 0 part
|-secretvg-homevol 253:1 0 9.8G 0 lvm /rootfs/home
|-secretvg-tmpvol 253:2 0 9.8G 0 lvm /rootfs/tmp
`-secretvg-varvol 253:3 0 9.8G 0 lvm /rootfs/var
sdb 8:16 1 465.8G 0 disk
sdc 8:32 1 232.9G 0 disk
But still no visible HDD:
[ceph: root@mostha1 ceph]# ceph orch apply osd --all-available-devices
Scheduled osd.all-available-devices update...
[ceph: root@mostha1 ceph]# ceph orch device ls
[ceph: root@mostha1 ceph]#
May be I have done something bad at install time as in the container
I've unintentionally run:
dnf -y install
https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noar…
(an awful copy/paste launching the command). Can this break The
container ? I do not know what should be available as ceph packages in
the container to remove properly this install (no dnf.log file in the
container)
Patrick
Le 12/05/2023 à 21:38, Beaman, Joshua a écrit :
The most significant point I see there, is you
have no OSD service
spec to tell orchestrator how to deploy OSDs. The easiest fix for
that would be “cephorchapplyosd--all-available-devices”
This will create a simple spec that should work for a test
environment. Most likely it will collocate the block, block.db, and
WAL all on the same device. Not ideal for prod environments, but fine
for practice and testing.
The other command I should have had you try is “cephadm ceph-volume
inventory”. That should show you the devices available for OSD
deployment, and hopefully matches up to what your “lsblk” shows. If
you need to zap HDDs and orchestrator is still not seeing them, you
can try “cephadm ceph-volume lvm zap /dev/sdb”
Thank you,
Josh Beaman
*From: *Patrick Begou <Patrick.Begou(a)univ-grenoble-alpes.fr>
*Date: *Friday, May 12, 2023 at 2:22 PM
*To: *Beaman, Joshua <Joshua_Beaman(a)comcast.com>om>, ceph-users
<ceph-users(a)ceph.io>
*Subject: *Re: [EXTERNAL] [ceph-users] [Pacific] ceph orch device ls
do not returns any HDD
Hi Joshua and thanks for this quick reply.
At this step I have only one node. I was checking what ceph was
returning with different commands on this host before adding new
hosts. Just to compare with my first Octopus install. As this hardware
is for testing only, it remains easy for me to break everything and
reinstall again.
[root@mostha1 ~]# cephadm check-host
podman (/usr/bin/podman) version 4.2.0 is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
[ceph: root@mostha1 /]# ceph -s
cluster:
id: 4b7a6504-f0be-11ed-be1a-00266cf8869c
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 3
services:
mon: 1 daemons, quorum mostha1.legi.grenoble-inp.fr (age 5h)
mgr: mostha1.legi.grenoble-inp.fr.hogwuz(active, since 5h)
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
[ceph: root@mostha1 /]# ceph orch ls
NAME PORTS RUNNING REFRESHED AGE PLACEMENT
alertmanager ?:9093,9094 1/1 6m ago 6h count:1
crash 1/1 6m ago 6h *
grafana ?:3000 1/1 6m ago 6h count:1
mgr 1/2 6m ago 6h count:2
mon 1/5 6m ago 6h count:5
node-exporter ?:9100 1/1 6m ago 6h *
prometheus ?:9095 1/1 6m ago 6h count:1
[ceph: root@mostha1 /]# ceph orch ls osd -export
No services reported
[ceph: root@mostha1 /]# ceph orch host ls
HOST ADDR LABELS STATUS
mostha1.legi.grenoble-inp.fr 194.254.66.34 _admin
1 hosts in cluster
[ceph: root@mostha1 /]# ceph log last cephadm
...
2023-05-12T15:19:58.754655+0000
mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1876 : cephadm
[INF] Zap device mostha1.legi.grenoble-inp.fr:/dev/sdb
2023-05-12T15:19:58.756639+0000
mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1877 : cephadm
[ERR] Device path '/dev/sdb' not found on host
'mostha1.legi.grenoble-inp.fr'
Traceback (most recent call last):
File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125,
in wrapper
return OrchResult(f(*args, **kwargs))
File "/usr/share/ceph/mgr/cephadm/module.py", line 2275, in
zap_device
f"Device path '{path}' not found on host '{host}'")
orchestrator._interface.OrchestratorError: Device path '/dev/sdb'
not found on host 'mostha1.legi.grenoble-inp.fr'
....
[ceph: root@mostha1 /]# ls -l /dev/sdb
brw-rw---- 1 root disk 8, 16 May 12 15:16 /dev/sdb
[ceph: root@mostha1 /]# lsblk /dev/sdb
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 1 465.8G 0 disk
`-sdb1 8:17 1 465.8G 0 part
I have crated a full partition on /dev/sdb (for testing) and /dev/sdc
has no partition table (removed).
But all seams fine with these commands.
Patrick
Le 12/05/2023 à 20:19, Beaman, Joshua a écrit :
I don’t quite understand why that zap would not work. But, here’s
where I’d start.
1. cephadm check-host
1. Run this on each of your hosts to make sure cephadm,
podman and all other prerequisites are installed and
recognized
2. ceph orch ls
1. This should show at least a mon, mgr, and osd spec deployed
3. ceph orch ls osd –export
1. This will show the OSD placement service specifications
that orchestrator uses to identify devices to deploy as OSDs
4. ceph orch host ls
1. This will list the hosts that have been added to
orchestrator’s inventory, and what labels are applied
which correlate to the service placement labels
5. ceph log last cephadm
1. This will show you what orchestrator has been trying to
do, and how it may be failing
Also, it’s never un-helpful to have a look at “ceph -s” and “ceph
health detail”, particularly for any people trying to help you
without access to your systems.
Best of luck,
Josh Beaman
*From: *Patrick Begou <Patrick.Begou(a)univ-grenoble-alpes.fr>
<mailto:Patrick.Begou@univ-grenoble-alpes.fr>
*Date: *Friday, May 12, 2023 at 10:45 AM
*To: *ceph-users <ceph-users(a)ceph.io> <mailto:ceph-users@ceph.io>
*Subject: *[EXTERNAL] [ceph-users] [Pacific] ceph orch device ls
do not returns any HDD
Hi everyone
I'm new to CEPH, just a french 4 days training session with
Octopus on
VMs that convince me to build my first cluster.
At this time I have 4 old identical nodes for testing with 3 HDDs
each,
2 network interfaces and running Alma Linux8 (el8). I try to
replay the
training session but it fails, breaking the web interface because of
some problems with podman 4.2 not compatible with Octopus.
So I try to deploy Pacific with cephadm tool on my first node
(mostha1)
(to enable testing also an upgrade later).
dnf -y install
https://urldefense.com/v3/__https://download.ceph.com/rpm-16.2.13/el8/noarc…
<
https://urldefense.com/v3/__https:/download.ceph.com/rpm-16.2.13/el8/noarch…
monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }')
cephadm bootstrap --mon-ip $monip --initial-dashboard-password
xxxxx \
--initial-dashboard-user admceph \
--allow-fqdn-hostname --cluster-network
10.1.0.0/16
This was sucessfull.
But running "*c**eph orch device ls*" do not show any HDD even if
I have
/dev/sda (used by the OS), /dev/sdb and /dev/sdc
The web interface shows a row capacity which is an aggregate of the
sizes of the 3 HDDs for the node.
I've also tried to reset /dev/sdb but cephadm do not see it:
[ceph: root@mostha1 /]# ceph orch device zap
mostha1.legi.grenoble-inp.fr /dev/sdb --force
Error EINVAL: Device path '/dev/sdb' not found on host
'mostha1.legi.grenoble-inp.fr'
On my first attempt with octopus, I was able to list the available
HDD
with this command line. Before moving to Pacific, the OS on this node
has been reinstalled from scratch.
Any advices for a CEPH beginner ?
Thanks
Patrick
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
26 May
26 May
9:59 p.m.
New subject: [EXTERNAL] [Pacific] ceph orch device ls do not returns any HDD
Hi,
I'm back working on this problem.
First of all, I saw that I had a hardware memory error so I had to solve
this first. It's done.
I've tested some different Ceph deployments, each time starting with a
full OS re-install (it requires some time for each test).
Using Octopus, the devices are found:
dnf -y install \
https://download.ceph.com/rpm-15.2.12/el8/noarch/cephadm-15.2.12-0.el8.noar…
monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }'))
cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \
--allow-fqdn-hostname
[ceph: root@mostha1 /]# *ceph orch device ls*
Hostname Path Type Serial Size Health
Ident Fault Available
mostha1.legi.grenoble-inp.fr /dev/sda hdd S2B5J90ZA02494 250G
Unknown N/A N/A Yes
mostha1.legi.grenoble-inp.fr /dev/sdc hdd WD-WMAYP0982329 500G
Unknown N/A N/A Yes
But with Pacific or Quincy the command returns nothing.
With Pacific:
dnf -y install \
https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noar…
monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }')
cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \
--allow-fqdn-hostname
"ceph orch device ls" doesn't return anything but "cephadm shell lsmcli
ldl" list all the devices.
[ceph: root@mostha1 /]# *ceph orch device ls --wide*
[ceph: root@mostha1 /]# *lsblk*
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 232.9G 0 disk
|-sda1 8:1 1 3.9G 0 part /rootfs/boot
|-sda2 8:2 1 78.1G 0 part
| `-osvg-rootvol 253:0 0 48.8G 0 lvm /rootfs
|-sda3 8:3 1 3.9G 0 part [SWAP]
`-sda4 8:4 1 146.9G 0 part
|-secretvg-homevol 253:1 0 9.8G 0 lvm /rootfs/home
|-secretvg-tmpvol 253:2 0 9.8G 0 lvm /rootfs/tmp
`-secretvg-varvol 253:3 0 9.8G 0 lvm /rootfs/var
sdb 8:16 1 232.9G 0 disk
sdc 8:32 1 465.8G 0 disk
[ceph: root@mostha1 /]# exit
[root@mostha1 ~]# *cephadm ceph-volume inventory*
Inferring fsid 2e3e85a8-fbcf-11ed-84e5-00266cf8869c
Using ceph image with id '0dc91bca92c2' and tag 'v17' created on
2023-05-25 16:26:31 +0000 UTC
quay.io/ceph/ceph@sha256:b8df01a568f4dec7bac6d5040f9391dcca14e00ec7f4de8a3dcf3f2a6502d3a9
Device Path Size Device nodes rotates
available Model name
[root@mostha1 ~]# *cephadm shell lsmcli ldl*
Inferring fsid 4d54823c-fb05-11ed-aecf-00266cf8869c
Inferring config
/var/lib/ceph/4d54823c-fb05-11ed-aecf-00266cf8869c/mon.mostha1/config
Using ceph image with id 'c9a1062f7289' and tag 'v17' created on
2023-04-25 16:04:33 +0000 UTC
quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e
Path | SCSI VPD 0x83 | Link Type | Serial Number | Health
Status
-------------------------------------------------------------------------
*/dev/sda | 50024e92039e4f1c | PATA/SATA | S2B5J90ZA10142 | Good**
**/dev/sdc | 50014ee0ad5953c9 | PATA/SATA | WD-WMAYP0982329 | Good**
**/dev/sdb | 50024e920387fa2c | PATA/SATA | S2B5J90ZA02494 | Good**
*
Could it be a bug in ceph-volume ?
Adam suggest looking to the underlying commands (lsblk, blkid, udevadm,
lvs, or pvs) but I'm not very comfortable with blkid and udevadm. Is
there a "debug flag" to set ceph more verbose ?
Thanks
Patrick
Le 15/05/2023 à 21:20, Adam King a écrit :
As you've already seem to have figured out,
"ceph orch device ls" is
populated with the results from "ceph-volume inventory". My best guess
to try and debug this would be to manually run "cephadm ceph-volume --
inventory" (the same as "cephadm ceph-volume inventory", I just like
to separate the ceph-volume command from cephadm itself with the " --
") and then check /var/log/ceph/<fsid>/ceph-volume.log from when you
ran the command onward to try and see why it isn't seeing your
devices. For example I can see a line like
[2023-05-15 19:11:58,048][ceph_volume.main][INFO ] Running command:
ceph-volume inventory
in there. Then if I look onward from there I can see it ran things like
lsblk -P -o
NAME,KNAME,PKNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL
as part of getting my device list. So if I was having issues I would
try running that directly and see what I got. Will note that
ceph-volume on certain more recent versions (not sure about octopus)
runs commands through nsenter, so you'd have to look past that part in
the log lines to the underlying command being used, typically
something with lsblk, blkid, udevadm, lvs, or pvs.
Also, if you want to see if it's an issue with a certain version of
ceph-volume, you can use different versions by passing the image flag
to cephadm. E.g.
cephadm --image quay.io/ceph/ceph:v17.2.6
<http://quay.io/ceph/ceph:v17.2.6> ceph-volume -- inventory
would use the 17.2.6 version of ceph-volume for the inventory. It
works by running ceph-volume through the container, so you don't have
to have to worry about installing different packages to try them and
it should pull the container image on its own if it isn't on the
machine already (but note that means the command will take longer as
it pulls the image the first time).
On Sat, May 13, 2023 at 4:34 AM Patrick Begou
<Patrick.Begou(a)univ-grenoble-alpes.fr> wrote:
Hi Joshua,
I've tried these commands but it looks like CEPH is unable to see and
configure these HDDs.
[root@mostha1 ~]# cephadm ceph-volume inventory
Inferring fsid 4b7a6504-f0be-11ed-be1a-00266cf8869c
Using recent ceph image
quay.io/ceph/ceph@sha256:e6919776f0ff8331a8e9c4b18d36c5e9eed31e1a80da62ae8454e42d10e95544
<http://quay.io/ceph/ceph@sha256:e6919776f0ff8331a8e9c4b18d36c5e9eed31e1a80da62ae8454e42d10e95544>
Device Path Size Device nodes rotates
available Model name
[root@mostha1 ~]# cephadm shell
[ceph: root@mostha1 /]# ceph orch apply osd --all-available-devices
Scheduled osd.all-available-devices update...
[ceph: root@mostha1 /]# ceph orch device ls[ceph: root@mostha1 /]#
ceph-volume lvm zap /dev/sdb
--> Zapping: /dev/sdb
--> --destroy was not specified, but zapping a whole device will
remove the partition table
Running command: /usr/bin/dd if=/dev/zero of=/dev/sdb bs=1M
count=10
conv=fsync
stderr: 10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.10039 s, 104 MB/s
--> Zapping successful for: <Raw Device: /dev/sdb>
I can check that /dev/sdb1 has been erased, so previous command is
successful
[ceph: root@mostha1 ceph]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 232.9G 0 disk
|-sda1 8:1 1 3.9G 0 part /rootfs/boot
|-sda2 8:2 1 78.1G 0 part
| `-osvg-rootvol 253:0 0 48.8G 0 lvm /rootfs
|-sda3 8:3 1 3.9G 0 part [SWAP]
`-sda4 8:4 1 146.9G 0 part
|-secretvg-homevol 253:1 0 9.8G 0 lvm /rootfs/home
|-secretvg-tmpvol 253:2 0 9.8G 0 lvm /rootfs/tmp
`-secretvg-varvol 253:3 0 9.8G 0 lvm /rootfs/var
sdb 8:16 1 465.8G 0 disk
sdc 8:32 1 232.9G 0 disk
But still no visible HDD:
[ceph: root@mostha1 ceph]# ceph orch apply osd --all-available-devices
Scheduled osd.all-available-devices update...
[ceph: root@mostha1 ceph]# ceph orch device ls
[ceph: root@mostha1 ceph]#
May be I have done something bad at install time as in the container
I've unintentionally run:
dnf -y install
https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noar…
(an awful copy/paste launching the command). Can this break The
container ? I do not know what should be available as ceph
packages in
the container to remove properly this install (no dnf.log file in the
container)
Patrick
Le 12/05/2023 à 21:38, Beaman, Joshua a écrit :
<https://urldefense.com/v3/__https:/download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noarch.rpm__;!!CQl3mcHX2A!H9cwNCJyKXYQ4BbGA3gwHHRitjOS4lBCZT9wlnBZ-8IDue0MvdcPD8Dnv5yQCZw_eA4BNDYaEq1eouKQcQO7HshgdUJ0SJ-EgLfaBGBmCQ$>
The most significant point I see there, is you
have no OSD service
spec to tell orchestrator how to deploy OSDs. The easiest fix for
that would be “cephorchapplyosd--all-available-devices”
This will create a simple spec that should work for a test
environment. Most likely it will collocate the block, block.db,
and
WAL all on the same device. Not ideal for prod
environments,
but fine
for practice and testing.
The other command I should have had you try is “cephadm ceph-volume
inventory”. That should show you the devices available for OSD
deployment, and hopefully matches up to what your “lsblk”
shows. If
you need to zap HDDs and orchestrator is still
not seeing them, you
can try “cephadm ceph-volume lvm zap /dev/sdb”
Thank you,
Josh Beaman
*From: *Patrick Begou <Patrick.Begou(a)univ-grenoble-alpes.fr>
*Date: *Friday, May 12, 2023 at 2:22 PM
*To: *Beaman, Joshua <Joshua_Beaman(a)comcast.com>om>, ceph-users
<ceph-users(a)ceph.io>
*Subject: *Re: [EXTERNAL] [ceph-users] [Pacific] ceph orch
device ls
do not returns any HDD
Hi Joshua and thanks for this quick reply.
At this step I have only one node. I was checking what ceph was
returning with different commands on this host before adding new
hosts. Just to compare with my first Octopus install. As this
hardware
is for testing only, it remains easy for me to
break everything and
reinstall again.
[root@mostha1 ~]# cephadm check-host
podman (/usr/bin/podman) version 4.2.0 is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
[ceph: root@mostha1 /]# ceph -s
cluster:
id: 4b7a6504-f0be-11ed-be1a-00266cf8869c
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 3
services:
mon: 1 daemons, quorum mostha1.legi.grenoble-inp.fr
<http://mostha1.legi.grenoble-inp.fr> (age 5h)
mgr: mostha1.legi.grenoble-inp.fr
<http://mostha1.legi.grenoble-inp.fr>.hogwuz(active, since 5h)
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
[ceph: root@mostha1 /]# ceph orch ls
NAME PORTS RUNNING REFRESHED AGE PLACEMENT
alertmanager ?:9093,9094 1/1 6m ago 6h count:1
crash 1/1 6m ago 6h *
grafana ?:3000 1/1 6m ago 6h count:1
mgr 1/2 6m ago 6h count:2
mon 1/5 6m ago 6h count:5
node-exporter ?:9100 1/1 6m ago 6h *
prometheus ?:9095 1/1 6m ago 6h count:1
[ceph: root@mostha1 /]# ceph orch ls osd -export
No services reported
[ceph: root@mostha1 /]# ceph orch host ls
HOST ADDR LABELS STATUS
mostha1.legi.grenoble-inp.fr
<http://mostha1.legi.grenoble-inp.fr>
194.254.66.34 _admin
1 hosts in cluster
[ceph: root@mostha1 /]# ceph log last cephadm
...
2023-05-12T15:19:58.754655+0000
mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1876 :
cephadm
[INF] Zap device
mostha1.legi.grenoble-inp.fr:/dev/sdb
2023-05-12T15:19:58.756639+0000
mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1877 :
cephadm
[ERR] Device path '/dev/sdb' not
found on host
'mostha1.legi.grenoble-inp.fr
<http://mostha1.legi.grenoble-inp.fr>'
Traceback (most recent call last):
File "/usr/share/ceph/mgr/orchestrator/_interface.py",
line
125,
in wrapper
return OrchResult(f(*args, **kwargs))
File "/usr/share/ceph/mgr/cephadm/module.py", line 2275, in
zap_device
f"Device path '{path}' not found on host '{host}'")
orchestrator._interface.OrchestratorError: Device path
'/dev/sdb'
not found on host
'mostha1.legi.grenoble-inp.fr
<http://mostha1.legi.grenoble-inp.fr>'
....
[ceph: root@mostha1 /]# ls -l /dev/sdb
brw-rw---- 1 root disk 8, 16 May 12 15:16 /dev/sdb
[ceph: root@mostha1 /]# lsblk /dev/sdb
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 1 465.8G 0 disk
`-sdb1 8:17 1 465.8G 0 part
I have crated a full partition on /dev/sdb (for testing) and
/dev/sdc
has no partition table (removed).
But all seams fine with these commands.
Patrick
Le 12/05/2023 à 20:19, Beaman, Joshua a écrit :
I don’t quite understand why that zap would not work. But,
here’s
where I’d start.
1. cephadm check-host
1. Run this on each of your hosts to make sure cephadm,
podman and all other prerequisites are installed and
recognized
2. ceph orch ls
1. This should show at least a mon, mgr, and osd spec
deployed
3. ceph orch ls osd –export
1. This will show the OSD placement service specifications
that orchestrator uses to identify devices to deploy
as OSDs
4. ceph orch host ls
1. This will list the hosts that have been added to
orchestrator’s inventory, and what labels are applied
which correlate to the service placement labels
5. ceph log last cephadm
1. This will show you what orchestrator has been trying to
do, and how it may be failing
Also, it’s never un-helpful to have a look at “ceph -s” and
“ceph
health detail”, particularly for any people
trying to help you
without access to your systems.
Best of luck,
Josh Beaman
*From: *Patrick Begou <Patrick.Begou(a)univ-grenoble-alpes.fr>
<mailto:Patrick.Begou@univ-grenoble-alpes.fr>
*Date: *Friday, May 12, 2023 at 10:45 AM
*To: *ceph-users <ceph-users(a)ceph.io>
<mailto:ceph-users@ceph.io>
*Subject: *[EXTERNAL] [ceph-users] [Pacific]
ceph orch device ls
do not returns any HDD
Hi everyone
I'm new to CEPH, just a french 4 days training session with
Octopus on
VMs that convince me to build my first cluster.
At this time I have 4 old identical nodes for testing with 3
HDDs
each,
2 network interfaces and running Alma Linux8 (el8). I try to
replay the
training session but it fails, breaking the web interface
because of
some problems with podman 4.2 not compatible
with Octopus.
So I try to deploy Pacific with cephadm tool on my first node
(mostha1)
(to enable testing also an upgrade later).
dnf -y install
https://urldefense.com/v3/__https://download.ceph.com/rpm-16.2.13/el8/noarc…
monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print
$1
}')
cephadm bootstrap --mon-ip $monip
--initial-dashboard-password
xxxxx \
--initial-dashboard-user admceph \
--allow-fqdn-hostname --cluster-network
10.1.0.0/16 <http://10.1.0.0/16>
This was sucessfull.
But running "*c**eph orch device ls*" do not show any HDD
even
if
I have
/dev/sda (used by the OS), /dev/sdb and /dev/sdc
The web interface shows a row capacity which is an aggregate
of the
sizes of the 3 HDDs for the node.
I've also tried to reset /dev/sdb but cephadm do not see it:
[ceph: root@mostha1 /]# ceph orch device zap
mostha1.legi.grenoble-inp.fr
<http://mostha1.legi.grenoble-inp.fr>
/dev/sdb --force
Error EINVAL: Device path
'/dev/sdb' not found on host
'mostha1.legi.grenoble-inp.fr
<http://mostha1.legi.grenoble-inp.fr>'
On my first attempt with octopus, I was able to list the
available
HDD
with this command line. Before moving to Pacific, the OS on
this node
has been reinstalled from scratch.
Any advices for a CEPH beginner ?
Thanks
Patrick
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:19 p.m.
New subject: [EXTERNAL] [Pacific] ceph orch device ls do not returns any HDD
Hi Patrick,
It is weird, we have a couple of clusters with cephadm and running pacify
or quincy and ceph orch device works well. Have you looked at the cephadm
logs (ceph log last cephadm)?
Except if you are using a very specific hardware, I suspect Ceph is
suffering of a problem outside it...
Cheers,
Michel
Sent from my mobile
Le 26 mai 2023 17:02:50 Patrick Begou
<Patrick.Begou(a)univ-grenoble-alpes.fr> a écrit :
Hi,
I'm back working on this problem.
First of all, I saw that I had a hardware memory error so I had to solve
this first. It's done.
I've tested some different Ceph deployments, each time starting with a
full OS re-install (it requires some time for each test).
Using Octopus, the devices are found:
dnf -y install \
https://download.ceph.com/rpm-15.2.12/el8/noarch/cephadm-15.2.12-0.el8.noar…
monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }'))
cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \
--allow-fqdn-hostname
[ceph: root@mostha1 /]# *ceph orch device ls*
Hostname Path Type Serial Size Health
Ident Fault Available
mostha1.legi.grenoble-inp.fr /dev/sda hdd S2B5J90ZA02494 250G
Unknown N/A N/A Yes
mostha1.legi.grenoble-inp.fr /dev/sdc hdd WD-WMAYP0982329 500G
Unknown N/A N/A Yes
But with Pacific or Quincy the command returns nothing.
With Pacific:
dnf -y install \
https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noar…
monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }')
cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \
--allow-fqdn-hostname
"ceph orch device ls" doesn't return anything but "cephadm shell
lsmcli
ldl" list all the devices.
[ceph: root@mostha1 /]# *ceph orch device ls --wide*
[ceph: root@mostha1 /]# *lsblk*
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 232.9G 0 disk
|-sda1 8:1 1 3.9G 0 part /rootfs/boot
|-sda2 8:2 1 78.1G 0 part
| `-osvg-rootvol 253:0 0 48.8G 0 lvm /rootfs
|-sda3 8:3 1 3.9G 0 part [SWAP]
`-sda4 8:4 1 146.9G 0 part
|-secretvg-homevol 253:1 0 9.8G 0 lvm /rootfs/home
|-secretvg-tmpvol 253:2 0 9.8G 0 lvm /rootfs/tmp
`-secretvg-varvol 253:3 0 9.8G 0 lvm /rootfs/var
sdb 8:16 1 232.9G 0 disk
sdc 8:32 1 465.8G 0 disk
[ceph: root@mostha1 /]# exit
[root@mostha1 ~]# *cephadm ceph-volume inventory*
Inferring fsid 2e3e85a8-fbcf-11ed-84e5-00266cf8869c
Using ceph image with id '0dc91bca92c2' and tag 'v17' created on
2023-05-25 16:26:31 +0000 UTC
quay.io/ceph/ceph@sha256:b8df01a568f4dec7bac6d5040f9391dcca14e00ec7f4de8a3dcf3f2a6502d3a9
Device Path Size Device nodes rotates
available Model name
[root@mostha1 ~]# *cephadm shell lsmcli ldl*
Inferring fsid 4d54823c-fb05-11ed-aecf-00266cf8869c
Inferring config
/var/lib/ceph/4d54823c-fb05-11ed-aecf-00266cf8869c/mon.mostha1/config
Using ceph image with id 'c9a1062f7289' and tag 'v17' created on
2023-04-25 16:04:33 +0000 UTC
quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e
Path | SCSI VPD 0x83 | Link Type | Serial Number | Health
Status
-------------------------------------------------------------------------
*/dev/sda | 50024e92039e4f1c | PATA/SATA | S2B5J90ZA10142 | Good**
**/dev/sdc | 50014ee0ad5953c9 | PATA/SATA | WD-WMAYP0982329 | Good**
**/dev/sdb | 50024e920387fa2c | PATA/SATA | S2B5J90ZA02494 | Good**
*
Could it be a bug in ceph-volume ?
Adam suggest looking to the underlying commands (lsblk, blkid, udevadm,
lvs, or pvs) but I'm not very comfortable with blkid and udevadm. Is
there a "debug flag" to set ceph more verbose ?
Thanks
Patrick
Le 15/05/2023 à 21:20, Adam King a écrit :
As you've already seem to have figured out,
"ceph orch device ls" is
populated with the results from "ceph-volume inventory". My best guess
to try and debug this would be to manually run "cephadm ceph-volume --
inventory" (the same as "cephadm ceph-volume inventory", I just like
to separate the ceph-volume command from cephadm itself with the " --
") and then check /var/log/ceph/<fsid>/ceph-volume.log from when you
ran the command onward to try and see why it isn't seeing your
devices. For example I can see a line like
[2023-05-15 19:11:58,048][ceph_volume.main][INFO ] Running command:
ceph-volume inventory
in there. Then if I look onward from there I can see it ran things like
lsblk -P -o
NAME,KNAME,PKNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL
as part of getting my device list. So if I was having issues I would
try running that directly and see what I got. Will note that
ceph-volume on certain more recent versions (not sure about octopus)
runs commands through nsenter, so you'd have to look past that part in
the log lines to the underlying command being used, typically
something with lsblk, blkid, udevadm, lvs, or pvs.
Also, if you want to see if it's an issue with a certain version of
ceph-volume, you can use different versions by passing the image flag
to cephadm. E.g.
cephadm --image quay.io/ceph/ceph:v17.2.6
<http://quay.io/ceph/ceph:v17.2.6> ceph-volume -- inventory
would use the 17.2.6 version of ceph-volume for the inventory. It
works by running ceph-volume through the container, so you don't have
to have to worry about installing different packages to try them and
it should pull the container image on its own if it isn't on the
machine already (but note that means the command will take longer as
it pulls the image the first time).
On Sat, May 13, 2023 at 4:34 AM Patrick Begou
<Patrick.Begou(a)univ-grenoble-alpes.fr> wrote:
Hi Joshua,
I've tried these commands but it looks like CEPH is unable to see and
configure these HDDs.
[root@mostha1 ~]# cephadm ceph-volume inventory
Inferring fsid 4b7a6504-f0be-11ed-be1a-00266cf8869c
Using recent ceph image
quay.io/ceph/ceph@sha256:e6919776f0ff8331a8e9c4b18d36c5e9eed31e1a80da62ae8454e42d10e95544
<http://quay.io/ceph/ceph@sha256:e6919776f0ff8331a8e9c4b18d36c5e9eed31e1a80da62ae8454e42d10e95544>
Device Path Size Device nodes rotates
available Model name
[root@mostha1 ~]# cephadm shell
[ceph: root@mostha1 /]# ceph orch apply osd --all-available-devices
Scheduled osd.all-available-devices update...
[ceph: root@mostha1 /]# ceph orch device ls[ceph: root@mostha1 /]#
ceph-volume lvm zap /dev/sdb
--> Zapping: /dev/sdb
--> --destroy was not specified, but zapping a whole device will
remove the partition table
Running command: /usr/bin/dd if=/dev/zero of=/dev/sdb bs=1M
count=10
conv=fsync
stderr: 10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.10039 s, 104 MB/s
--> Zapping successful for: <Raw Device: /dev/sdb>
I can check that /dev/sdb1 has been erased, so previous command is
successful
[ceph: root@mostha1 ceph]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 232.9G 0 disk
|-sda1 8:1 1 3.9G 0 part /rootfs/boot
|-sda2 8:2 1 78.1G 0 part
| `-osvg-rootvol 253:0 0 48.8G 0 lvm /rootfs
|-sda3 8:3 1 3.9G 0 part [SWAP]
`-sda4 8:4 1 146.9G 0 part
|-secretvg-homevol 253:1 0 9.8G 0 lvm /rootfs/home
|-secretvg-tmpvol 253:2 0 9.8G 0 lvm /rootfs/tmp
`-secretvg-varvol 253:3 0 9.8G 0 lvm /rootfs/var
sdb 8:16 1 465.8G 0 disk
sdc 8:32 1 232.9G 0 disk
But still no visible HDD:
[ceph: root@mostha1 ceph]# ceph orch apply osd --all-available-devices
Scheduled osd.all-available-devices update...
[ceph: root@mostha1 ceph]# ceph orch device ls
[ceph: root@mostha1 ceph]#
May be I have done something bad at install time as in the container
I've unintentionally run:
dnf -y install
https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noar…
(an awful copy/paste launching the command). Can this break The
container ? I do not know what should be available as ceph
packages in
the container to remove properly this install (no dnf.log file in the
container)
Patrick
Le 12/05/2023 à 21:38, Beaman, Joshua a écrit :
<https://urldefense.com/v3/__https:/download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noarch.rpm__;!!CQl3mcHX2A!H9cwNCJyKXYQ4BbGA3gwHHRitjOS4lBCZT9wlnBZ-8IDue0MvdcPD8Dnv5yQCZw_eA4BNDYaEq1eouKQcQO7HshgdUJ0SJ-EgLfaBGBmCQ$>
> monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print
$1 }')
> This was sucessfull.
> But running "*c**eph
orch device ls*" do not show any HDD
even if
> I have
> /dev/sda (used by the OS), /dev/sdb and /dev/sdc
> The web interface shows a row
capacity which is an aggregate
of the
> sizes of the 3 HDDs for the node.
> I've also tried to reset
/dev/sdb but cephadm do not see it:
> [ceph: root@mostha1 /]#
ceph orch device zap
> mostha1.legi.grenoble-inp.fr
<http://mostha1.legi.grenoble-inp.fr> /dev/sdb --force
> On my first attempt with
octopus, I was able to list the
available
> Any advices for a CEPH
beginner ?
> Thanks
> Patrick
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io The most significant point I see there, is you
have no OSD service
spec to tell orchestrator how to deploy OSDs. The easiest fix for
that would be “cephorchapplyosd--all-available-devices”
This will create a simple spec that should work for a test
environment. Most likely it will collocate the block, block.db,
and
WAL all on the same device. Not ideal for prod
environments,
but fine
for practice and testing.
The other command I should have had you try is “cephadm ceph-volume
inventory”. That should show you the devices available for OSD
deployment, and hopefully matches up to what your “lsblk”
shows. If
you need to zap HDDs and orchestrator is still
not seeing them, you
can try “cephadm ceph-volume lvm zap /dev/sdb”
Thank you,
Josh Beaman
*From: *Patrick Begou <Patrick.Begou(a)univ-grenoble-alpes.fr>
*Date: *Friday, May 12, 2023 at 2:22 PM
*To: *Beaman, Joshua <Joshua_Beaman(a)comcast.com>om>, ceph-users
<ceph-users(a)ceph.io>
*Subject: *Re: [EXTERNAL] [ceph-users] [Pacific] ceph orch
device ls
do not returns any HDD
Hi Joshua and thanks for this quick reply.
At this step I have only one node. I was checking what ceph was
returning with different commands on this host before adding new
hosts. Just to compare with my first Octopus install. As this
hardware
is for testing only, it remains easy for me to
break everything and
reinstall again.
[root@mostha1 ~]# cephadm check-host
podman (/usr/bin/podman) version 4.2.0 is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
[ceph: root@mostha1 /]# ceph -s
cluster:
id: 4b7a6504-f0be-11ed-be1a-00266cf8869c
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 3
services:
mon: 1 daemons, quorum mostha1.legi.grenoble-inp.fr
<http://mostha1.legi.grenoble-inp.fr> (age 5h)
mgr: mostha1.legi.grenoble-inp.fr
<http://mostha1.legi.grenoble-inp.fr>.hogwuz(active, since 5h)
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
[ceph: root@mostha1 /]# ceph orch ls
NAME PORTS RUNNING REFRESHED AGE PLACEMENT
alertmanager ?:9093,9094 1/1 6m ago 6h count:1
crash 1/1 6m ago 6h *
grafana ?:3000 1/1 6m ago 6h count:1
mgr 1/2 6m ago 6h count:2
mon 1/5 6m ago 6h count:5
node-exporter ?:9100 1/1 6m ago 6h *
prometheus ?:9095 1/1 6m ago 6h count:1
[ceph: root@mostha1 /]# ceph orch ls osd -export
No services reported
[ceph: root@mostha1 /]# ceph orch host ls
HOST ADDR LABELS STATUS
mostha1.legi.grenoble-inp.fr
<http://mostha1.legi.grenoble-inp.fr>
194.254.66.34 _admin
1 hosts in cluster
[ceph: root@mostha1 /]# ceph log last cephadm
...
2023-05-12T15:19:58.754655+0000
mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1876 :
cephadm
[INF] Zap device
mostha1.legi.grenoble-inp.fr:/dev/sdb
2023-05-12T15:19:58.756639+0000
mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1877 :
cephadm
[ERR] Device path '/dev/sdb' not
found on host
'mostha1.legi.grenoble-inp.fr
<http://mostha1.legi.grenoble-inp.fr>'
Traceback (most recent call last):
File "/usr/share/ceph/mgr/orchestrator/_interface.py",
line
125,
in wrapper
return OrchResult(f(*args, **kwargs))
File "/usr/share/ceph/mgr/cephadm/module.py", line 2275, in
zap_device
f"Device path '{path}' not found on host '{host}'")
orchestrator._interface.OrchestratorError: Device path
'/dev/sdb'
not found on host
'mostha1.legi.grenoble-inp.fr
<http://mostha1.legi.grenoble-inp.fr>'
....
[ceph: root@mostha1 /]# ls -l /dev/sdb
brw-rw---- 1 root disk 8, 16 May 12 15:16 /dev/sdb
[ceph: root@mostha1 /]# lsblk /dev/sdb
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 1 465.8G 0 disk
`-sdb1 8:17 1 465.8G 0 part
I have crated a full partition on /dev/sdb (for testing) and
/dev/sdc
has no partition table (removed).
But all seams fine with these commands.
Patrick
Le 12/05/2023 à 20:19, Beaman, Joshua a écrit :
I don’t quite understand why that zap would not work. But,
here’s
where I’d start.
1. cephadm check-host
1. Run this on each of your hosts to make sure cephadm,
podman and all other prerequisites are installed and
recognized
2. ceph orch ls
1. This should show at least a mon, mgr, and osd spec
deployed
3. ceph orch ls osd –export
1. This will show the OSD placement service specifications
that orchestrator uses to identify devices to deploy
as OSDs
4. ceph orch host ls
1. This will list the hosts that have been added to
orchestrator’s inventory, and what labels are applied
which correlate to the service placement labels
5. ceph log last cephadm
1. This will show you what orchestrator has been trying to
do, and how it may be failing
Also, it’s never un-helpful to have a look at “ceph -s” and
“ceph
health detail”, particularly for any people
trying to help you
without access to your systems.
Best of luck,
Josh Beaman
*From: *Patrick Begou <Patrick.Begou(a)univ-grenoble-alpes.fr>
<mailto:Patrick.Begou@univ-grenoble-alpes.fr>
*Date: *Friday, May 12, 2023 at 10:45 AM
*To: *ceph-users <ceph-users(a)ceph.io>
<mailto:ceph-users@ceph.io>
*Subject: *[EXTERNAL] [ceph-users] [Pacific]
ceph orch device ls
do not returns any HDD
Hi everyone
I'm new to CEPH, just a french 4 days training session with
Octopus on
VMs that convince me to build my first cluster.
At this time I have 4 old identical nodes for testing with 3
HDDs
each,
2 network interfaces and running Alma Linux8 (el8). I try to
replay the
training session but it fails, breaking the web interface
because of
some problems with podman 4.2 not compatible
with Octopus.
So I try to deploy Pacific with cephadm tool on my first node
(mostha1)
(to enable testing also an upgrade later).
dnf -y install
https://urldefense.com/v3/__https://download.ceph.com/rpm-16.2.13/el8/noarc…
cephadm bootstrap --mon-ip $monip
--initial-dashboard-password
> xxxxx \
> --initial-dashboard-user admceph \
> --allow-fqdn-hostname --cluster-network
> 10.1.0.0/16 <http://10.1.0.0/16>
Error EINVAL: Device path
'/dev/sdb' not found on host
'mostha1.legi.grenoble-inp.fr
<http://mostha1.legi.grenoble-inp.fr>'
HDD
with this command line. Before moving to Pacific, the OS on
this node
> has been reinstalled from scratch.
11:34 p.m.
New subject: [EXTERNAL] [Pacific] ceph orch device ls do not returns any HDD
Hi Michel,
I do not notice anything strange in the logs files (looking for errors
or warnings).
The hardware is a DELL C6100 sled (from 2011) running Alma Linux8
up-to-date. It uses 3 sata disks.
Is there a way to force osd installation by hand with providing the
device /dev/sdc for example ? A "do what I say" approach...
Is it a good try to deploy Octopus on the nodes, configure the osd (even
if podman 4.2.0 is not validated for Octopus) and then upgrade to
Pacific? Could this be a workaround for this sort of regression from
Octopus to Pacific ?
May be updating the BIOS from 1.7.1 to 1.8.1 ?
All this is a little bit confusing for me as I'm trying to discover Ceph 😁
Thanks
Patrick
Le 26/05/2023 à 17:19, Michel Jouvin a écrit :
Hi Patrick,
It is weird, we have a couple of clusters with cephadm and running
pacify or quincy and ceph orch device works well. Have you looked at
the cephadm logs (ceph log last cephadm)?
Except if you are using a very specific hardware, I suspect Ceph is
suffering of a problem outside it...
Cheers,
Michel
Sent from my mobile
Le 26 mai 2023 17:02:50 Patrick Begou
<Patrick.Begou(a)univ-grenoble-alpes.fr> a écrit :
Hi,
I'm back working on this problem.
First of all, I saw that I had a hardware memory error so I had to solve
this first. It's done.
I've tested some different Ceph deployments, each time starting with a
full OS re-install (it requires some time for each test).
Using Octopus, the devices are found:
dnf -y install \
https://download.ceph.com/rpm-15.2.12/el8/noarch/cephadm-15.2.12-0.el8.noar…
monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }'))
cephadm bootstrap --mon-ip $monip --initial-dashboard-password
xxxxx \
--allow-fqdn-hostname
[ceph: root@mostha1 /]# *ceph orch device ls*
Hostname Path Type Serial Size Health
Ident Fault Available
mostha1.legi.grenoble-inp.fr /dev/sda hdd S2B5J90ZA02494 250G
Unknown N/A N/A Yes
mostha1.legi.grenoble-inp.fr /dev/sdc hdd WD-WMAYP0982329 500G
Unknown N/A N/A Yes
But with Pacific or Quincy the command returns nothing.
With Pacific:
dnf -y install \
https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noar…
monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }')
cephadm bootstrap --mon-ip $monip --initial-dashboard-password
xxxxx \
--allow-fqdn-hostname
"ceph orch device ls" doesn't return anything but "cephadm shell
lsmcli
ldl" list all the devices.
[ceph: root@mostha1 /]# *ceph orch device ls --wide*
[ceph: root@mostha1 /]# *lsblk*
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 232.9G 0 disk
|-sda1 8:1 1 3.9G 0 part /rootfs/boot
|-sda2 8:2 1 78.1G 0 part
| `-osvg-rootvol 253:0 0 48.8G 0 lvm /rootfs
|-sda3 8:3 1 3.9G 0 part [SWAP]
`-sda4 8:4 1 146.9G 0 part
|-secretvg-homevol 253:1 0 9.8G 0 lvm /rootfs/home
|-secretvg-tmpvol 253:2 0 9.8G 0 lvm /rootfs/tmp
`-secretvg-varvol 253:3 0 9.8G 0 lvm /rootfs/var
sdb 8:16 1 232.9G 0 disk
sdc 8:32 1 465.8G 0 disk
[ceph: root@mostha1 /]# exit
[root@mostha1 ~]# *cephadm ceph-volume inventory*
Inferring fsid 2e3e85a8-fbcf-11ed-84e5-00266cf8869c
Using ceph image with id '0dc91bca92c2' and tag 'v17' created on
2023-05-25 16:26:31 +0000 UTC
quay.io/ceph/ceph@sha256:b8df01a568f4dec7bac6d5040f9391dcca14e00ec7f4de8a3dcf3f2a6502d3a9
Device Path Size Device nodes rotates
available Model name
[root@mostha1 ~]# *cephadm shell lsmcli ldl*
Inferring fsid 4d54823c-fb05-11ed-aecf-00266cf8869c
Inferring config
/var/lib/ceph/4d54823c-fb05-11ed-aecf-00266cf8869c/mon.mostha1/config
Using ceph image with id 'c9a1062f7289' and tag 'v17' created on
2023-04-25 16:04:33 +0000 UTC
quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e
Path | SCSI VPD 0x83 | Link Type | Serial Number | Health
Status
-------------------------------------------------------------------------
*/dev/sda | 50024e92039e4f1c | PATA/SATA | S2B5J90ZA10142 | Good**
**/dev/sdc | 50014ee0ad5953c9 | PATA/SATA | WD-WMAYP0982329 | Good**
**/dev/sdb | 50024e920387fa2c | PATA/SATA | S2B5J90ZA02494 | Good**
*
Could it be a bug in ceph-volume ?
Adam suggest looking to the underlying commands (lsblk, blkid, udevadm,
lvs, or pvs) but I'm not very comfortable with blkid and udevadm. Is
there a "debug flag" to set ceph more verbose ?
Thanks
Patrick
Le 15/05/2023 à 21:20, Adam King a écrit :
As you've already seem to have figured out,
"ceph orch device ls" is
populated with the results from "ceph-volume inventory". My best guess
to try and debug this would be to manually run "cephadm ceph-volume --
inventory" (the same as "cephadm ceph-volume inventory", I just like
to separate the ceph-volume command from cephadm itself with the " --
") and then check /var/log/ceph/<fsid>/ceph-volume.log from when you
ran the command onward to try and see why it isn't seeing your
devices. For example I can see a line like
[2023-05-15 19:11:58,048][ceph_volume.main][INFO ] Running command:
ceph-volume inventory
in there. Then if I look onward from there I can see it ran things like
lsblk -P -o
NAME,KNAME,PKNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL
as part of getting my device list. So if I was having issues I would
try running that directly and see what I got. Will note that
ceph-volume on certain more recent versions (not sure about octopus)
runs commands through nsenter, so you'd have to look past that part in
the log lines to the underlying command being used, typically
something with lsblk, blkid, udevadm, lvs, or pvs.
Also, if you want to see if it's an issue with a certain version of
ceph-volume, you can use different versions by passing the image flag
to cephadm. E.g.
cephadm --image quay.io/ceph/ceph:v17.2.6
<http://quay.io/ceph/ceph:v17.2.6> ceph-volume -- inventory
would use the 17.2.6 version of ceph-volume for the inventory. It
works by running ceph-volume through the container, so you don't have
to have to worry about installing different packages to try them and
it should pull the container image on its own if it isn't on the
machine already (but note that means the command will take longer as
it pulls the image the first time).
On Sat, May 13, 2023 at 4:34 AM Patrick Begou
<Patrick.Begou(a)univ-grenoble-alpes.fr> wrote:
Hi Joshua,
I've tried these commands but it looks like CEPH is unable to see and
configure these HDDs.
[root@mostha1 ~]# cephadm ceph-volume inventory
Inferring fsid 4b7a6504-f0be-11ed-be1a-00266cf8869c
Using recent ceph image
quay.io/ceph/ceph@sha256:e6919776f0ff8331a8e9c4b18d36c5e9eed31e1a80da62ae8454e42d10e95544
<http://quay.io/ceph/ceph@sha256:e6919776f0ff8331a8e9c4b18d36c5e9eed31e1a80da62ae8454e42d10e95544>
Device Path Size Device nodes rotates
available Model name
[root@mostha1 ~]# cephadm shell
[ceph: root@mostha1 /]# ceph orch apply osd --all-available-devices
Scheduled osd.all-available-devices update...
[ceph: root@mostha1 /]# ceph orch device ls[ceph: root@mostha1 /]#
ceph-volume lvm zap /dev/sdb
--> Zapping: /dev/sdb
--> --destroy was not specified, but zapping a whole device will
remove the partition table
Running command: /usr/bin/dd if=/dev/zero of=/dev/sdb bs=1M
count=10
conv=fsync
stderr: 10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.10039 s, 104 MB/s
--> Zapping successful for: <Raw Device: /dev/sdb>
I can check that /dev/sdb1 has been erased, so previous command is
successful
[ceph: root@mostha1 ceph]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 232.9G 0 disk
|-sda1 8:1 1 3.9G 0 part /rootfs/boot
|-sda2 8:2 1 78.1G 0 part
| `-osvg-rootvol 253:0 0 48.8G 0 lvm /rootfs
|-sda3 8:3 1 3.9G 0 part [SWAP]
`-sda4 8:4 1 146.9G 0 part
|-secretvg-homevol 253:1 0 9.8G 0 lvm /rootfs/home
|-secretvg-tmpvol 253:2 0 9.8G 0 lvm /rootfs/tmp
`-secretvg-varvol 253:3 0 9.8G 0 lvm /rootfs/var
sdb 8:16 1 465.8G 0 disk
sdc 8:32 1 232.9G 0 disk
But still no visible HDD:
[ceph: root@mostha1 ceph]# ceph orch apply osd --all-available-devices
Scheduled osd.all-available-devices update...
[ceph: root@mostha1 ceph]# ceph orch device ls
[ceph: root@mostha1 ceph]#
May be I have done something bad at install time as in the container
I've unintentionally run:
dnf -y install
https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noar…
(an awful copy/paste launching the command). Can this break The
container ? I do not know what should be available as ceph
packages in
the container to remove properly this install (no dnf.log file in the
container)
Patrick
Le 12/05/2023 à 21:38, Beaman, Joshua a écrit :
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io The most significant point I see there, is you
have no OSD service
spec to tell orchestrator how to deploy OSDs. The easiest fix for
that would be “cephorchapplyosd--all-available-devices”
This will create a simple spec that should work for a test
environment. Most likely it will collocate the block, block.db,
and
WAL all on the same device. Not ideal for prod
environments,
but fine
for practice and testing.
The other command I should have had you try is “cephadm ceph-volume
inventory”. That should show you the devices available for OSD
deployment, and hopefully matches up to what your “lsblk”
shows. If
you need to zap HDDs and orchestrator is still
not seeing them, you
can try “cephadm ceph-volume lvm zap /dev/sdb”
Thank you,
Josh Beaman
*From: *Patrick Begou <Patrick.Begou(a)univ-grenoble-alpes.fr>
*Date: *Friday, May 12, 2023 at 2:22 PM
*To: *Beaman, Joshua <Joshua_Beaman(a)comcast.com>om>, ceph-users
<ceph-users(a)ceph.io>
*Subject: *Re: [EXTERNAL] [ceph-users] [Pacific] ceph orch
device ls
do not returns any HDD
Hi Joshua and thanks for this quick reply.
At this step I have only one node. I was checking what ceph was
returning with different commands on this host before adding new
hosts. Just to compare with my first Octopus install. As this
hardware
is for testing only, it remains easy for me to
break everything and
reinstall again.
[root@mostha1 ~]# cephadm check-host
podman (/usr/bin/podman) version 4.2.0 is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
[ceph: root@mostha1 /]# ceph -s
cluster:
id: 4b7a6504-f0be-11ed-be1a-00266cf8869c
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 3
services:
mon: 1 daemons, quorum mostha1.legi.grenoble-inp.fr
<http://mostha1.legi.grenoble-inp.fr> (age 5h)
mgr: mostha1.legi.grenoble-inp.fr
<http://mostha1.legi.grenoble-inp.fr>.hogwuz(active, since 5h)
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
[ceph: root@mostha1 /]# ceph orch ls
NAME PORTS RUNNING REFRESHED AGE PLACEMENT
alertmanager ?:9093,9094 1/1 6m ago 6h count:1
crash 1/1 6m ago 6h *
grafana ?:3000 1/1 6m ago 6h count:1
mgr 1/2 6m ago 6h count:2
mon 1/5 6m ago 6h count:5
node-exporter ?:9100 1/1 6m ago 6h *
prometheus ?:9095 1/1 6m ago 6h count:1
[ceph: root@mostha1 /]# ceph orch ls osd -export
No services reported
[ceph: root@mostha1 /]# ceph orch host ls
HOST ADDR LABELS STATUS
mostha1.legi.grenoble-inp.fr
<http://mostha1.legi.grenoble-inp.fr>
194.254.66.34 _admin
1 hosts in cluster
[ceph: root@mostha1 /]# ceph log last cephadm
...
2023-05-12T15:19:58.754655+0000
mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1876 :
cephadm
[INF] Zap device
mostha1.legi.grenoble-inp.fr:/dev/sdb
2023-05-12T15:19:58.756639+0000
mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1877 :
cephadm
[ERR] Device path '/dev/sdb' not
found on host
'mostha1.legi.grenoble-inp.fr
<http://mostha1.legi.grenoble-inp.fr>'
Traceback (most recent call last):
File "/usr/share/ceph/mgr/orchestrator/_interface.py",
line 125,
in wrapper
return OrchResult(f(*args, **kwargs))
File "/usr/share/ceph/mgr/cephadm/module.py", line 2275, in
zap_device
f"Device path '{path}' not found on host '{host}'")
orchestrator._interface.OrchestratorError: Device path
'/dev/sdb'
not found on host
'mostha1.legi.grenoble-inp.fr
<http://mostha1.legi.grenoble-inp.fr>'
....
[ceph: root@mostha1 /]# ls -l /dev/sdb
brw-rw---- 1 root disk 8, 16 May 12 15:16 /dev/sdb
[ceph: root@mostha1 /]# lsblk /dev/sdb
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 1 465.8G 0 disk
`-sdb1 8:17 1 465.8G 0 part
I have crated a full partition on /dev/sdb (for testing) and
/dev/sdc
has no partition table (removed).
But all seams fine with these commands.
Patrick
Le 12/05/2023 à 20:19, Beaman, Joshua a écrit :
I don’t quite understand why that zap would not work. But,
here’s
where I’d start.
1. cephadm check-host
1. Run this on each of your hosts to make sure cephadm,
podman and all other prerequisites are installed and
recognized
2. ceph orch ls
1. This should show at least a mon, mgr, and osd spec
deployed
3. ceph orch ls osd –export
1. This will show the OSD placement service specifications
that orchestrator uses to identify devices to deploy
as OSDs
4. ceph orch host ls
1. This will list the hosts that have been added to
orchestrator’s inventory, and what labels are applied
which correlate to the service placement labels
5. ceph log last cephadm
1. This will show you what orchestrator has been trying to
do, and how it may be failing
Also, it’s never un-helpful to have a look at “ceph -s” and
“ceph
health detail”, particularly for any people
trying to help you
without access to your systems.
Best of luck,
Josh Beaman
*From: *Patrick Begou <Patrick.Begou(a)univ-grenoble-alpes.fr>
<mailto:Patrick.Begou@univ-grenoble-alpes.fr>
*Date: *Friday, May 12, 2023 at 10:45 AM
*To: *ceph-users <ceph-users(a)ceph.io>
<mailto:ceph-users@ceph.io>
*Subject: *[EXTERNAL] [ceph-users] [Pacific]
ceph orch device ls
do not returns any HDD
Hi everyone
I'm new to CEPH, just a french 4 days training session with
Octopus on
VMs that convince me to build my first cluster.
At this time I have 4 old identical nodes for testing with 3
HDDs
each,
2 network interfaces and running Alma Linux8 (el8). I try to
replay the
training session but it fails, breaking the web interface
because of
some problems with podman 4.2 not compatible
with Octopus.
So I try to deploy Pacific with cephadm tool on my first node
(mostha1)
(to enable testing also an upgrade later).
dnf -y install
https://urldefense.com/v3/__https://download.ceph.com/rpm-16.2.13/el8/noarc…
<https://urldefense.com/v3/__https:/download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noarch.rpm__;!!CQl3mcHX2A!H9cwNCJyKXYQ4BbGA3gwHHRitjOS4lBCZT9wlnBZ-8IDue0MvdcPD8Dnv5yQCZw_eA4BNDYaEq1eouKQcQO7HshgdUJ0SJ-EgLfaBGBmCQ$>
monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print
$1 }')
cephadm bootstrap --mon-ip $monip
--initial-dashboard-password
xxxxx \
--initial-dashboard-user admceph \
--allow-fqdn-hostname --cluster-network
10.1.0.0/16 <http://10.1.0.0/16>
This was sucessfull.
But running "*c**eph orch device ls*" do not show any HDD
even if
I have
/dev/sda (used by the OS), /dev/sdb and /dev/sdc
The web interface shows a row capacity which is an aggregate
of the
sizes of the 3 HDDs for the node.
I've also tried to reset /dev/sdb but cephadm do not see it:
[ceph: root@mostha1 /]# ceph orch device zap
mostha1.legi.grenoble-inp.fr
<http://mostha1.legi.grenoble-inp.fr> /dev/sdb
--force
Error EINVAL: Device path
'/dev/sdb' not found on host
'mostha1.legi.grenoble-inp.fr
<http://mostha1.legi.grenoble-inp.fr>'
On my first attempt with octopus, I was able to list the
available
HDD
with this command line. Before moving to Pacific, the OS on
this node
has been reinstalled from scratch.
Any advices for a CEPH beginner ?
Thanks
Patrick
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:50 p.m.
New subject: [EXTERNAL] [Pacific] ceph orch device ls do not returns any HDD
Patrick,
I can only say that I would not expect a specific problem due to your
hardware. Upgrading the firmware is generally a good idea but I wouldn't
expect it helps in your case if the osk (lsblk) sees the disk.
As for starting with octopus I don't know if it will help... But we are
also using the same os as you (centos stream in fact but basically the
same). We have been running octopus (with cephadm) on this os version
without problem and upgraded since then in pacific and quincy. In fact on
of our cluster started in infernetis, the other in luminous and they have
been upgraded without problems since then...
Michel
Sent from my mobile
Le 26 mai 2023 18:34:22 Patrick Begou
<Patrick.Begou(a)univ-grenoble-alpes.fr> a écrit :
Hi Michel,
I do not notice anything strange in the logs files (looking for errors or
warnings).
The hardware is a DELL C6100 sled (from 2011) running Alma Linux8
up-to-date. It uses 3 sata disks.
Is there a way to force osd installation by hand with providing the device
/dev/sdc for example ? A "do what I say" approach...
Is it a good try to deploy Octopus on the nodes, configure the osd (even if
podman 4.2.0 is not validated for Octopus) and then upgrade to Pacific?
Could this be a workaround for this sort of regression from Octopus to
Pacific ?
May be updating the BIOS from 1.7.1 to 1.8.1 ?
All this is a little bit confusing for me as I'm trying to discover Ceph 😁
Thanks
Patrick
Le 26/05/2023 à 17:19, Michel Jouvin a écrit :
> Hi Patrick,
>
> It is weird, we have a couple of clusters with cephadm and running pacify
> or quincy and ceph orch device works well. Have you looked at the cephadm
> logs (ceph log last cephadm)?
>
> Except if you are using a very specific hardware, I suspect Ceph is
> suffering of a problem outside it...
>
> Cheers,
>
> Michel
> Sent from my mobile
>
> Le 26 mai 2023 17:02:50 Patrick Begou
> <Patrick.Begou(a)univ-grenoble-alpes.fr> a écrit :
>
>> Hi,
>>
>> I'm back working on this problem.
>>
>> First of all, I saw that I had a hardware memory error so I had to solve
>> this first. It's done.
>>
>> I've tested some different Ceph deployments, each time starting with a
>> full OS re-install (it requires some time for each test).
>>
>> Using Octopus, the devices are found:
>>
>> dnf -y install \
>>
https://download.ceph.com/rpm-15.2.12/el8/noarch/cephadm-15.2.12-0.el8.noar…
>> monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }'))
>> cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \
>> --allow-fqdn-hostname
>>
>> [ceph: root@mostha1 /]# *ceph orch device ls*
>> Hostname Path Type Serial Size Health
>> Ident Fault Available
>> mostha1.legi.grenoble-inp.fr /dev/sda hdd S2B5J90ZA02494 250G
>> Unknown N/A N/A Yes
>> mostha1.legi.grenoble-inp.fr /dev/sdc hdd WD-WMAYP0982329 500G
>> Unknown N/A N/A Yes
>>
>>
>> But with Pacific or Quincy the command returns nothing.
>>
>> With Pacific:
>>
>> dnf -y install \
>>
https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noar…
>> monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }')
>> cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \
>> --allow-fqdn-hostname
>>
>>
>> "ceph orch device ls" doesn't return anything but "cephadm
shell lsmcli
>> ldl" list all the devices.
>>
>> [ceph: root@mostha1 /]# *ceph orch device ls --wide*
>> [ceph: root@mostha1 /]# *lsblk*
>> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
>> sda 8:0 1 232.9G 0 disk
>> |-sda1 8:1 1 3.9G 0 part /rootfs/boot
>> |-sda2 8:2 1 78.1G 0 part
>> | `-osvg-rootvol 253:0 0 48.8G 0 lvm /rootfs
>> |-sda3 8:3 1 3.9G 0 part [SWAP]
>> `-sda4 8:4 1 146.9G 0 part
>> |-secretvg-homevol 253:1 0 9.8G 0 lvm /rootfs/home
>> |-secretvg-tmpvol 253:2 0 9.8G 0 lvm /rootfs/tmp
>> `-secretvg-varvol 253:3 0 9.8G 0 lvm /rootfs/var
>> sdb 8:16 1 232.9G 0 disk
>> sdc 8:32 1 465.8G 0 disk
>> [ceph: root@mostha1 /]# exit
>> [root@mostha1 ~]# *cephadm ceph-volume inventory*
>> Inferring fsid 2e3e85a8-fbcf-11ed-84e5-00266cf8869c
>> Using ceph image with id '0dc91bca92c2' and tag 'v17' created on
>> 2023-05-25 16:26:31 +0000 UTC
>>
quay.io/ceph/ceph@sha256:b8df01a568f4dec7bac6d5040f9391dcca14e00ec7f4de8a3dcf3f2a6502d3a9
>>
>> Device Path Size Device nodes rotates
>> available Model name
>>
>> [root@mostha1 ~]# *cephadm shell lsmcli ldl*
>> Inferring fsid 4d54823c-fb05-11ed-aecf-00266cf8869c
>> Inferring config
>> /var/lib/ceph/4d54823c-fb05-11ed-aecf-00266cf8869c/mon.mostha1/config
>> Using ceph image with id 'c9a1062f7289' and tag 'v17' created on
>> 2023-04-25 16:04:33 +0000 UTC
>>
quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e
>> Path | SCSI VPD 0x83 | Link Type | Serial Number | Health
>> Status
>> -------------------------------------------------------------------------
>> */dev/sda | 50024e92039e4f1c | PATA/SATA | S2B5J90ZA10142 | Good**
>> **/dev/sdc | 50014ee0ad5953c9 | PATA/SATA | WD-WMAYP0982329 | Good**
>> **/dev/sdb | 50024e920387fa2c | PATA/SATA | S2B5J90ZA02494 | Good**
>> *
>>
>>
>> Could it be a bug in ceph-volume ?
>> Adam suggest looking to the underlying commands (lsblk, blkid, udevadm,
>> lvs, or pvs) but I'm not very comfortable with blkid and udevadm. Is
>> there a "debug flag" to set ceph more verbose ?
>>
>> Thanks
>>
>> Patrick
>>
>> Le 15/05/2023 à 21:20, Adam King a écrit :
>>> As you've already seem to have figured out, "ceph orch device
ls" is
>>> populated with the results from "ceph-volume inventory". My best
guess
>>> to try and debug this would be to manually run "cephadm ceph-volume --
>>> inventory" (the same as "cephadm ceph-volume inventory", I
just like
>>> to separate the ceph-volume command from cephadm itself with the " --
>>> ") and then check /var/log/ceph/<fsid>/ceph-volume.log from when
you
>>> ran the command onward to try and see why it isn't seeing your
>>> devices. For example I can see a line like
>>>
>>> [2023-05-15 19:11:58,048][ceph_volume.main][INFO ] Running command:
>>> ceph-volume inventory
>>>
>>> in there. Then if I look onward from there I can see it ran things like
>>>
>>> lsblk -P -o
>>>
NAME,KNAME,PKNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL
>>>
>>> as part of getting my device list. So if I was having issues I would
>>> try running that directly and see what I got. Will note that
>>> ceph-volume on certain more recent versions (not sure about octopus)
>>> runs commands through nsenter, so you'd have to look past that part in
>>> the log lines to the underlying command being used, typically
>>> something with lsblk, blkid, udevadm, lvs, or pvs.
>>>
>>> Also, if you want to see if it's an issue with a certain version of
>>> ceph-volume, you can use different versions by passing the image flag
>>> to cephadm. E.g.
>>>
>>> cephadm --image quay.io/ceph/ceph:v17.2.6
>>> <http://quay.io/ceph/ceph:v17.2.6> ceph-volume -- inventory
>>>
>>> would use the 17.2.6 version of ceph-volume for the inventory. It
>>> works by running ceph-volume through the container, so you don't have
>>> to have to worry about installing different packages to try them and
>>> it should pull the container image on its own if it isn't on the
>>> machine already (but note that means the command will take longer as
>>> it pulls the image the first time).
>>>
>>>
>>>
>>> On Sat, May 13, 2023 at 4:34 AM Patrick Begou
>>> <Patrick.Begou(a)univ-grenoble-alpes.fr> wrote:
>>>
>>> Hi Joshua,
>>>
>>> I've tried these commands but it looks like CEPH is unable to see and
>>> configure these HDDs.
>>> [root@mostha1 ~]# cephadm ceph-volume inventory
>>>
>>> Inferring fsid 4b7a6504-f0be-11ed-be1a-00266cf8869c
>>> Using recent ceph image
>>>
quay.io/ceph/ceph@sha256:e6919776f0ff8331a8e9c4b18d36c5e9eed31e1a80da62ae8454e42d10e95544
>>>
<http://quay.io/ceph/ceph@sha256:e6919776f0ff8331a8e9c4b18d36c5e9eed31e1a80da62ae8454e42d10e95544>
>>>
>>> Device Path Size Device nodes rotates
>>> available Model name
>>>
>>> [root@mostha1 ~]# cephadm shell
>>>
>>> [ceph: root@mostha1 /]# ceph orch apply osd --all-available-devices
>>>
>>> Scheduled osd.all-available-devices update...
>>>
>>> [ceph: root@mostha1 /]# ceph orch device ls[ceph: root@mostha1 /]#
>>> ceph-volume lvm zap /dev/sdb
>>>
>>> --> Zapping: /dev/sdb
>>> --> --destroy was not specified, but zapping a whole device will
>>> remove the partition table
>>> Running command: /usr/bin/dd if=/dev/zero of=/dev/sdb bs=1M
>>> count=10
>>> conv=fsync
>>> stderr: 10+0 records in
>>> 10+0 records out
>>> 10485760 bytes (10 MB, 10 MiB) copied, 0.10039 s, 104 MB/s
>>> --> Zapping successful for: <Raw Device: /dev/sdb>
>>>
>>> I can check that /dev/sdb1 has been erased, so previous command is
>>> successful
>>> [ceph: root@mostha1 ceph]# lsblk
>>> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
>>> sda 8:0 1 232.9G 0 disk
>>> |-sda1 8:1 1 3.9G 0 part /rootfs/boot
>>> |-sda2 8:2 1 78.1G 0 part
>>> | `-osvg-rootvol 253:0 0 48.8G 0 lvm /rootfs
>>> |-sda3 8:3 1 3.9G 0 part [SWAP]
>>> `-sda4 8:4 1 146.9G 0 part
>>> |-secretvg-homevol 253:1 0 9.8G 0 lvm /rootfs/home
>>> |-secretvg-tmpvol 253:2 0 9.8G 0 lvm /rootfs/tmp
>>> `-secretvg-varvol 253:3 0 9.8G 0 lvm /rootfs/var
>>> sdb 8:16 1 465.8G 0 disk
>>> sdc 8:32 1 232.9G 0 disk
>>>
>>> But still no visible HDD:
>>>
>>> [ceph: root@mostha1 ceph]# ceph orch apply osd --all-available-devices
>>>
>>> Scheduled osd.all-available-devices update...
>>>
>>> [ceph: root@mostha1 ceph]# ceph orch device ls
>>> [ceph: root@mostha1 ceph]#
>>>
>>> May be I have done something bad at install time as in the container
>>> I've unintentionally run:
>>>
>>> dnf -y install
>>>
https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noar…
>>>
>>> (an awful copy/paste launching the command). Can this break The
>>> container ? I do not know what should be available as ceph
>>> packages in
>>> the container to remove properly this install (no dnf.log file in the
>>> container)
>>>
>>> Patrick
>>>
>>>
>>> Le 12/05/2023 à 21:38, Beaman, Joshua a écrit :
>>>> The most significant point I see there, is you have no OSD service
>>>> spec to tell orchestrator how to deploy OSDs. The easiest fix for
>>>> that would be “cephorchapplyosd--all-available-devices”
>>>>
>>>> This will create a simple spec that should work for a test
>>>> environment. Most likely it will collocate the block, block.db,
>>> and
>>>> WAL all on the same device. Not ideal for prod environments,
>>> but fine
>>>> for practice and testing.
>>>>
>>>> The other command I should have had you try is “cephadm ceph-volume
>>>> inventory”. That should show you the devices available for OSD
>>>> deployment, and hopefully matches up to what your “lsblk”
>>> shows. If
>>>> you need to zap HDDs and orchestrator is still not seeing them, you
>>>> can try “cephadm ceph-volume lvm zap /dev/sdb”
>>>>
>>>> Thank you,
>>>>
>>>> Josh Beaman
>>>>
>>>> *From: *Patrick Begou <Patrick.Begou(a)univ-grenoble-alpes.fr>
>>>> *Date: *Friday, May 12, 2023 at 2:22 PM
>>>> *To: *Beaman, Joshua <Joshua_Beaman(a)comcast.com>om>, ceph-users
>>>> <ceph-users(a)ceph.io>
>>>> *Subject: *Re: [EXTERNAL] [ceph-users] [Pacific] ceph orch
>>> device ls
>>>> do not returns any HDD
>>>>
>>>> Hi Joshua and thanks for this quick reply.
>>>>
>>>> At this step I have only one node. I was checking what ceph was
>>>> returning with different commands on this host before adding new
>>>> hosts. Just to compare with my first Octopus install. As this
>>> hardware
>>>> is for testing only, it remains easy for me to break everything and
>>>> reinstall again.
>>>>
>>>> [root@mostha1 ~]# cephadm check-host
>>>>
>>>> podman (/usr/bin/podman) version 4.2.0 is present
>>>> systemctl is present
>>>> lvcreate is present
>>>> Unit chronyd.service is enabled and running
>>>> Host looks OK
>>>>
>>>> [ceph: root@mostha1 /]# ceph -s
>>>>
>>>> cluster:
>>>> id: 4b7a6504-f0be-11ed-be1a-00266cf8869c
>>>> health: HEALTH_WARN
>>>> OSD count 0 < osd_pool_default_size 3
>>>>
>>>> services:
>>>> mon: 1 daemons, quorum mostha1.legi.grenoble-inp.fr
>>> <http://mostha1.legi.grenoble-inp.fr> (age 5h)
>>>> mgr: mostha1.legi.grenoble-inp.fr
>>> <http://mostha1.legi.grenoble-inp.fr>.hogwuz(active, since 5h)
>>>> osd: 0 osds: 0 up, 0 in
>>>>
>>>> data:
>>>> pools: 0 pools, 0 pgs
>>>> objects: 0 objects, 0 B
>>>> usage: 0 B used, 0 B / 0 B avail
>>>> pgs:
>>>>
>>>> [ceph: root@mostha1 /]# ceph orch ls
>>>>
>>>> NAME PORTS RUNNING REFRESHED AGE PLACEMENT
>>>> alertmanager ?:9093,9094 1/1 6m ago 6h count:1
>>>> crash 1/1 6m ago 6h *
>>>> grafana ?:3000 1/1 6m ago 6h count:1
>>>> mgr 1/2 6m ago 6h count:2
>>>> mon 1/5 6m ago 6h count:5
>>>> node-exporter ?:9100 1/1 6m ago 6h *
>>>> prometheus ?:9095 1/1 6m ago 6h count:1
>>>>
>>>> [ceph: root@mostha1 /]# ceph orch ls osd -export
>>>>
>>>> No services reported
>>>>
>>>> [ceph: root@mostha1 /]# ceph orch host ls
>>>>
>>>> HOST ADDR LABELS STATUS
>>>> mostha1.legi.grenoble-inp.fr
>>> <http://mostha1.legi.grenoble-inp.fr> 194.254.66.34 _admin
>>>> 1 hosts in cluster
>>>>
>>>> [ceph: root@mostha1 /]# ceph log last cephadm
>>>>
>>>> ...
>>>> 2023-05-12T15:19:58.754655+0000
>>>> mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1876 :
>>> cephadm
>>>> [INF] Zap device mostha1.legi.grenoble-inp.fr:/dev/sdb
>>>> 2023-05-12T15:19:58.756639+0000
>>>> mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1877 :
>>> cephadm
>>>> [ERR] Device path '/dev/sdb' not found on host
>>>> 'mostha1.legi.grenoble-inp.fr
>>> <http://mostha1.legi.grenoble-inp.fr>'
>>>> Traceback (most recent call last):
>>>> File "/usr/share/ceph/mgr/orchestrator/_interface.py",
>>> line 125,
>>>> in wrapper
>>>> return OrchResult(f(*args, **kwargs))
>>>> File "/usr/share/ceph/mgr/cephadm/module.py", line 2275, in
>>>> zap_device
>>>> f"Device path '{path}' not found on host
'{host}'")
>>>> orchestrator._interface.OrchestratorError: Device path
>>> '/dev/sdb'
>>>> not found on host 'mostha1.legi.grenoble-inp.fr
>>> <http://mostha1.legi.grenoble-inp.fr>'
>>>> ....
>>>>
>>>> [ceph: root@mostha1 /]# ls -l /dev/sdb
>>>>
>>>> brw-rw---- 1 root disk 8, 16 May 12 15:16 /dev/sdb
>>>>
>>>> [ceph: root@mostha1 /]# lsblk /dev/sdb
>>>>
>>>> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
>>>> sdb 8:16 1 465.8G 0 disk
>>>> `-sdb1 8:17 1 465.8G 0 part
>>>>
>>>> I have crated a full partition on /dev/sdb (for testing) and
>>> /dev/sdc
>>>> has no partition table (removed).
>>>>
>>>> But all seams fine with these commands.
>>>>
>>>> Patrick
>>>>
>>>> Le 12/05/2023 à 20:19, Beaman, Joshua a écrit :
>>>>
>>>> I don’t quite understand why that zap would not work. But,
>>> here’s
>>>> where I’d start.
>>>>
>>>> 1. cephadm check-host
>>>>
>>>> 1. Run this on each of your hosts to make sure cephadm,
>>>> podman and all other prerequisites are installed and
>>>> recognized
>>>>
>>>> 2. ceph orch ls
>>>>
>>>> 1. This should show at least a mon, mgr, and osd spec
>>> deployed
>>>>
>>>> 3. ceph orch ls osd –export
>>>>
>>>> 1. This will show the OSD placement service specifications
>>>> that orchestrator uses to identify devices to deploy
>>> as OSDs
>>>>
>>>> 4. ceph orch host ls
>>>>
>>>> 1. This will list the hosts that have been added to
>>>> orchestrator’s inventory, and what labels are applied
>>>> which correlate to the service placement labels
>>>>
>>>> 5. ceph log last cephadm
>>>>
>>>> 1. This will show you what orchestrator has been trying to
>>>> do, and how it may be failing
>>>>
>>>> Also, it’s never un-helpful to have a look at “ceph -s” and
>>> “ceph
>>>> health detail”, particularly for any people trying to help you
>>>> without access to your systems.
>>>>
>>>> Best of luck,
>>>>
>>>> Josh Beaman
>>>>
>>>> *From: *Patrick Begou <Patrick.Begou(a)univ-grenoble-alpes.fr>
>>>> <mailto:Patrick.Begou@univ-grenoble-alpes.fr>
>>>> *Date: *Friday, May 12, 2023 at 10:45 AM
>>>> *To: *ceph-users <ceph-users(a)ceph.io>
>>> <mailto:ceph-users@ceph.io>
>>>> *Subject: *[EXTERNAL] [ceph-users] [Pacific] ceph orch device ls
>>>> do not returns any HDD
>>>>
>>>> Hi everyone
>>>>
>>>> I'm new to CEPH, just a french 4 days training session with
>>>> Octopus on
>>>> VMs that convince me to build my first cluster.
>>>>
>>>> At this time I have 4 old identical nodes for testing with 3
>>> HDDs
>>>> each,
>>>> 2 network interfaces and running Alma Linux8 (el8). I try to
>>>> replay the
>>>> training session but it fails, breaking the web interface
>>> because of
>>>> some problems with podman 4.2 not compatible with Octopus.
>>>>
>>>> So I try to deploy Pacific with cephadm tool on my first node
>>>> (mostha1)
>>>> (to enable testing also an upgrade later).
>>>>
>>>> dnf -y install
>>>
https://urldefense.com/v3/__https://download.ceph.com/rpm-16.2.13/el8/noarc…
>>>>
>>>
<https://urldefense.com/v3/__https:/download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noarch.rpm__;!!CQl3mcHX2A!H9cwNCJyKXYQ4BbGA3gwHHRitjOS4lBCZT9wlnBZ-8IDue0MvdcPD8Dnv5yQCZw_eA4BNDYaEq1eouKQcQO7HshgdUJ0SJ-EgLfaBGBmCQ$>
>>>>
>>>>
>>>> monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print
>>> $1 }')
>>>> cephadm bootstrap --mon-ip $monip
>>> --initial-dashboard-password
>>>> xxxxx \
>>>> --initial-dashboard-user admceph \
>>>> --allow-fqdn-hostname --cluster-network
>>>> 10.1.0.0/16 <http://10.1.0.0/16>
>>>>
>>>> This was sucessfull.
>>>>
>>>> But running "*c**eph orch device ls*" do not show any HDD
>>> even if
>>>> I have
>>>> /dev/sda (used by the OS), /dev/sdb and /dev/sdc
>>>>
>>>> The web interface shows a row capacity which is an aggregate
>>> of the
>>>> sizes of the 3 HDDs for the node.
>>>>
>>>> I've also tried to reset /dev/sdb but cephadm do not see it:
>>>>
>>>> [ceph: root@mostha1 /]# ceph orch device zap
>>>> mostha1.legi.grenoble-inp.fr
>>> <http://mostha1.legi.grenoble-inp.fr> /dev/sdb --force
>>>> Error EINVAL: Device path '/dev/sdb' not found on host
>>>> 'mostha1.legi.grenoble-inp.fr
>>> <http://mostha1.legi.grenoble-inp.fr>'
>>>>
>>>> On my first attempt with octopus, I was able to list the
>>> available
>>>> HDD
>>>> with this command line. Before moving to Pacific, the OS on
>>> this node
>>>> has been reinstalled from scratch.
>>>>
>>>> Any advices for a CEPH beginner ?
>>>>
>>>> Thanks
>>>>
>>>> Patrick
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
28 Jun
28 Jun
5:55 p.m.
New subject: [EXTERNAL] [Pacific] ceph orch device ls do not returns any HDD
Hi Patrick,
A follow-up on this. As I told you offline, we were about to start a new
cluster (with recent HW!). We did it this morning, using cephadm and
Quincy. It worked like a charm and all the devices were properly
discovered and configured as OSDs. So it seems to confirm something
weird in your OS configuration (we are using plain CentOS Stream) or a
firmware issue (unlikely) or a raid controller configuration where the
devices are not exported as JBOD but RAID volumes preventing them to be
seen (but I tend to remember that the devices were present at the OS
level)...
Cheers,
Michel
Le 26/05/2023 à 18:50, Michel Jouvin a écrit :
> Patrick,
>
> I can only say that I would not expect a specific problem due to your
> hardware. Upgrading the firmware is generally a good idea but I
> wouldn't expect it helps in your case if the osk (lsblk) sees the disk.
>
> As for starting with octopus I don't know if it will help... But we
> are also using the same os as you (centos stream in fact but basically
> the same). We have been running octopus (with cephadm) on this os
> version without problem and upgraded since then in pacific and quincy.
> In fact on of our cluster started in infernetis, the other in luminous
> and they have been upgraded without problems since then...
>
> Michel
> Sent from my mobile
> Le 26 mai 2023 18:34:22 Patrick Begou
> <Patrick.Begou(a)univ-grenoble-alpes.fr> a écrit :
>> Hi Michel,
>> I do not notice anything strange in the logs files (looking for
>> errors or warnings).
>> The hardware is a DELL C6100 sled (from 2011) running Alma Linux8
>> up-to-date. It uses 3 sata disks.
>> Is there a way to force osd installation by hand with providing the
>> device /dev/sdc for example ? A "do what I say" approach...
>> Is it a good try to deploy Octopus on the nodes, configure the osd
>> (even if podman 4.2.0 is not validated for Octopus) and then upgrade
>> to Pacific? Could this be a workaround for this sort of regression
>> from Octopus to Pacific ?
>> May be updating the BIOS from 1.7.1 to 1.8.1 ?
>>
>> All this is a little bit confusing for me as I'm trying to discover
>> Ceph 😁
>> Thanks
>> Patrick
>>
>> Le 26/05/2023 à 17:19, Michel Jouvin a écrit :
>>> Hi Patrick,
>>>
>>> It is weird, we have a couple of clusters with cephadm and running
>>> pacify or quincy and ceph orch device works well. Have you looked at
>>> the cephadm logs (ceph log last cephadm)?
>>>
>>> Except if you are using a very specific hardware, I suspect Ceph is
>>> suffering of a problem outside it...
>>>
>>> Cheers,
>>>
>>> Michel
>>> Sent from my mobile
>>>
>>> Le 26 mai 2023 17:02:50 Patrick Begou
>>> <Patrick.Begou(a)univ-grenoble-alpes.fr> a écrit :
>>>
>>>> Hi,
>>>>
>>>> I'm back working on this problem.
>>>>
>>>> First of all, I saw that I had a hardware memory error so I had to
>>>> solve
>>>> this first. It's done.
>>>>
>>>> I've tested some different Ceph deployments, each time starting with
a
>>>> full OS re-install (it requires some time for each test).
>>>>
>>>> Using Octopus, the devices are found:
>>>>
>>>> dnf -y install \
>>>>
https://download.ceph.com/rpm-15.2.12/el8/noarch/cephadm-15.2.12-0.el8.noar…
>>>>
>>>> monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }'))
>>>> cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \
>>>> --allow-fqdn-hostname
>>>>
>>>> [ceph: root@mostha1 /]# *ceph orch device ls*
>>>> Hostname Path Type Serial Size Health
>>>> Ident Fault Available
>>>> mostha1.legi.grenoble-inp.fr /dev/sda hdd S2B5J90ZA02494 250G
>>>> Unknown N/A N/A Yes
>>>> mostha1.legi.grenoble-inp.fr /dev/sdc hdd WD-WMAYP0982329 500G
>>>> Unknown N/A N/A Yes
>>>>
>>>>
>>>> But with Pacific or Quincy the command returns nothing.
>>>>
>>>> With Pacific:
>>>>
>>>> dnf -y install \
>>>>
https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noar…
>>>>
>>>> monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }')
>>>> cephadm bootstrap --mon-ip $monip --initial-dashboard-password xxxxx \
>>>> --allow-fqdn-hostname
>>>>
>>>>
>>>> "ceph orch device ls" doesn't return anything but
"cephadm shell
>>>> lsmcli
>>>> ldl" list all the devices.
>>>>
>>>> [ceph: root@mostha1 /]# *ceph orch device ls --wide*
>>>> [ceph: root@mostha1 /]# *lsblk*
>>>> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
>>>> sda 8:0 1 232.9G 0 disk
>>>> |-sda1 8:1 1 3.9G 0 part /rootfs/boot
>>>> |-sda2 8:2 1 78.1G 0 part
>>>> | `-osvg-rootvol 253:0 0 48.8G 0 lvm /rootfs
>>>> |-sda3 8:3 1 3.9G 0 part [SWAP]
>>>> `-sda4 8:4 1 146.9G 0 part
>>>> |-secretvg-homevol 253:1 0 9.8G 0 lvm /rootfs/home
>>>> |-secretvg-tmpvol 253:2 0 9.8G 0 lvm /rootfs/tmp
>>>> `-secretvg-varvol 253:3 0 9.8G 0 lvm /rootfs/var
>>>> sdb 8:16 1 232.9G 0 disk
>>>> sdc 8:32 1 465.8G 0 disk
>>>> [ceph: root@mostha1 /]# exit
>>>> [root@mostha1 ~]# *cephadm ceph-volume inventory*
>>>> Inferring fsid 2e3e85a8-fbcf-11ed-84e5-00266cf8869c
>>>> Using ceph image with id '0dc91bca92c2' and tag 'v17'
created on
>>>> 2023-05-25 16:26:31 +0000 UTC
>>>>
quay.io/ceph/ceph@sha256:b8df01a568f4dec7bac6d5040f9391dcca14e00ec7f4de8a3dcf3f2a6502d3a9
>>>>
>>>>
>>>> Device Path Size Device nodes rotates
>>>> available Model name
>>>>
>>>> [root@mostha1 ~]# *cephadm shell lsmcli ldl*
>>>> Inferring fsid 4d54823c-fb05-11ed-aecf-00266cf8869c
>>>> Inferring config
>>>> /var/lib/ceph/4d54823c-fb05-11ed-aecf-00266cf8869c/mon.mostha1/config
>>>> Using ceph image with id 'c9a1062f7289' and tag 'v17'
created on
>>>> 2023-04-25 16:04:33 +0000 UTC
>>>>
quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c64ca62a0b38ab362e614ad671efa4a0547e
>>>>
>>>> Path | SCSI VPD 0x83 | Link Type | Serial Number | Health
>>>> Status
>>>> -------------------------------------------------------------------------
>>>>
>>>> */dev/sda | 50024e92039e4f1c | PATA/SATA | S2B5J90ZA10142 | Good**
>>>> **/dev/sdc | 50014ee0ad5953c9 | PATA/SATA | WD-WMAYP0982329 | Good**
>>>> **/dev/sdb | 50024e920387fa2c | PATA/SATA | S2B5J90ZA02494 | Good**
>>>> *
>>>>
>>>>
>>>> Could it be a bug in ceph-volume ?
>>>> Adam suggest looking to the underlying commands (lsblk, blkid,
>>>> udevadm,
>>>> lvs, or pvs) but I'm not very comfortable with blkid and udevadm. Is
>>>> there a "debug flag" to set ceph more verbose ?
>>>>
>>>> Thanks
>>>>
>>>> Patrick
>>>>
>>>> Le 15/05/2023 à 21:20, Adam King a écrit :
>>>>> As you've already seem to have figured out, "ceph orch
device ls" is
>>>>> populated with the results from "ceph-volume inventory". My
best
>>>>> guess
>>>>> to try and debug this would be to manually run "cephadm
>>>>> ceph-volume --
>>>>> inventory" (the same as "cephadm ceph-volume
inventory", I just like
>>>>> to separate the ceph-volume command from cephadm itself with the
" --
>>>>> ") and then check /var/log/ceph/<fsid>/ceph-volume.log
from when you
>>>>> ran the command onward to try and see why it isn't seeing your
>>>>> devices. For example I can see a line like
>>>>>
>>>>> [2023-05-15 19:11:58,048][ceph_volume.main][INFO ] Running command:
>>>>> ceph-volume inventory
>>>>>
>>>>> in there. Then if I look onward from there I can see it ran things
>>>>> like
>>>>>
>>>>> lsblk -P -o
>>>>>
NAME,KNAME,PKNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL
>>>>>
>>>>>
>>>>> as part of getting my device list. So if I was having issues I would
>>>>> try running that directly and see what I got. Will note that
>>>>> ceph-volume on certain more recent versions (not sure about octopus)
>>>>> runs commands through nsenter, so you'd have to look past that
>>>>> part in
>>>>> the log lines to the underlying command being used, typically
>>>>> something with lsblk, blkid, udevadm, lvs, or pvs.
>>>>>
>>>>> Also, if you want to see if it's an issue with a certain version
of
>>>>> ceph-volume, you can use different versions by passing the image
flag
>>>>> to cephadm. E.g.
>>>>>
>>>>> cephadm --image quay.io/ceph/ceph:v17.2.6
>>>>> <http://quay.io/ceph/ceph:v17.2.6> ceph-volume -- inventory
>>>>>
>>>>> would use the 17.2.6 version of ceph-volume for the inventory. It
>>>>> works by running ceph-volume through the container, so you don't
have
>>>>> to have to worry about installing different packages to try them and
>>>>> it should pull the container image on its own if it isn't on the
>>>>> machine already (but note that means the command will take longer as
>>>>> it pulls the image the first time).
>>>>>
>>>>>
>>>>>
>>>>> On Sat, May 13, 2023 at 4:34 AM Patrick Begou
>>>>> <Patrick.Begou(a)univ-grenoble-alpes.fr> wrote:
>>>>>
>>>>> Hi Joshua,
>>>>>
>>>>> I've tried these commands but it looks like CEPH is unable to see
and
>>>>> configure these HDDs.
>>>>> [root@mostha1 ~]# cephadm ceph-volume inventory
>>>>>
>>>>> Inferring fsid 4b7a6504-f0be-11ed-be1a-00266cf8869c
>>>>> Using recent ceph image
>>>>>
quay.io/ceph/ceph@sha256:e6919776f0ff8331a8e9c4b18d36c5e9eed31e1a80da62ae8454e42d10e95544
>>>>>
>>>>>
<http://quay.io/ceph/ceph@sha256:e6919776f0ff8331a8e9c4b18d36c5e9eed31e1a80da62ae8454e42d10e95544>
>>>>>
>>>>>
>>>>> Device Path Size Device nodes rotates
>>>>> available Model name
>>>>>
>>>>> [root@mostha1 ~]# cephadm shell
>>>>>
>>>>> [ceph: root@mostha1 /]# ceph orch apply osd --all-available-devices
>>>>>
>>>>> Scheduled osd.all-available-devices update...
>>>>>
>>>>> [ceph: root@mostha1 /]# ceph orch device ls[ceph: root@mostha1 /]#
>>>>> ceph-volume lvm zap /dev/sdb
>>>>>
>>>>> --> Zapping: /dev/sdb
>>>>> --> --destroy was not specified, but zapping a whole device will
>>>>> remove the partition table
>>>>> Running command: /usr/bin/dd if=/dev/zero of=/dev/sdb bs=1M
>>>>> count=10
>>>>> conv=fsync
>>>>> stderr: 10+0 records in
>>>>> 10+0 records out
>>>>> 10485760 bytes (10 MB, 10 MiB) copied, 0.10039 s, 104 MB/s
>>>>> --> Zapping successful for: <Raw Device: /dev/sdb>
>>>>>
>>>>> I can check that /dev/sdb1 has been erased, so previous command is
>>>>> successful
>>>>> [ceph: root@mostha1 ceph]# lsblk
>>>>> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
>>>>> sda 8:0 1 232.9G 0 disk
>>>>> |-sda1 8:1 1 3.9G 0 part /rootfs/boot
>>>>> |-sda2 8:2 1 78.1G 0 part
>>>>> | `-osvg-rootvol 253:0 0 48.8G 0 lvm /rootfs
>>>>> |-sda3 8:3 1 3.9G 0 part [SWAP]
>>>>> `-sda4 8:4 1 146.9G 0 part
>>>>> |-secretvg-homevol 253:1 0 9.8G 0 lvm /rootfs/home
>>>>> |-secretvg-tmpvol 253:2 0 9.8G 0 lvm /rootfs/tmp
>>>>> `-secretvg-varvol 253:3 0 9.8G 0 lvm /rootfs/var
>>>>> sdb 8:16 1 465.8G 0 disk
>>>>> sdc 8:32 1 232.9G 0 disk
>>>>>
>>>>> But still no visible HDD:
>>>>>
>>>>> [ceph: root@mostha1 ceph]# ceph orch apply osd
>>>>> --all-available-devices
>>>>>
>>>>> Scheduled osd.all-available-devices update...
>>>>>
>>>>> [ceph: root@mostha1 ceph]# ceph orch device ls
>>>>> [ceph: root@mostha1 ceph]#
>>>>>
>>>>> May be I have done something bad at install time as in the container
>>>>> I've unintentionally run:
>>>>>
>>>>> dnf -y install
>>>>>
https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noar…
>>>>>
>>>>>
>>>>> (an awful copy/paste launching the command). Can this break The
>>>>> container ? I do not know what should be available as ceph
>>>>> packages in
>>>>> the container to remove properly this install (no dnf.log file in
the
>>>>> container)
>>>>>
>>>>> Patrick
>>>>>
>>>>>
>>>>> Le 12/05/2023 à 21:38, Beaman, Joshua a écrit :
>>>>>> The most significant point I see there, is you have no OSD
service
>>>>>> spec to tell orchestrator how to deploy OSDs. The easiest fix
for
>>>>>> that would be “cephorchapplyosd--all-available-devices”
>>>>>>
>>>>>> This will create a simple spec that should work for a test
>>>>>> environment. Most likely it will collocate the block, block.db,
>>>>> and
>>>>>> WAL all on the same device. Not ideal for prod environments,
>>>>> but fine
>>>>>> for practice and testing.
>>>>>>
>>>>>> The other command I should have had you try is “cephadm
ceph-volume
>>>>>> inventory”. That should show you the devices available for OSD
>>>>>> deployment, and hopefully matches up to what your “lsblk”
>>>>> shows. If
>>>>>> you need to zap HDDs and orchestrator is still not seeing them,
you
>>>>>> can try “cephadm ceph-volume lvm zap /dev/sdb”
>>>>>>
>>>>>> Thank you,
>>>>>>
>>>>>> Josh Beaman
>>>>>>
>>>>>> *From: *Patrick Begou
<Patrick.Begou(a)univ-grenoble-alpes.fr>
>>>>>> *Date: *Friday, May 12, 2023 at 2:22 PM
>>>>>> *To: *Beaman, Joshua <Joshua_Beaman(a)comcast.com>om>,
ceph-users
>>>>>> <ceph-users(a)ceph.io>
>>>>>> *Subject: *Re: [EXTERNAL] [ceph-users] [Pacific] ceph orch
>>>>> device ls
>>>>>> do not returns any HDD
>>>>>>
>>>>>> Hi Joshua and thanks for this quick reply.
>>>>>>
>>>>>> At this step I have only one node. I was checking what ceph was
>>>>>> returning with different commands on this host before adding new
>>>>>> hosts. Just to compare with my first Octopus install. As this
>>>>> hardware
>>>>>> is for testing only, it remains easy for me to break everything
and
>>>>>> reinstall again.
>>>>>>
>>>>>> [root@mostha1 ~]# cephadm check-host
>>>>>>
>>>>>> podman (/usr/bin/podman) version 4.2.0 is present
>>>>>> systemctl is present
>>>>>> lvcreate is present
>>>>>> Unit chronyd.service is enabled and running
>>>>>> Host looks OK
>>>>>>
>>>>>> [ceph: root@mostha1 /]# ceph -s
>>>>>>
>>>>>> cluster:
>>>>>> id: 4b7a6504-f0be-11ed-be1a-00266cf8869c
>>>>>> health: HEALTH_WARN
>>>>>> OSD count 0 < osd_pool_default_size 3
>>>>>>
>>>>>> services:
>>>>>> mon: 1 daemons, quorum mostha1.legi.grenoble-inp.fr
>>>>> <http://mostha1.legi.grenoble-inp.fr> (age 5h)
>>>>>> mgr: mostha1.legi.grenoble-inp.fr
>>>>> <http://mostha1.legi.grenoble-inp.fr>.hogwuz(active, since 5h)
>>>>>> osd: 0 osds: 0 up, 0 in
>>>>>>
>>>>>> data:
>>>>>> pools: 0 pools, 0 pgs
>>>>>> objects: 0 objects, 0 B
>>>>>> usage: 0 B used, 0 B / 0 B avail
>>>>>> pgs:
>>>>>>
>>>>>> [ceph: root@mostha1 /]# ceph orch ls
>>>>>>
>>>>>> NAME PORTS RUNNING REFRESHED AGE PLACEMENT
>>>>>> alertmanager ?:9093,9094 1/1 6m ago 6h count:1
>>>>>> crash 1/1 6m ago 6h *
>>>>>> grafana ?:3000 1/1 6m ago 6h count:1
>>>>>> mgr 1/2 6m ago 6h count:2
>>>>>> mon 1/5 6m ago 6h count:5
>>>>>> node-exporter ?:9100 1/1 6m ago 6h *
>>>>>> prometheus ?:9095 1/1 6m ago 6h count:1
>>>>>>
>>>>>> [ceph: root@mostha1 /]# ceph orch ls osd -export
>>>>>>
>>>>>> No services reported
>>>>>>
>>>>>> [ceph: root@mostha1 /]# ceph orch host ls
>>>>>>
>>>>>> HOST ADDR LABELS STATUS
>>>>>> mostha1.legi.grenoble-inp.fr
>>>>> <http://mostha1.legi.grenoble-inp.fr> 194.254.66.34 _admin
>>>>>> 1 hosts in cluster
>>>>>>
>>>>>> [ceph: root@mostha1 /]# ceph log last cephadm
>>>>>>
>>>>>> ...
>>>>>> 2023-05-12T15:19:58.754655+0000
>>>>>> mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1876 :
>>>>> cephadm
>>>>>> [INF] Zap device mostha1.legi.grenoble-inp.fr:/dev/sdb
>>>>>> 2023-05-12T15:19:58.756639+0000
>>>>>> mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1877 :
>>>>> cephadm
>>>>>> [ERR] Device path '/dev/sdb' not found on host
>>>>>> 'mostha1.legi.grenoble-inp.fr
>>>>> <http://mostha1.legi.grenoble-inp.fr>'
>>>>>> Traceback (most recent call last):
>>>>>> File "/usr/share/ceph/mgr/orchestrator/_interface.py",
>>>>> line 125,
>>>>>> in wrapper
>>>>>> return OrchResult(f(*args, **kwargs))
>>>>>> File "/usr/share/ceph/mgr/cephadm/module.py", line
2275, in
>>>>>> zap_device
>>>>>> f"Device path '{path}' not found on host
'{host}'")
>>>>>> orchestrator._interface.OrchestratorError: Device path
>>>>> '/dev/sdb'
>>>>>> not found on host 'mostha1.legi.grenoble-inp.fr
>>>>> <http://mostha1.legi.grenoble-inp.fr>'
>>>>>> ....
>>>>>>
>>>>>> [ceph: root@mostha1 /]# ls -l /dev/sdb
>>>>>>
>>>>>> brw-rw---- 1 root disk 8, 16 May 12 15:16 /dev/sdb
>>>>>>
>>>>>> [ceph: root@mostha1 /]# lsblk /dev/sdb
>>>>>>
>>>>>> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
>>>>>> sdb 8:16 1 465.8G 0 disk
>>>>>> `-sdb1 8:17 1 465.8G 0 part
>>>>>>
>>>>>> I have crated a full partition on /dev/sdb (for testing) and
>>>>> /dev/sdc
>>>>>> has no partition table (removed).
>>>>>>
>>>>>> But all seams fine with these commands.
>>>>>>
>>>>>> Patrick
>>>>>>
>>>>>> Le 12/05/2023 à 20:19, Beaman, Joshua a écrit :
>>>>>>
>>>>>> I don’t quite understand why that zap would not work. But,
>>>>> here’s
>>>>>> where I’d start.
>>>>>>
>>>>>> 1. cephadm check-host
>>>>>>
>>>>>> 1. Run this on each of your hosts to make sure cephadm,
>>>>>> podman and all other prerequisites are installed and
>>>>>> recognized
>>>>>>
>>>>>> 2. ceph orch ls
>>>>>>
>>>>>> 1. This should show at least a mon, mgr, and osd spec
>>>>> deployed
>>>>>>
>>>>>> 3. ceph orch ls osd –export
>>>>>>
>>>>>> 1. This will show the OSD placement service specifications
>>>>>> that orchestrator uses to identify devices to deploy
>>>>> as OSDs
>>>>>>
>>>>>> 4. ceph orch host ls
>>>>>>
>>>>>> 1. This will list the hosts that have been added to
>>>>>> orchestrator’s inventory, and what labels are applied
>>>>>> which correlate to the service placement labels
>>>>>>
>>>>>> 5. ceph log last cephadm
>>>>>>
>>>>>> 1. This will show you what orchestrator has been trying to
>>>>>> do, and how it may be failing
>>>>>>
>>>>>> Also, it’s never un-helpful to have a look at “ceph -s” and
>>>>> “ceph
>>>>>> health detail”, particularly for any people trying to help you
>>>>>> without access to your systems.
>>>>>>
>>>>>> Best of luck,
>>>>>>
>>>>>> Josh Beaman
>>>>>>
>>>>>> *From: *Patrick Begou
<Patrick.Begou(a)univ-grenoble-alpes.fr>
>>>>>> <mailto:Patrick.Begou@univ-grenoble-alpes.fr>
>>>>>> *Date: *Friday, May 12, 2023 at 10:45 AM
>>>>>> *To: *ceph-users <ceph-users(a)ceph.io>
>>>>> <mailto:ceph-users@ceph.io>
>>>>>> *Subject: *[EXTERNAL] [ceph-users] [Pacific] ceph orch device ls
>>>>>> do not returns any HDD
>>>>>>
>>>>>> Hi everyone
>>>>>>
>>>>>> I'm new to CEPH, just a french 4 days training session with
>>>>>> Octopus on
>>>>>> VMs that convince me to build my first cluster.
>>>>>>
>>>>>> At this time I have 4 old identical nodes for testing with 3
>>>>> HDDs
>>>>>> each,
>>>>>> 2 network interfaces and running Alma Linux8 (el8). I try to
>>>>>> replay the
>>>>>> training session but it fails, breaking the web interface
>>>>> because of
>>>>>> some problems with podman 4.2 not compatible with Octopus.
>>>>>>
>>>>>> So I try to deploy Pacific with cephadm tool on my first node
>>>>>> (mostha1)
>>>>>> (to enable testing also an upgrade later).
>>>>>>
>>>>>> dnf -y install
>>>>>
https://urldefense.com/v3/__https://download.ceph.com/rpm-16.2.13/el8/noarc…
>>>>>
>>>>>>
>>>>>
<https://urldefense.com/v3/__https:/download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noarch.rpm__;!!CQl3mcHX2A!H9cwNCJyKXYQ4BbGA3gwHHRitjOS4lBCZT9wlnBZ-8IDue0MvdcPD8Dnv5yQCZw_eA4BNDYaEq1eouKQcQO7HshgdUJ0SJ-EgLfaBGBmCQ$>
>>>>>
>>>>>>
>>>>>>
>>>>>> monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print
>>>>> $1 }')
>>>>>> cephadm bootstrap --mon-ip $monip
>>>>> --initial-dashboard-password
>>>>>> xxxxx \
>>>>>> --initial-dashboard-user admceph \
>>>>>> --allow-fqdn-hostname --cluster-network
>>>>>> 10.1.0.0/16 <http://10.1.0.0/16>
>>>>>>
>>>>>> This was sucessfull.
>>>>>>
>>>>>> But running "*c**eph orch device ls*" do not show any
HDD
>>>>> even if
>>>>>> I have
>>>>>> /dev/sda (used by the OS), /dev/sdb and /dev/sdc
>>>>>>
>>>>>> The web interface shows a row capacity which is an aggregate
>>>>> of the
>>>>>> sizes of the 3 HDDs for the node.
>>>>>>
>>>>>> I've also tried to reset /dev/sdb but cephadm do not see it:
>>>>>>
>>>>>> [ceph: root@mostha1 /]# ceph orch device zap
>>>>>> mostha1.legi.grenoble-inp.fr
>>>>> <http://mostha1.legi.grenoble-inp.fr> /dev/sdb --force
>>>>>> Error EINVAL: Device path '/dev/sdb' not found on host
>>>>>> 'mostha1.legi.grenoble-inp.fr
>>>>> <http://mostha1.legi.grenoble-inp.fr>'
>>>>>>
>>>>>> On my first attempt with octopus, I was able to list the
>>>>> available
>>>>>> HDD
>>>>>> with this command line. Before moving to Pacific, the OS on
>>>>> this node
>>>>>> has been reinstalled from scratch.
>>>>>>
>>>>>> Any advices for a CEPH beginner ?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Patrick
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>>> _______________________________________________
>>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
89
days inactive
136
days old
11 comments
4 participants
participants (4)
-
Adam King -
Beaman, Joshua -
Michel Jouvin -
Patrick Begou