Ceph OSD not mounting after reboot
Hi, Just rebooted one of my OSD servers after upgrading Ceph from 14.2.9 to 14.2.10 and it's OSDs won't come up. I find the following messages on my log: 4991 Jul 3 17:24:03 osdserver1-df ceph-osd[1272]: 2020-07-03 17:24:03.036 7fcc497f1c00 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 4992 Jul 3 17:24:03 osdserver1-df ceph-osd[1272]: 2020-07-03 17:24:03.036 7fcc497f1c00 -1 AuthRegistry(0x55e2ff810140) no keyring found at /var/lib/ceph/osd/ceph-6/keyring, disabling cephx and my /var/lib/ceph/osd/ceph-6 directory is empty. I see that on my other servers these /var/lib/ceph/osd/ceph-? directories are tmpfs mounts but I can't understand who is responsible for mounting them as there are no entries for them in /etc/fstab. How can I fix this osd server? Regards, Rodrigo Severo
So mount it, if it is empty -----Original Message----- To: ceph-users Subject: [ceph-users] Ceph OSD not mounting after reboot Hi, Just rebooted one of my OSD servers after upgrading Ceph from 14.2.9 to 14.2.10 and it's OSDs won't come up. I find the following messages on my log: 4991 Jul 3 17:24:03 osdserver1-df ceph-osd[1272]: 2020-07-03 17:24:03.036 7fcc497f1c00 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 4992 Jul 3 17:24:03 osdserver1-df ceph-osd[1272]: 2020-07-03 17:24:03.036 7fcc497f1c00 -1 AuthRegistry(0x55e2ff810140) no keyring found at /var/lib/ceph/osd/ceph-6/keyring, disabling cephx and my /var/lib/ceph/osd/ceph-6 directory is empty. I see that on my other servers these /var/lib/ceph/osd/ceph-? directories are tmpfs mounts but I can't understand who is responsible for mounting them as there are no entries for them in /etc/fstab. How can I fix this osd server? Regards, Rodrigo Severo _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Sat, Jul 4, 2020 at 1:37 AM Rodrigo Severo - Fábrica <rodrigo@fabricadeideias.com> wrote:
Hi,
Just rebooted one of my OSD servers after upgrading Ceph from 14.2.9 to 14.2.10 and it's OSDs won't come up.
I find the following messages on my log:
4991 Jul 3 17:24:03 osdserver1-df ceph-osd[1272]: 2020-07-03 17:24:03.036 7fcc497f1c00 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 4992 Jul 3 17:24:03 osdserver1-df ceph-osd[1272]: 2020-07-03 17:24:03.036 7fcc497f1c00 -1 AuthRegistry(0x55e2ff810140) no keyring found at /var/lib/ceph/osd/ceph-6/keyring, disabling cephx
and my /var/lib/ceph/osd/ceph-6 directory is empty.
I see that on my other servers these /var/lib/ceph/osd/ceph-? directories are tmpfs mounts but I can't understand who is responsible for mounting them as there are no entries for them in /etc/fstab.
How can I fix this osd server?
Hi, it is not possible to figure this out based on just the information that you provided. E.g., how was the OSD initially provisioned? Was it with "ceph-volume lvm"? In any case, the following commands (please run as root) would help debugging: lsblk lvs -a -o name,lv_tags -- Alexander E. Patrakov CV: http://pc.cd/PLz7
On Debian I have had the experience that one of the OSD-related systemd unit files was not installed - sorry that I can't recall which one, but the error you describe is the same one I had. The required file was in the source package, but was apparently left off the list of files to be copied into the install package. Once I copied the file over to /lib/systemd/system everything started working again. If I had to guess it was either ceph-volume@.service, or more likely - based on timestamps on my one of my OSD servers, ceph-osd@.service. Hope this helps. -Dave Dave Hall Binghamton University kdhall@binghamton.edu On 7/3/2020 4:58 PM, Alexander E. Patrakov wrote:
Hi,
Just rebooted one of my OSD servers after upgrading Ceph from 14.2.9 to 14.2.10 and it's OSDs won't come up.
I find the following messages on my log:
4991 Jul 3 17:24:03 osdserver1-df ceph-osd[1272]: 2020-07-03 17:24:03.036 7fcc497f1c00 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 4992 Jul 3 17:24:03 osdserver1-df ceph-osd[1272]: 2020-07-03 17:24:03.036 7fcc497f1c00 -1 AuthRegistry(0x55e2ff810140) no keyring found at /var/lib/ceph/osd/ceph-6/keyring, disabling cephx
and my /var/lib/ceph/osd/ceph-6 directory is empty.
I see that on my other servers these /var/lib/ceph/osd/ceph-? directories are tmpfs mounts but I can't understand who is responsible for mounting them as there are no entries for them in /etc/fstab.
How can I fix this osd server? Hi, it is not possible to figure this out based on just the information that you provided. E.g., how was the OSD initially
On Sat, Jul 4, 2020 at 1:37 AM Rodrigo Severo - Fábrica <rodrigo@fabricadeideias.com> wrote: provisioned? Was it with "ceph-volume lvm"?
In any case, the following commands (please run as root) would help debugging:
lsblk lvs -a -o name,lv_tags
Em sex., 3 de jul. de 2020 às 17:58, Alexander E. Patrakov <patrakov@gmail.com> escreveu:
On Sat, Jul 4, 2020 at 1:37 AM Rodrigo Severo - Fábrica <rodrigo@fabricadeideias.com> wrote:
Hi,
Just rebooted one of my OSD servers after upgrading Ceph from 14.2.9 to 14.2.10 and it's OSDs won't come up.
I find the following messages on my log:
4991 Jul 3 17:24:03 osdserver1-df ceph-osd[1272]: 2020-07-03 17:24:03.036 7fcc497f1c00 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 4992 Jul 3 17:24:03 osdserver1-df ceph-osd[1272]: 2020-07-03 17:24:03.036 7fcc497f1c00 -1 AuthRegistry(0x55e2ff810140) no keyring found at /var/lib/ceph/osd/ceph-6/keyring, disabling cephx
and my /var/lib/ceph/osd/ceph-6 directory is empty.
I see that on my other servers these /var/lib/ceph/osd/ceph-? directories are tmpfs mounts but I can't understand who is responsible for mounting them as there are no entries for them in /etc/fstab.
How can I fix this osd server?
Hi, it is not possible to figure this out based on just the information that you provided. E.g., how was the OSD initially provisioned? Was it with "ceph-volume lvm"?
I used ceph-deploy. Each osd is a lvm in a different disk.
In any case, the following commands (please run as root) would help debugging:
lsblk lvs -a -o name,lv_tags
The output of the above commands: root@osdserver1-df:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 223.6G 0 disk └─sda1 8:1 0 223.6G 0 part ├─vg-root 253:2 0 50G 0 lvm / ├─vg-swap_1 253:3 0 8G 0 lvm [SWAP] ├─vg-home 253:4 0 10G 0 lvm /home ├─vg-opt 253:5 0 10G 0 lvm /opt ├─vg-var 253:6 0 30G 0 lvm /var └─vg-tmp 253:7 0 10G 0 lvm /tmp sdb 8:16 0 3.7T 0 disk └─sdb1 8:17 0 3.7T 0 part └─vg_ceph_slow_8-lv_ceph_slow_8 253:1 0 3.7T 0 lvm sdc 8:32 0 1.8T 0 disk └─sdc1 8:33 0 1.8T 0 part └─ceph--bfcfde03--3a62--41ed--a037--d8aaf030a6d8-osd--block--445c4224--5087--4a28--a1ee--f23c5768207c 253:0 0 1.8T 0 lvm root@osdserver1-df:~# lvs -a -o name,lv_tags LV LV Tags osd-block-445c4224-5087-4a28-a1ee-f23c5768207c ceph.block_device=/dev/ceph-bfcfde03-3a62-41ed-a037-d8aaf030a6d8/osd-block-445c4224-5087-4a28-a1ee-f23c5768207c,ceph.block_uuid=NzwNAF-zVwO-t1sh-OMZa-yFs1-QLWs-GZUvUT,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=e348b63c-d239-4a15-a2ce-32f29a00431c,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=0,ceph.osd_fsid=445c4224-5087-4a28-a1ee-f23c5768207c,ceph.osd_id=6,ceph.type=block,ceph.vdo=0 home opt root swap_1 tmp var lv_ceph_slow_8 ceph.block_device=/dev/vg_ceph_slow_8/lv_ceph_slow_8,ceph.block_uuid=yrS8R7-N9rN-r0Jx-JoOU-Rapz-uoKJ-2FI1p4,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=e348b63c-d239-4a15-a2ce-32f29a00431c,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=0,ceph.osd_fsid=45b3c8ed-3852-4f1c-84b3-7d8a7cb129cb,ceph.osd_id=8,ceph.type=block,ceph.vdo=0 root@osdserver1-df:~# Any ideas on how to further debug this issue? Regards, Rodrigo Severo
participants (4)
-
Alexander E. Patrakov
-
Dave Hall
-
Marc Roos
-
Rodrigo Severo - Fábrica