Re: help with failed osds after reboot
Hi, which ceph release are you using? You mention ceph-disk so your OSDs are not LVM based, I assume? I've seen these messages a lot when testing in my virtual lab environment although I don't believe it's the cluster's fsid but the OSD's fsid that's in the error message (the OSDs have their own ID, too, take a look in /var/lib/ceph/osd/ceph-<ID>/fsid). When I did several re-installs of the whole cluster I had to make sure to properly wipe the disks but sometimes only a reboot did the trick. Of course, this is not an option in your situation. If your OSDs are systemd units check for orphaned units that need to be to disabled before restarting the correct ones. Did you re-deploy some of those disks? Regards, Eugen Zitat von Seth Duncan <Seth.Duncan2@bd.com>:
I had 5 of 10 osds fail on one of my nodes, after reboot the other 5 osds failed to start.
I have tried running ceph-disk activate-all and get back and error message about the cluster fsid not matching in /etc/ceph/ceph.conf
Has anyone experienced an issue such as this?
******************************************************************* IMPORTANT MESSAGE FOR RECIPIENTS IN THE U.S.A.: This message may constitute an advertisement of a BD group's products or services or a solicitation of interest in them. If this is such a message and you would like to opt out of receiving future advertisements or solicitations from this BD group, please forward this e-mail to optoutbygroup@bd.com. [BD.v1.0] ******************************************************************* This message (which includes any attachments) is intended only for the designated recipient(s). It may contain confidential or proprietary information and may be subject to the attorney-client privilege or other confidentiality protections. If you are not a designated recipient, you may not review, use, copy or distribute this message. If you received this in error, please notify the sender by reply e-mail and delete this message. Thank you. ******************************************************************* Corporate Headquarters Mailing Address: BD (Becton, Dickinson and Company) 1 Becton Drive Franklin Lakes, NJ 07417 U.S.A. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Maybe you have the same issue? https://tracker.ceph.com/issues/44102#change-167531 In my case an update(?) disabled osd runlevels. systemctl is-enabled ceph-osd@0 -----Original Message----- To: ceph-users@ceph.io Subject: [ceph-users] Re: help with failed osds after reboot Hi, which ceph release are you using? You mention ceph-disk so your OSDs are not LVM based, I assume? I've seen these messages a lot when testing in my virtual lab environment although I don't believe it's the cluster's fsid but the OSD's fsid that's in the error message (the OSDs have their own ID, too, take a look in /var/lib/ceph/osd/ceph-<ID>/fsid). When I did several re-installs of the whole cluster I had to make sure to properly wipe the disks but sometimes only a reboot did the trick. Of course, this is not an option in your situation. If your OSDs are systemd units check for orphaned units that need to be to disabled before restarting the correct ones. Did you re-deploy some of those disks? Regards, Eugen Zitat von Seth Duncan <Seth.Duncan2@bd.com>:
I had 5 of 10 osds fail on one of my nodes, after reboot the other 5 osds failed to start.
I have tried running ceph-disk activate-all and get back and error message about the cluster fsid not matching in /etc/ceph/ceph.conf
Has anyone experienced an issue such as this?
******************************************************************* IMPORTANT MESSAGE FOR RECIPIENTS IN THE U.S.A.: This message may constitute an advertisement of a BD group's products or services or a solicitation of interest in them. If this is such a message and you would like to opt out of receiving future advertisements or solicitations from this BD group, please forward this e-mail to optoutbygroup@bd.com. [BD.v1.0] ******************************************************************* This message (which includes any attachments) is intended only for the
designated recipient(s). It may contain confidential or proprietary information and may be subject to the attorney-client privilege or other confidentiality protections. If you are not a designated recipient, you may not review, use, copy or distribute this message. If you received this in error, please notify the sender by reply e-mail and delete this message. Thank you. ******************************************************************* Corporate Headquarters Mailing Address: BD (Becton, Dickinson and Company) 1 Becton Drive Franklin Lakes, NJ 07417 U.S.A. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ceph version 10.2.7 ceph.conf [global] fsid = 75d6dba9-2144-47b1-87ef-1fe21d3c58a8 mon_initial_members = chad, jesse, seth mon_host = 192.168.10.41,192.168.10.40,192.168.10.39 mon warn on legacy crush tunables = false auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public network = 192.168.10.0/23 cluster network = 192.168.9.0/24 #Choose reasonable numbers for number of replicas and placement groups. osd pool default size = 3 # Write an object 3 times osd pool default min size = 1 # Allow writing 1 copy in a degraded state osd pool default pg num = 1024 osd pool default pgp num = 1024 #Choose a reasonable crush leaf type #0 for a 1-node cluster. #1 for a multi node cluster in a single rack #2 for a multi node, multi chassis cluster with multiple hosts in a chassis #3 for a multi node cluster with hosts across racks, etc. osd crush chooseleaf type = 1 Output of systemctl --failed UNIT LOAD ACTIVE SUB DESCRIPTION ● ceph-disk@dev-sdb1.service loaded failed failed Ceph disk activation: /dev/sdb1 ● ceph-disk@dev-sdb2.service loaded failed failed Ceph disk activation: /dev/sdb2 ● ceph-disk@dev-sdb3.service loaded failed failed Ceph disk activation: /dev/sdb3 ● ceph-disk@dev-sdb4.service loaded failed failed Ceph disk activation: /dev/sdb4 ● ceph-disk@dev-sdb5.service loaded failed failed Ceph disk activation: /dev/sdb5 ● ceph-disk@dev-sdc1.service loaded failed failed Ceph disk activation: /dev/sdc1 ● ceph-disk@dev-sdc2.service loaded failed failed Ceph disk activation: /dev/sdc2 ● ceph-disk@dev-sdc3.service loaded failed failed Ceph disk activation: /dev/sdc3 ● ceph-disk@dev-sdc4.service loaded failed failed Ceph disk activation: /dev/sdc4 ● ceph-disk@dev-sdc5.service loaded failed failed Ceph disk activation: /dev/sdc5 ● ceph-disk@dev-sdd1.service loaded failed failed Ceph disk activation: /dev/sdd1 ● ceph-disk@dev-sde1.service loaded failed failed Ceph disk activation: /dev/sde1 ● ceph-disk@dev-sdf1.service loaded failed failed Ceph disk activation: /dev/sdf1 ● ceph-disk@dev-sdg1.service loaded failed failed Ceph disk activation: /dev/sdg1 ● ceph-disk@dev-sdh1.service loaded failed failed Ceph disk activation: /dev/sdh1 ● ceph-disk@dev-sdi1.service loaded failed failed Ceph disk activation: /dev/sdi1 ● ceph-disk@dev-sdj1.service loaded failed failed Ceph disk activation: /dev/sdj1 ● ceph-disk@dev-sdk1.service loaded failed failed Ceph disk activation: /dev/sdk1 ● ceph-disk@dev-sdl1.service loaded failed failed Ceph disk activation: /dev/sdl1 ● ceph-disk@dev-sdm1.service loaded failed failed Ceph disk activation: /dev/sdm1 ● ceph-osd@20.service loaded failed failed Ceph object storage daemon ● ceph-osd@21.service loaded failed failed Ceph object storage daemon ● ceph-osd@22.service loaded failed failed Ceph object storage daemon ● ceph-osd@23.service loaded failed failed Ceph object storage daemon ● ceph-osd@24.service loaded failed failed Ceph object storage daemon ● ceph-osd@25.service loaded failed failed Ceph object storage daemon ● ceph-osd@26.service loaded failed failed Ceph object storage daemon ● ceph-osd@27.service loaded failed failed Ceph object storage daemon ● ceph-osd@28.service loaded failed failed Ceph object storage daemon ● ceph-osd@29.service loaded failed failed Ceph object storage daemon ls -l /var/lib/ceph/osd drwxr-xr-x. 3 ceph ceph 228 Jun 11 14:32 ceph-20 drwxr-xr-x. 3 ceph ceph 228 Jun 11 14:32 ceph-21 drwxr-xr-x. 3 ceph ceph 228 Jun 11 14:32 ceph-22 drwxr-xr-x. 3 ceph ceph 228 Jun 11 14:32 ceph-23 drwxr-xr-x. 3 ceph ceph 228 Jun 11 14:32 ceph-24 drwxr-xr-x. 3 ceph ceph 228 Jun 11 14:35 ceph-25 drwxr-xr-x. 3 ceph ceph 228 Jun 11 14:32 ceph-26 drwxr-xr-x. 3 ceph ceph 228 Jun 11 14:32 ceph-27 drwxr-xr-x. 3 ceph ceph 228 Jun 11 14:32 ceph-28 drwxr-xr-x. 3 ceph ceph 228 Jun 11 14:32 ceph-29 and ls -la of ceph-20 drwxr-xr-x. 3 ceph ceph 228 Jun 11 14:32 . drwxr-x---. 12 ceph ceph 4096 Apr 10 2017 .. -rw-r--r--. 1 root root 479 Oct 10 2016 activate.monmap -rw-r--r--. 1 ceph ceph 3 Oct 10 2016 active -rw-r--r--. 1 ceph ceph 37 Oct 10 2016 ceph_fsid drwxr-xr-x. 332 ceph ceph 12288 Mar 5 17:10 current -rw-r--r--. 1 ceph ceph 37 Oct 10 2016 fsid lrwxrwxrwx. 1 ceph ceph 9 Oct 10 2016 journal -> /dev/sdb1 -rw-r--r--. 1 ceph ceph 37 Oct 10 2016 journal_uuid -rw-------. 1 ceph ceph 57 Oct 10 2016 keyring -rw-r--r--. 1 ceph ceph 21 Oct 10 2016 magic -rw-r--r--. 1 ceph ceph 6 Oct 10 2016 ready -rw-r--r--. 1 ceph ceph 4 Oct 10 2016 store_version -rw-r--r--. 1 ceph ceph 53 Oct 10 2016 superblock -rw-r--r--. 1 root root 0 Oct 10 2016 systemd -rw-r--r--. 1 ceph ceph 10 Oct 10 2016 type -rw-r--r--. 1 ceph ceph 3 Oct 10 2016 whoami and here is what is in /var/lib/ceph/osd-20/fsid 133b7d94-dce1-4965-b378-3b1b1c00f18f ^ does not match what is in /etc/ceph/ceph.conf ^^ is also not what is being reported when I run ceph-disk activate-all mount_activate: Failed to activate ceph-disk: Error: No cluster conf found in /etc/ceph with fsid e1d7b4ae-2dcd-40ee-bea5-d103fe1fa9c9 mount_activate: Failed to activate ceph-disk: Error: No cluster conf found in /etc/ceph with fsid e1d7b4ae-2dcd-40ee-bea5-d103fe1fa9c9 mount_activate: Failed to activate ceph-disk: Error: No cluster conf found in /etc/ceph with fsid e1d7b4ae-2dcd-40ee-bea5-d103fe1fa9c9 mount_activate: Failed to activate ceph-disk: Error: No cluster conf found in /etc/ceph with fsid e1d7b4ae-2dcd-40ee-bea5-d103fe1fa9c9 mount_activate: Failed to activate ceph-disk: Error: No cluster conf found in /etc/ceph with fsid e1d7b4ae-2dcd-40ee-bea5-d103fe1fa9c9 mount_activate: Failed to activate ceph-disk: Error: No cluster conf found in /etc/ceph with fsid e1d7b4ae-2dcd-40ee-bea5-d103fe1fa9c9 mount_activate: Failed to activate ceph-disk: Error: No cluster conf found in /etc/ceph with fsid e1d7b4ae-2dcd-40ee-bea5-d103fe1fa9c9 mount_activate: Failed to activate ceph-disk: Error: No cluster conf found in /etc/ceph with fsid e1d7b4ae-2dcd-40ee-bea5-d103fe1fa9c9 mount_activate: Failed to activate ceph-disk: Error: No cluster conf found in /etc/ceph with fsid e1d7b4ae-2dcd-40ee-bea5-d103fe1fa9c9 mount_activate: Failed to activate ceph-disk: Error: No cluster conf found in /etc/ceph with fsid e1d7b4ae-2dcd-40ee-bea5-d103fe1fa9c9 ceph-disk: Error: One or more partitions failed to activate I am trying to gather as many details as possible, is there anything I am missing that I should take a look at? I still have not figured out why this started being a problem or how to resolve.
On Mon, Jun 15, 2020 at 7:01 PM <seth.duncan2@bd.com> wrote:
Ceph version 10.2.7
ceph.conf [global] fsid = 75d6dba9-2144-47b1-87ef-1fe21d3c58a8
(...)
mount_activate: Failed to activate ceph-disk: Error: No cluster conf found in /etc/ceph with fsid e1d7b4ae-2dcd-40ee-bea5-d103fe1fa9c9
-- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90
mount_activate: Failed to activate ceph-disk: Error: No cluster conf found in /etc/ceph with fsid e1d7b4ae-2dcd-40ee-bea5-d103fe1fa9c9 mount_activate: Failed to activate ceph-disk: Error: No cluster conf found in /etc/ceph with fsid e1d7b4ae-2dcd-40ee-bea5-d103fe1fa9c9 mount_activate: Failed to activate ceph-disk: Error: No cluster conf found in /etc/ceph with fsid e1d7b4ae-2dcd-40ee-bea5-d103fe1fa9c9 mount_activate: Failed to activate ceph-disk: Error: No cluster conf found in /etc/ceph with fsid e1d7b4ae-2dcd-40ee-bea5-d103fe1fa9c9 mount_activate: Failed to activate ceph-disk: Error: No cluster conf found in /etc/ceph with fsid e1d7b4ae-2dcd-40ee-bea5-d103fe1fa9c9 mount_activate: Failed to activate ceph-disk: Error: No cluster conf found in /etc/ceph with fsid e1d7b4ae-2dcd-40ee-bea5-d103fe1fa9c9 mount_activate: Failed to activate ceph-disk: Error: No cluster conf found in /etc/ceph with fsid e1d7b4ae-2dcd-40ee-bea5-d103fe1fa9c9 mount_activate: Failed to activate ceph-disk: Error: No cluster conf found in /etc/ceph with fsid e1d7b4ae-2dcd-40ee-bea5-d103fe1fa9c9 mount_activate: Failed to activate ceph-disk: Error: No cluster conf found in /etc/ceph with fsid e1d7b4ae-2dcd-40ee-bea5-d103fe1fa9c9 ceph-disk: Error: One or more partitions failed to activate
I am trying to gather as many details as possible, is there anything I am missing that I should take a look at? I still have not figured out why this started being a problem or how to resolve. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Eugen Block
-
Marc Roos
-
Paul Emmerich
-
seth.duncan2＠bd.com