ceph octopus mysterious OSD crash
I've been banging on my ceph octopus test cluster for a few days now. 8 nodes. each node has 2 SSDs and 8 HDDs. They were all autoprovisioned so that each HDD gets an LVM slice of an SSD as a db partition. service_type: osd service_id: osd_spec_default placement: host_pattern: '*' data_devices: rotational: 1 db_devices: rotational: 0 things were going pretty good, until... yesterday.. i noticed TWO of the OSDs were "down". I went to check the logs, with journalctl -u ceph-xxxx@osd.xxx all it showed were a bunch of generic debug info, and the fact that it stopped. and various automatic attempts to restart. but no indication of what was wrong, and why the restarts KEEP failing. sample output: systemd[1]: Stopped Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00. systemd[1]: Starting Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00... bash[9340]: ceph-e51eb2fa-7f82-11eb-94d5-78e3b5148f00-osd.33-activate bash[9340]: WARNING: The same type, major and minor should not be used for multiple devices. bash[9340]: WARNING: The same type, major and minor should not be used for multiple devices. podman[9369]: 2021-03-07 16:00:15.543010794 -0800 PST m=+0.318475882 container create podman[9369]: 2021-03-07 16:00:15.73461926 -0800 PST m=+0.510084288 container init ..... bash[1611473]: --> ceph-volume lvm activate successful for osd ID: 33 podman[1611501]: 2021-03-18 10:23:02.564242824 -0700 PDT m=+1.379793448 container died bash[1611473]: ceph-xx-xx-xx-xx-osd.33 bash[1611473]: WARNING: The same type, major and minor should not be used for multiple devices. (repeat, repeat...) podman[1611615]: 2021-03-18 10:23:03.530992487 -0700 PDT m=+0.333130660 container create .... systemd[1]: Started Ceph osd.33 for xx-xx-xx-xx systemd[1]: ceph-xx-xx-xx-xx@osd.33.service: main process exited, code=exited, status=1/FAILURE bash[1611797]: ceph-xx-xx-xx-xx-osd.33-deactivate and eventually it just gives up. smartctl -a doesnt show any errors on the HDD dmesg doesnt show anything. So... what do I do? -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown@medata.com| www.medata.com
On 3/18/21 9:28 PM, Philip Brown wrote:
I've been banging on my ceph octopus test cluster for a few days now. 8 nodes. each node has 2 SSDs and 8 HDDs. They were all autoprovisioned so that each HDD gets an LVM slice of an SSD as a db partition.
service_type: osd service_id: osd_spec_default placement: host_pattern: '*' data_devices: rotational: 1 db_devices: rotational: 0
things were going pretty good, until... yesterday.. i noticed TWO of the OSDs were "down".
I went to check the logs, with journalctl -u ceph-xxxx@osd.xxx
all it showed were a bunch of generic debug info, and the fact that it stopped. and various automatic attempts to restart. but no indication of what was wrong, and why the restarts KEEP failing.
It's a deployment made with cephadm? Looks like it as I see podman messages. Are these all the log messages you can find on those OSDs? I.e. have you tried to gather logs with cephadm logs [1]. Gr. Stefan [1]: https://docs.ceph.com/en/latest/cephadm/troubleshooting/#gathering-log-files
yup cephadm and orch was used to set all this up. Current state of things: ceph osd tree shows 33 hdd 1.84698 osd.33 destroyed 0 1.00000 cephadm logs --name osd.33 --fsid xx-xx-xx-xx along with the systemctl stuff I already saw, showed me new things such as ceph-osd[1645438]: did not load config file, using default settings. ceph-osd[1645438]: 2021-03-18T14:31:32.990-0700 7f8bf14e3bc0 -1 parse_file: filesystem error: cannot get file size: No such file or directory This suggested to me that I needed to copy over /etc/ceph/ceph.conf to the OSD node. which I did. I then also copied over the admin key and generated a fresh bootstrap-osd key with it, just for good measure, with ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring I had saved the previous output of ceph-volume lvm list and on the OSD node, ran ceph-volume lvm prepare --data xxxx --block.db xxxx But it says osd is already prepared. I tried an activate... it tells me --> ceph-volume lvm activate successful for osd ID: 33 but now the cephadm logs output shows me ceph-osd[1677135]: 2021-03-18T17:57:47.982-0700 7ff64593f700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] Not the best error message :-} Now what do I need to do? ----- Original Message ----- From: "Stefan Kooman" <stefan@bit.nl> To: "Philip Brown" <pbrown@medata.com>, "ceph-users" <ceph-users@ceph.io> Sent: Thursday, March 18, 2021 2:04:09 PM Subject: Re: [ceph-users] ceph octopus mysterious OSD crash On 3/18/21 9:28 PM, Philip Brown wrote:
I've been banging on my ceph octopus test cluster for a few days now. 8 nodes. each node has 2 SSDs and 8 HDDs. They were all autoprovisioned so that each HDD gets an LVM slice of an SSD as a db partition.
service_type: osd service_id: osd_spec_default placement: host_pattern: '*' data_devices: rotational: 1 db_devices: rotational: 0
things were going pretty good, until... yesterday.. i noticed TWO of the OSDs were "down".
I went to check the logs, with journalctl -u ceph-xxxx@osd.xxx
all it showed were a bunch of generic debug info, and the fact that it stopped. and various automatic attempts to restart. but no indication of what was wrong, and why the restarts KEEP failing.
It's a deployment made with cephadm? Looks like it as I see podman messages. Are these all the log messages you can find on those OSDs? I.e. have you tried to gather logs with cephadm logs [1]. Gr. Stefan [1]: https://docs.ceph.com/en/latest/cephadm/troubleshooting/#gathering-log-files
On 3/19/21 2:20 AM, Philip Brown wrote:
yup cephadm and orch was used to set all this up.
Current state of things:
ceph osd tree shows
33 hdd 1.84698 osd.33 destroyed 0 1.00000
^^ Destroyed, ehh, this doesn't look good to me. Ceph thinks this OSD is destroyed. Do you know what might have happened to osd.33? Did you perform a "kill an OSD" while testing? AFAIK you can't fix that anymore. You will have to remove it and redploy it. Might even get a new osd.id.
cephadm logs --name osd.33 --fsid xx-xx-xx-xx
along with the systemctl stuff I already saw, showed me new things such as
ceph-osd[1645438]: did not load config file, using default settings.
ceph-osd[1645438]: 2021-03-18T14:31:32.990-0700 7f8bf14e3bc0 -1 parse_file: filesystem error: cannot get file size: No such file or directory
This suggested to me that I needed to copy over /etc/ceph/ceph.conf to the OSD node. which I did. I then also copied over the admin key and generated a fresh bootstrap-osd key with it, just for good measure, with ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring
I had saved the previous output of ceph-volume lvm list and on the OSD node, ran
ceph-volume lvm prepare --data xxxx --block.db xxxx
But it says osd is already prepared.
I tried an activate... it tells me
--> ceph-volume lvm activate successful for osd ID: 33
but now the cephadm logs output shows me
ceph-osd[1677135]: 2021-03-18T17:57:47.982-0700 7ff64593f700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
Not the best error message :-}
Indeed, would be nice to have a references to [2]. But I think why you get this is because of the destroyed OSD. I would use cephadm docu on how to replace an osd. Does that exist? We add a large thread about this "container" topic (see "[ceph-users] ceph-ansible in Pacific and beyond?").
Now what do I need to do?
I would remove osd.33. Even manually editing crushmaps if need to (should not be the case), and then redeploy this osd and wait for recovery. If you have not manually "destroyed" this osd than either things work differently in Octopus from things I have seen so far, my memory is failing me, or some really weird stuff is happening and I would really like to know what that is. Wat version are you running? Do note that 15.2.10 has been released. Gr. Stefan
mkay. Sooo... what's the new and nifty proper way to clean this up? The outsider's view is, "I should just be able to run 'ceph orch osd rm 33'" but that returns Unable to find OSDs: ['33'] ----- Original Message ----- From: "Stefan Kooman" <stefan@bit.nl> To: "Philip Brown" <pbrown@medata.com> Cc: "ceph-users" <ceph-users@ceph.io> Sent: Thursday, March 18, 2021 10:09:28 PM Subject: Re: [ceph-users] ceph octopus mysterious OSD crash On 3/19/21 2:20 AM, Philip Brown wrote:
yup cephadm and orch was used to set all this up.
Current state of things:
ceph osd tree shows
33 hdd 1.84698 osd.33 destroyed 0 1.00000
^^ Destroyed, ehh, this doesn't look good to me. Ceph thinks this OSD is destroyed. Do you know what might have happened to osd.33? Did you perform a "kill an OSD" while testing? AFAIK you can't fix that anymore. You will have to remove it and redploy it. Might even get a new osd.id.
On 3/19/21 3:53 PM, Philip Brown wrote:
mkay. Sooo... what's the new and nifty proper way to clean this up? The outsider's view is, "I should just be able to run 'ceph orch osd rm 33'"
Can you spawn a cephadm shell and run: ceph osd rm 33? And / or: ceph osd crush rm 33, or try to do it with cephadm. Does this work: ceph orch osd crush rm 33? Gr. Stefan P.s. I'll have to install an octopus release with cephadm to get myself up to speed here.
I made *some* progress for cleanup. I could already do "ceph osd rm 33" from my master. But doing the cleanup on the actual OSD node was problematical. ceph-volume lvm zap xxx wasnt working properly.. because the device wasnt fully released.... because at the regular OS level, it cant even SEE the VGs?? That caught me by surprise. But doing cephadm shell let me see the vgs, remove it, and thus have the zap work. so now we move on to reconstructing the hybrid OSD. First off, by default, the cephadm shell did not have permission to create OSDs. so I had to do [ceph: root@dxxxx /]# ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring Unfortunately, since I had run the lvm zap on both the data /dev/sdX, AND the db lv partition.. attempting to recreate the OSD with ceph-volume lvm prepare --data /dev/sdb --block.db /dev/ceph-xx-xx-xx/osd-db-xxxx (the original db lvm on SSD, which still technically existed) FAILED, because --> blkid could not detect a PARTUUID for device: /dev/ceph-xxxx/osd-xxx --> Was unable to complete a new OSD, will rollback changes cmon.... just MAKE one for me??? :-( Happily, i could grep for osd-db-specific-id-here in /var/log/ceph/ceph-volume.log and found the exact original lvcreate syntax to remake it. BUT.... lvm prepare once again complained about not detecting a PARTUUID. I think there may be a command to do that, that is left out of the ceph-volume.log :( So.. now what can I do? ----- Original Message ----- From: "Stefan Kooman" <stefan@bit.nl> To: "Philip Brown" <pbrown@medata.com> Cc: "ceph-users" <ceph-users@ceph.io> Sent: Friday, March 19, 2021 9:58:56 AM Subject: Re: [ceph-users] ceph octopus mysterious OSD crash On 3/19/21 3:53 PM, Philip Brown wrote:
mkay. Sooo... what's the new and nifty proper way to clean this up? The outsider's view is, "I should just be able to run 'ceph orch osd rm 33'"
Can you spawn a cephadm shell and run: ceph osd rm 33? And / or: ceph osd crush rm 33, or try to do it with cephadm. Does this work: ceph orch osd crush rm 33? Gr. Stefan P.s. I'll have to install an octopus release with cephadm to get myself up to speed here.
On 3/19/21 6:22 PM, Philip Brown wrote:
I made *some* progress for cleanup. I could already do "ceph osd rm 33" from my master. But doing the cleanup on the actual OSD node was problematical.
ceph-volume lvm zap xxx
wasnt working properly.. because the device wasnt fully released.... because at the regular OS level, it cant even SEE the VGs?? That caught me by surprise. But doing cephadm shell let me see the vgs, remove it, and thus have the zap work.
so now we move on to reconstructing the hybrid OSD. First off, by default, the cephadm shell did not have permission to create OSDs. so I had to do [ceph: root@dxxxx /]# ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring
Unfortunately, since I had run the lvm zap on both the data /dev/sdX, AND the db lv partition.. attempting to recreate the OSD with ceph-volume lvm prepare --data /dev/sdb --block.db /dev/ceph-xx-xx-xx/osd-db-xxxx
(the original db lvm on SSD, which still technically existed)
FAILED, because --> blkid could not detect a PARTUUID for device: /dev/ceph-xxxx/osd-xxx --> Was unable to complete a new OSD, will rollback changes
cmon.... just MAKE one for me???
:-(
Happily, i could grep for osd-db-specific-id-here in /var/log/ceph/ceph-volume.log and found the exact original lvcreate syntax to remake it. BUT.... lvm prepare once again complained about not detecting a PARTUUID. I think there may be a command to do that, that is left out of the ceph-volume.log :(
So.. now what can I do?
Not to try to create OSDs by hand. That might be an old way to do it, but with containers this is just a PITY. So, my best guess is that you put the OSD in a state that cephadm expects it to be to be: https://docs.ceph.com/en/latest/cephadm/osd/#deploy-osds Make sure all the conditions are met: The device must have no partitions. The device must not have any LVM state. The device must not be mounted. The device must not contain a file system. The device must not contain a Ceph BlueStore OSD. The device must be larger than 5 GB. So make sure everything is destroyed from this device. ceph orch daemon add osd hostname:/dev/device or ceph orch apply osd --all-available-devices I guess that should be the same device (no other unused disks). And let cephadm just make it happen ... wishfull thinking at this point ;-). Gr. Stefan
Unfortunately, neither of those things will work. because ceph orch daemon add does not have a syntax that lets me add an SSD as a journal to a HDD and likewise ceph orch apply osd --all-available-devices will not do the right thing. both for mixed ssd/hdd.. but also, even though I have a lovely deployspec that does data_devices: rotational: 1 db_devices: rotational: 0 and works WONDERFULLY if you are giving it brand new unused SSDS and HDDS.... (it will actually carve up the SSD into lvns based on how many HDDS are present)... I dont think it works when 7/8 devices are already configured, and the SSD is already mostly sliced. ----- Original Message ----- From: "Stefan Kooman" <stefan@bit.nl> To: "Philip Brown" <pbrown@medata.com> Cc: "ceph-users" <ceph-users@ceph.io> Sent: Friday, March 19, 2021 11:34:58 AM Subject: Re: [ceph-users] ceph octopus mysterious OSD crash So make sure everything is destroyed from this device. ceph orch daemon add osd hostname:/dev/device or ceph orch apply osd --all-available-devices
On 3/19/21 7:47 PM, Philip Brown wrote: I see.
I dont think it works when 7/8 devices are already configured, and the SSD is already mostly sliced.
OK. If it is a test cluster you might just blow it all away. By doing this you are simulating a "SSD" failure taking down all HDDs with it. It sure isn't pretty. I would say the situation you ended up with is not a corner case by any means. I am afraid I would really need to set up a test cluster with cephadm to help you further at this point, besides the suggestion above. Gr. Stefan
if we cant replace a drive on a node in a crash situation, without blowing away the entire node.... seems to me ceph octopus fails the "test" part of the "test cluster" :-/ I vaguely recall running into this "doesnt have PARTUUID" problem before. THAT time, I did end up wiping the entire machine I think. But for preparing for production use, I really need to have a better documented method. I note that I cant even fall back to "ceph-disk". since that is no longer in the distribution, it would seem. That would be the "easy" way to deal with this... but it is not here. ----- Original Message ----- From: "Stefan Kooman" <stefan@bit.nl> To: "Philip Brown" <pbrown@medata.com> Cc: "ceph-users" <ceph-users@ceph.io> Sent: Friday, March 19, 2021 12:04:30 PM Subject: Re: [ceph-users] ceph octopus mysterious OSD crash On 3/19/21 7:47 PM, Philip Brown wrote: I see.
I dont think it works when 7/8 devices are already configured, and the SSD is already mostly sliced.
OK. If it is a test cluster you might just blow it all away. By doing this you are simulating a "SSD" failure taking down all HDDs with it. It sure isn't pretty. I would say the situation you ended up with is not a corner case by any means. I am afraid I would really need to set up a test cluster with cephadm to help you further at this point, besides the suggestion above. Gr. Stefan
On 3/19/21 9:11 PM, Philip Brown wrote:
if we cant replace a drive on a node in a crash situation, without blowing away the entire node.... seems to me ceph octopus fails the "test" part of the "test cluster" :-/
I agree. This should not be necessary. And I'm sure there is, or there will be found a solution to fix this issue. If you think it's a bug please create an issue for it [1]. Do note however you do not need to use cephadm / containers. You can still install ceph through regular packages and configure by hand. Sure, the project is definitely moving into the containerized deploy direction, and I guess it might be more future proof to start a new cluster deployed with cephadm. Having said that, I do read in the cephadm documentation that the following should be possible: data_devices: paths: - /dev/sdb db_devices: paths: - /dev/sdc What if you change your spec file to reflect the devices you want to use and try again? Gr. Stefan [1]: https://tracker.ceph.com/
I am quite sure that this case is covered by cephadm already. A few months ago I tested it after a major rework of ceph-volume. I don’t have any links right now. But I had a lab environment with multiple OSDs per node with rocksDB on SSD and after wiping both HDD and DB LV cephadm automatically redeployed the OSD according to my drive group file. Zitat von Stefan Kooman <stefan@bit.nl>:
On 3/19/21 7:47 PM, Philip Brown wrote:
I see.
I dont think it works when 7/8 devices are already configured, and the SSD is already mostly sliced.
OK. If it is a test cluster you might just blow it all away. By doing this you are simulating a "SSD" failure taking down all HDDs with it. It sure isn't pretty. I would say the situation you ended up with is not a corner case by any means. I am afraid I would really need to set up a test cluster with cephadm to help you further at this point, besides the suggestion above.
Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Wow. My expectations have been adjusted. Thank you for detailing your experience, so I had motivation to try again. Explicit steps I took: 1. went into "cephadm shell" and did a vgremove on the HDD 2. ceph-volume zap /dev/(hdd) 3. lvremove (the matching old lv). This meant that the VG on the SSD had 25% space available. At this point, "ceph-volume inventory" shows the HDD as "available=True", but the shared SSD as false. 4. on my actual admin node, "ceph orch apply osd -i osd.deployspec.yml" and after a few minutes... it DID actually pick up the disk and make the OSD. (I had prevously "ceph osd rm"'d the id. so it used the prior ID) SO... there's still the concern about why the thing mysteriosly crashed in the first place :-/ (on TWO osd's!) But at least I know how to rebuild a single disk. ----- Original Message ----- From: "Eugen Block" <eblock@nde.ag> To: "Stefan Kooman" <stefan@bit.nl> Cc: "ceph-users" <ceph-users@ceph.io>, "Philip Brown" <pbrown@medata.com> Sent: Friday, March 19, 2021 2:19:55 PM Subject: [BULK] Re: [ceph-users] Re: ceph octopus mysterious OSD crash I am quite sure that this case is covered by cephadm already. A few months ago I tested it after a major rework of ceph-volume. I don’t have any links right now. But I had a lab environment with multiple OSDs per node with rocksDB on SSD and after wiping both HDD and DB LV cephadm automatically redeployed the OSD according to my drive group file. Zitat von Stefan Kooman <stefan@bit.nl>:
On 3/19/21 7:47 PM, Philip Brown wrote:
I see.
I dont think it works when 7/8 devices are already configured, and the SSD is already mostly sliced.
OK. If it is a test cluster you might just blow it all away. By doing this you are simulating a "SSD" failure taking down all HDDs with it. It sure isn't pretty. I would say the situation you ended up with is not a corner case by any means. I am afraid I would really need to set up a test cluster with cephadm to help you further at this point, besides the suggestion above.
Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Are you sure the OSD is with DB/WAL on SSD? Tony ________________________________________ From: Philip Brown <pbrown@medata.com> Sent: March 19, 2021 02:49 PM To: Eugen Block Cc: ceph-users Subject: [ceph-users] Re: [BULK] Re: Re: ceph octopus mysterious OSD crash Wow. My expectations have been adjusted. Thank you for detailing your experience, so I had motivation to try again. Explicit steps I took: 1. went into "cephadm shell" and did a vgremove on the HDD 2. ceph-volume zap /dev/(hdd) 3. lvremove (the matching old lv). This meant that the VG on the SSD had 25% space available. At this point, "ceph-volume inventory" shows the HDD as "available=True", but the shared SSD as false. 4. on my actual admin node, "ceph orch apply osd -i osd.deployspec.yml" and after a few minutes... it DID actually pick up the disk and make the OSD. (I had prevously "ceph osd rm"'d the id. so it used the prior ID) SO... there's still the concern about why the thing mysteriosly crashed in the first place :-/ (on TWO osd's!) But at least I know how to rebuild a single disk. ----- Original Message ----- From: "Eugen Block" <eblock@nde.ag> To: "Stefan Kooman" <stefan@bit.nl> Cc: "ceph-users" <ceph-users@ceph.io>, "Philip Brown" <pbrown@medata.com> Sent: Friday, March 19, 2021 2:19:55 PM Subject: [BULK] Re: [ceph-users] Re: ceph octopus mysterious OSD crash I am quite sure that this case is covered by cephadm already. A few months ago I tested it after a major rework of ceph-volume. I don’t have any links right now. But I had a lab environment with multiple OSDs per node with rocksDB on SSD and after wiping both HDD and DB LV cephadm automatically redeployed the OSD according to my drive group file. Zitat von Stefan Kooman <stefan@bit.nl>:
On 3/19/21 7:47 PM, Philip Brown wrote:
I see.
I dont think it works when 7/8 devices are already configured, and the SSD is already mostly sliced.
OK. If it is a test cluster you might just blow it all away. By doing this you are simulating a "SSD" failure taking down all HDDs with it. It sure isn't pretty. I would say the situation you ended up with is not a corner case by any means. I am afraid I would really need to set up a test cluster with cephadm to help you further at this point, besides the suggestion above.
Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
you mean, "are you sure the new one got created that way"? yes, because it created a new lv on the ssd for it but I admire your sysadmin paranoia so I'll double check just for you, with ceph-volume lvm list :-) yup. separate block and db devs. block is on HDD db is on ssd ----- Original Message ----- From: "Tony Liu" <tonyliu0592@hotmail.com> To: "Philip Brown" <pbrown@medata.com>, "Eugen Block" <eblock@nde.ag> Cc: "ceph-users" <ceph-users@ceph.io> Sent: Friday, March 19, 2021 4:09:55 PM Subject: Re: ceph octopus mysterious OSD crash Are you sure the OSD is with DB/WAL on SSD? Tony ________________________________________ From: Philip Brown <pbrown@medata.com> Sent: March 19, 2021 02:49 PM To: Eugen Block Cc: ceph-users Subject: [ceph-users] Re: [BULK] Re: Re: ceph octopus mysterious OSD crash Wow. My expectations have been adjusted. Thank you for detailing your experience, so I had motivation to try again. Explicit steps I took: 1. went into "cephadm shell" and did a vgremove on the HDD 2. ceph-volume zap /dev/(hdd) 3. lvremove (the matching old lv). This meant that the VG on the SSD had 25% space available. At this point, "ceph-volume inventory" shows the HDD as "available=True", but the shared SSD as false. 4. on my actual admin node, "ceph orch apply osd -i osd.deployspec.yml" and after a few minutes... it DID actually pick up the disk and make the OSD. (I had prevously "ceph osd rm"'d the id. so it used the prior ID) SO... there's still the concern about why the thing mysteriosly crashed in the first place :-/ (on TWO osd's!) But at least I know how to rebuild a single disk. ----- Original Message ----- From: "Eugen Block" <eblock@nde.ag> To: "Stefan Kooman" <stefan@bit.nl> Cc: "ceph-users" <ceph-users@ceph.io>, "Philip Brown" <pbrown@medata.com> Sent: Friday, March 19, 2021 2:19:55 PM Subject: [BULK] Re: [ceph-users] Re: ceph octopus mysterious OSD crash I am quite sure that this case is covered by cephadm already. A few months ago I tested it after a major rework of ceph-volume. I don’t have any links right now. But I had a lab environment with multiple OSDs per node with rocksDB on SSD and after wiping both HDD and DB LV cephadm automatically redeployed the OSD according to my drive group file. Zitat von Stefan Kooman <stefan@bit.nl>:
On 3/19/21 7:47 PM, Philip Brown wrote:
I see.
I dont think it works when 7/8 devices are already configured, and the SSD is already mostly sliced.
OK. If it is a test cluster you might just blow it all away. By doing this you are simulating a "SSD" failure taking down all HDDs with it. It sure isn't pretty. I would say the situation you ended up with is not a corner case by any means. I am afraid I would really need to set up a test cluster with cephadm to help you further at this point, besides the suggestion above.
Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We also ran into a scenario in which I did exactly this, and it did _not_ work. It created the OSD, but did not put the DB/WAL on the NVME (didn't even create an LV). I'm wondering if there's some constraint applied (haven't looked at code yet) that when the NVME already has all but the one DB on it, it may not have the minimum space required (even though it's plenty based on the specification). Our service specification looks like this: service_type: osd service_id: osd_spec_test placement: host_pattern: '*' data_devices: rotational: 1 db_devices: rotational: 0 db_slots: 12 It works fine when fed an empty machine, but I've yet to get it to work when I've had an OSD fail, and I wipe out the LV for the DB and OSD. I'll get a new OSD, but no DB. On one of our clusters, due to the NVME sizing (800GB / 745.2G usable) + 24 OSDs the DBs (12 per NVME, two NVMEs per server) end up being ~62.1G, so there's about 62.1G free when we clear out the LV. I'm not sure why it doesn't 'do the right thing' and using that when spinning up the replaced OSD. I'm also curious what happens if two OSDs were to fail, you deleted two DBs, then added one OSD back. Would Ceph be smart enough to see the 12 slots per non-rotational in the osd specification and not allocate a 124.2G DB/WAL to that single OSD, preserving enough space for a second (for adding the second OSD later) - assuming this entire process worked as designed? David On Fri, Mar 19, 2021 at 4:20 PM Eugen Block <eblock@nde.ag> wrote:
I am quite sure that this case is covered by cephadm already. A few months ago I tested it after a major rework of ceph-volume. I don’t have any links right now. But I had a lab environment with multiple OSDs per node with rocksDB on SSD and after wiping both HDD and DB LV cephadm automatically redeployed the OSD according to my drive group file.
Zitat von Stefan Kooman <stefan@bit.nl>:
On 3/19/21 7:47 PM, Philip Brown wrote:
I see.
I dont think it works when 7/8 devices are already configured, and the SSD is already mostly sliced.
OK. If it is a test cluster you might just blow it all away. By doing this you are simulating a "SSD" failure taking down all HDDs with it. It sure isn't pretty. I would say the situation you ended up with is not a corner case by any means. I am afraid I would really need to set up a test cluster with cephadm to help you further at this point, besides the suggestion above.
Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
As we wanted to verify this behavior with 15.2.10, we went ahead and tested with a failed OSD. The drive was replaced, and we followed the steps below (comments for clarity on our process) - this assumes you have a service specification that will perform deployment once matched: # capture "db device" associated with OSD ceph-volume list | less # drain drive if possible, do this when planning replacement, otherwise do once failure has occurred ceph orch osd rm 391 --replace # One drained (or if failure occurred), using "db device" path from the ceph-volume list lvremove /dev/ceph-blah/osd-db-blah # monitor ceph for replacement ceph -W cephadm # once daemon has been deployed "TIMESTAMP mgr.cephXX.XXXXX [INF] Deploying daemon osd.391 on cephXX", watch for rebalance to complete ceph -s -------------------- ### consider increasing max_backfills if it's just a single drive replacement: ceph config set osd osd_max_backfills 10 ### if you do, after backfilling is complete: ceph config rm osd osd_max_backfills Following these steps, as soon as we completed the lvremove of the db device in question, the OSD was rebuilt, and we verified a new NVME-based db LV was created as per our specification: service_type: osd service_id: osd_spec_XXXXX service_name: osd.osd_spec_XXXX placement: host_pattern: '*' spec: data_devices: rotational: 1 db_devices: rotational: 0 db_slots: 12 filter_logic: AND objectstore: bluestore Hope this helps out others in the future who need to deal with drive replacements on cephadm/containerized deployments, David On Fri, Mar 19, 2021 at 4:57 PM David Orman <ormandj@corenode.com> wrote:
We also ran into a scenario in which I did exactly this, and it did _not_ work. It created the OSD, but did not put the DB/WAL on the NVME (didn't even create an LV). I'm wondering if there's some constraint applied (haven't looked at code yet) that when the NVME already has all but the one DB on it, it may not have the minimum space required (even though it's plenty based on the specification).
Our service specification looks like this:
service_type: osd service_id: osd_spec_test placement: host_pattern: '*' data_devices: rotational: 1 db_devices: rotational: 0 db_slots: 12
It works fine when fed an empty machine, but I've yet to get it to work when I've had an OSD fail, and I wipe out the LV for the DB and OSD. I'll get a new OSD, but no DB. On one of our clusters, due to the NVME sizing (800GB / 745.2G usable) + 24 OSDs the DBs (12 per NVME, two NVMEs per server) end up being ~62.1G, so there's about 62.1G free when we clear out the LV. I'm not sure why it doesn't 'do the right thing' and using that when spinning up the replaced OSD.
I'm also curious what happens if two OSDs were to fail, you deleted two DBs, then added one OSD back. Would Ceph be smart enough to see the 12 slots per non-rotational in the osd specification and not allocate a 124.2G DB/WAL to that single OSD, preserving enough space for a second (for adding the second OSD later) - assuming this entire process worked as designed?
David
On Fri, Mar 19, 2021 at 4:20 PM Eugen Block <eblock@nde.ag> wrote:
I am quite sure that this case is covered by cephadm already. A few months ago I tested it after a major rework of ceph-volume. I don’t have any links right now. But I had a lab environment with multiple OSDs per node with rocksDB on SSD and after wiping both HDD and DB LV cephadm automatically redeployed the OSD according to my drive group file.
Zitat von Stefan Kooman <stefan@bit.nl>:
On 3/19/21 7:47 PM, Philip Brown wrote:
I see.
I dont think it works when 7/8 devices are already configured, and the SSD is already mostly sliced.
OK. If it is a test cluster you might just blow it all away. By doing this you are simulating a "SSD" failure taking down all HDDs with it. It sure isn't pretty. I would say the situation you ended up with is not a corner case by any means. I am afraid I would really need to set up a test cluster with cephadm to help you further at this point, besides the suggestion above.
Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
DOhhh... Read David's proceedure. And was surprised. I thought the wording of the --replace flag was so blindingly obvious, I didnt think I needed to read the docs. except apparently I do. (ceph orch osd rm --replace) "This follows the same procedure as the “Remove OSD” part with the exception that the OSD is not permanently removed from the CRUSH hierarchy, but is assigned a ‘destroyed’ flag." Oh come on, guys... This is a flag to an *orchestration* suite. Dont you think that when the user/admin uses a --replace flag, the orchestration suite should.. ya know.. orchestrate a replacement? otherwise, in my opinion, that flag really needs to be renamed to something else. ----- Original Message ----- From: "David Orman" <ormandj@corenode.com> To: "Eugen Block" <eblock@nde.ag> Cc: "Stefan Kooman" <stefan@bit.nl>, "ceph-users" <ceph-users@ceph.io>, "Philip Brown" <pbrown@medata.com> Sent: Thursday, March 25, 2021 12:04:17 PM Subject: Re: [ceph-users] Re: ceph octopus mysterious OSD crash As we wanted to verify this behavior with 15.2.10, we went ahead and tested with a failed OSD. The drive was replaced, and we followed the steps below (comments for clarity on our process) - this assumes you have a service specification that will perform deployment once matched: # capture "db device" associated with OSD ceph-volume list | less # drain drive if possible, do this when planning replacement, otherwise do once failure has occurred ceph orch osd rm 391 --replace # One drained (or if failure occurred), using "db device" path from the ceph-volume list lvremove /dev/ceph-blah/osd-db-blah # monitor ceph for replacement ceph -W cephadm # once daemon has been deployed "TIMESTAMP mgr.cephXX.XXXXX [INF] Deploying daemon osd.391 on cephXX", watch for rebalance to complete ceph -s -------------------- ### consider increasing max_backfills if it's just a single drive replacement: ceph config set osd osd_max_backfills 10 ### if you do, after backfilling is complete: ceph config rm osd osd_max_backfills Following these steps, as soon as we completed the lvremove of the db device in question, the OSD was rebuilt, and we verified a new NVME-based db LV was created as per our specification: service_type: osd service_id: osd_spec_XXXXX service_name: osd.osd_spec_XXXX placement: host_pattern: '*' spec: data_devices: rotational: 1 db_devices: rotational: 0 db_slots: 12 filter_logic: AND objectstore: bluestore Hope this helps out others in the future who need to deal with drive replacements on cephadm/containerized deployments, David On Fri, Mar 19, 2021 at 4:57 PM David Orman <ormandj@corenode.com> wrote:
We also ran into a scenario in which I did exactly this, and it did _not_ work. It created the OSD, but did not put the DB/WAL on the NVME (didn't even create an LV). I'm wondering if there's some constraint applied (haven't looked at code yet) that when the NVME already has all but the one DB on it, it may not have the minimum space required (even though it's plenty based on the specification).
Our service specification looks like this:
service_type: osd service_id: osd_spec_test placement: host_pattern: '*' data_devices: rotational: 1 db_devices: rotational: 0 db_slots: 12
It works fine when fed an empty machine, but I've yet to get it to work when I've had an OSD fail, and I wipe out the LV for the DB and OSD. I'll get a new OSD, but no DB. On one of our clusters, due to the NVME sizing (800GB / 745.2G usable) + 24 OSDs the DBs (12 per NVME, two NVMEs per server) end up being ~62.1G, so there's about 62.1G free when we clear out the LV. I'm not sure why it doesn't 'do the right thing' and using that when spinning up the replaced OSD.
I'm also curious what happens if two OSDs were to fail, you deleted two DBs, then added one OSD back. Would Ceph be smart enough to see the 12 slots per non-rotational in the osd specification and not allocate a 124.2G DB/WAL to that single OSD, preserving enough space for a second (for adding the second OSD later) - assuming this entire process worked as designed?
David
On Fri, Mar 19, 2021 at 4:20 PM Eugen Block <eblock@nde.ag> wrote:
I am quite sure that this case is covered by cephadm already. A few months ago I tested it after a major rework of ceph-volume. I don’t have any links right now. But I had a lab environment with multiple OSDs per node with rocksDB on SSD and after wiping both HDD and DB LV cephadm automatically redeployed the OSD according to my drive group file.
Zitat von Stefan Kooman <stefan@bit.nl>:
On 3/19/21 7:47 PM, Philip Brown wrote:
I see.
I dont think it works when 7/8 devices are already configured, and the SSD is already mostly sliced.
OK. If it is a test cluster you might just blow it all away. By doing this you are simulating a "SSD" failure taking down all HDDs with it. It sure isn't pretty. I would say the situation you ended up with is not a corner case by any means. I am afraid I would really need to set up a test cluster with cephadm to help you further at this point, besides the suggestion above.
Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Use journalctl -xe (maybe with -S/-U if you want to filter) to find the time period in which a restart attempt has happened, and see what's logged at that period. If that's not helpful, then what you may want to do is disable that service (systemctl disable blah) then get the ExecStart out of it, then try running it by hand and seeing what happens (the symlinked systemd unit will will refer to a unit.run file in /var/lib/ceph that will have the actual podman cmd). If the pod dies, you should still see it in podman ps -a and you can podman logs on it to get the details. Then you can correct the issue then re-enable the service and restart it properly to do the housekeeping. Follow these directions at your own risk; make sure you understand the ramifications of whatever you might be doing! David On Thu, Mar 18, 2021 at 3:29 PM Philip Brown <pbrown@medata.com> wrote:
I've been banging on my ceph octopus test cluster for a few days now. 8 nodes. each node has 2 SSDs and 8 HDDs. They were all autoprovisioned so that each HDD gets an LVM slice of an SSD as a db partition.
service_type: osd service_id: osd_spec_default placement: host_pattern: '*' data_devices: rotational: 1 db_devices: rotational: 0
things were going pretty good, until... yesterday.. i noticed TWO of the OSDs were "down".
I went to check the logs, with journalctl -u ceph-xxxx@osd.xxx
all it showed were a bunch of generic debug info, and the fact that it stopped. and various automatic attempts to restart. but no indication of what was wrong, and why the restarts KEEP failing.
sample output:
systemd[1]: Stopped Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00. systemd[1]: Starting Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00... bash[9340]: ceph-e51eb2fa-7f82-11eb-94d5-78e3b5148f00-osd.33-activate bash[9340]: WARNING: The same type, major and minor should not be used for multiple devices. bash[9340]: WARNING: The same type, major and minor should not be used for multiple devices. podman[9369]: 2021-03-07 16:00:15.543010794 -0800 PST m=+0.318475882 container create podman[9369]: 2021-03-07 16:00:15.73461926 -0800 PST m=+0.510084288 container init ..... bash[1611473]: --> ceph-volume lvm activate successful for osd ID: 33 podman[1611501]: 2021-03-18 10:23:02.564242824 -0700 PDT m=+1.379793448 container died bash[1611473]: ceph-xx-xx-xx-xx-osd.33 bash[1611473]: WARNING: The same type, major and minor should not be used for multiple devices. (repeat, repeat...) podman[1611615]: 2021-03-18 10:23:03.530992487 -0700 PDT m=+0.333130660 container create
.... systemd[1]: Started Ceph osd.33 for xx-xx-xx-xx systemd[1]: ceph-xx-xx-xx-xx@osd.33.service: main process exited, code=exited, status=1/FAILURE bash[1611797]: ceph-xx-xx-xx-xx-osd.33-deactivate
and eventually it just gives up.
smartctl -a doesnt show any errors on the HDD
dmesg doesnt show anything.
So... what do I do?
-- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown@medata.com| www.medata.com _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
“podman logs ceph-xxxxxxx-osd-xxx” may contains additional logs.
在 2021年3月19日，04:29，Philip Brown <pbrown@medata.com> 写道：
I've been banging on my ceph octopus test cluster for a few days now. 8 nodes. each node has 2 SSDs and 8 HDDs. They were all autoprovisioned so that each HDD gets an LVM slice of an SSD as a db partition.
service_type: osd service_id: osd_spec_default placement: host_pattern: '*' data_devices: rotational: 1 db_devices: rotational: 0
things were going pretty good, until... yesterday.. i noticed TWO of the OSDs were "down".
I went to check the logs, with journalctl -u ceph-xxxx@osd.xxx
all it showed were a bunch of generic debug info, and the fact that it stopped. and various automatic attempts to restart. but no indication of what was wrong, and why the restarts KEEP failing.
sample output:
systemd[1]: Stopped Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00. systemd[1]: Starting Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00... bash[9340]: ceph-e51eb2fa-7f82-11eb-94d5-78e3b5148f00-osd.33-activate bash[9340]: WARNING: The same type, major and minor should not be used for multiple devices. bash[9340]: WARNING: The same type, major and minor should not be used for multiple devices. podman[9369]: 2021-03-07 16:00:15.543010794 -0800 PST m=+0.318475882 container create podman[9369]: 2021-03-07 16:00:15.73461926 -0800 PST m=+0.510084288 container init ..... bash[1611473]: --> ceph-volume lvm activate successful for osd ID: 33 podman[1611501]: 2021-03-18 10:23:02.564242824 -0700 PDT m=+1.379793448 container died bash[1611473]: ceph-xx-xx-xx-xx-osd.33 bash[1611473]: WARNING: The same type, major and minor should not be used for multiple devices. (repeat, repeat...) podman[1611615]: 2021-03-18 10:23:03.530992487 -0700 PDT m=+0.333130660 container create
.... systemd[1]: Started Ceph osd.33 for xx-xx-xx-xx systemd[1]: ceph-xx-xx-xx-xx@osd.33.service: main process exited, code=exited, status=1/FAILURE bash[1611797]: ceph-xx-xx-xx-xx-osd.33-deactivate
and eventually it just gives up.
smartctl -a doesnt show any errors on the HDD
dmesg doesnt show anything.
So... what do I do?
-- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown@medata.com| https://apac01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.medata... _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Unfortunately, the pod wont stay up. So "podman logs" wont work for it. it is not even visible with "podman ps -a" ----- Original Message ----- From: "胡 玮文" <huww98@outlook.com> To: "Philip Brown" <pbrown@medata.com> Cc: "ceph-users" <ceph-users@ceph.io> Sent: Thursday, March 18, 2021 5:56:20 PM Subject: Re: [ceph-users] ceph octopus mysterious OSD crash “podman logs ceph-xxxxxxx-osd-xxx” may contains additional logs.
在 2021年3月19日，04:29，Philip Brown <pbrown@medata.com> 写道：
I've been banging on my ceph octopus test cluster for a few days now. 8 nodes. each node has 2 SSDs and 8 HDDs. They were all autoprovisioned so that each HDD gets an LVM slice of an SSD as a db partition.
service_type: osd service_id: osd_spec_default placement: host_pattern: '*' data_devices: rotational: 1 db_devices: rotational: 0
things were going pretty good, until... yesterday.. i noticed TWO of the OSDs were "down".
I went to check the logs, with journalctl -u ceph-xxxx@osd.xxx
all it showed were a bunch of generic debug info, and the fact that it stopped. and various automatic attempts to restart. but no indication of what was wrong, and why the restarts KEEP failing.
sample output:
systemd[1]: Stopped Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00. systemd[1]: Starting Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00... bash[9340]: ceph-e51eb2fa-7f82-11eb-94d5-78e3b5148f00-osd.33-activate bash[9340]: WARNING: The same type, major and minor should not be used for multiple devices. bash[9340]: WARNING: The same type, major and minor should not be used for multiple devices. podman[9369]: 2021-03-07 16:00:15.543010794 -0800 PST m=+0.318475882 container create podman[9369]: 2021-03-07 16:00:15.73461926 -0800 PST m=+0.510084288 container init ..... bash[1611473]: --> ceph-volume lvm activate successful for osd ID: 33 podman[1611501]: 2021-03-18 10:23:02.564242824 -0700 PDT m=+1.379793448 container died bash[1611473]: ceph-xx-xx-xx-xx-osd.33 bash[1611473]: WARNING: The same type, major and minor should not be used for multiple devices. (repeat, repeat...) podman[1611615]: 2021-03-18 10:23:03.530992487 -0700 PDT m=+0.333130660 container create
.... systemd[1]: Started Ceph osd.33 for xx-xx-xx-xx systemd[1]: ceph-xx-xx-xx-xx@osd.33.service: main process exited, code=exited, status=1/FAILURE bash[1611797]: ceph-xx-xx-xx-xx-osd.33-deactivate
and eventually it just gives up.
smartctl -a doesnt show any errors on the HDD
dmesg doesnt show anything.
So... what do I do?
-- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown@medata.com| https://apac01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.medata... _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (6)
-
David Orman
-
Eugen Block
-
Philip Brown
-
Stefan Kooman
-
Tony Liu
-
胡 玮文