18 Mar
2021
18 Mar
'21
11:58 p.m.
I've been banging on my ceph octopus test cluster for a few days now.
8 nodes. each node has 2 SSDs and 8 HDDs.
They were all autoprovisioned so that each HDD gets an LVM slice of an SSD as a db
partition.
service_type: osd
service_id: osd_spec_default
placement:
host_pattern: '*'
data_devices:
rotational: 1
db_devices:
rotational: 0
things were going pretty good, until... yesterday.. i noticed TWO of the OSDs were
"down".
I went to check the logs, with
journalctl -u ceph-xxxx(a)osd.xxx
all it showed were a bunch of generic debug info, and the fact that it stopped.
and various automatic attempts to restart.
but no indication of what was wrong, and why the restarts KEEP failing.
sample output:
systemd[1]: Stopped Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00.
systemd[1]: Starting Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00...
bash[9340]: ceph-e51eb2fa-7f82-11eb-94d5-78e3b5148f00-osd.33-activate
bash[9340]: WARNING: The same type, major and minor should not be used for multiple
devices.
bash[9340]: WARNING: The same type, major and minor should not be used for multiple
devices.
podman[9369]: 2021-03-07 16:00:15.543010794 -0800 PST m=+0.318475882 container create
podman[9369]: 2021-03-07 16:00:15.73461926 -0800 PST m=+0.510084288 container init
.....
bash[1611473]: --> ceph-volume lvm activate successful for osd ID: 33
podman[1611501]: 2021-03-18 10:23:02.564242824 -0700 PDT m=+1.379793448 container died
bash[1611473]: ceph-xx-xx-xx-xx-osd.33
bash[1611473]: WARNING: The same type, major and minor should not be used for multiple
devices.
(repeat, repeat...)
podman[1611615]: 2021-03-18 10:23:03.530992487 -0700 PDT m=+0.333130660 container create
....
systemd[1]: Started Ceph osd.33 for xx-xx-xx-xx
systemd[1]: ceph-xx-xx-xx-xx(a)osd.33.service: main process exited, code=exited,
status=1/FAILURE
bash[1611797]: ceph-xx-xx-xx-xx-osd.33-deactivate
and eventually it just gives up.
smartctl -a doesnt show any errors on the HDD
dmesg doesnt show anything.
So... what do I do?
--
Philip Brown| Sr. Linux System Administrator | Medata, Inc.
5 Peters Canyon Rd Suite 250
Irvine CA 92606
Office 714.918.1310| Fax 714.918.1325
pbrown(a)medata.com| www.medata.com
19 Mar
19 Mar
12:34 a.m.
On 3/18/21 9:28 PM, Philip Brown wrote:
I've been banging on my ceph octopus test cluster
for a few days now.
8 nodes. each node has 2 SSDs and 8 HDDs.
They were all autoprovisioned so that each HDD gets an LVM slice of an SSD as a db
partition.
service_type: osd
service_id: osd_spec_default
placement:
host_pattern: '*'
data_devices:
rotational: 1
db_devices:
rotational: 0
things were going pretty good, until... yesterday.. i noticed TWO of the OSDs were
"down".
I went to check the logs, with
journalctl -u ceph-xxxx(a)osd.xxx
all it showed were a bunch of generic debug info, and the fact that it stopped.
and various automatic attempts to restart.
but no indication of what was wrong, and why the restarts KEEP failing.
It's a deployment made with cephadm? Looks like it as I see podman
messages. Are these all the log messages you can find on those OSDs?
I.e. have you tried to gather logs with cephadm logs [1].
Gr. Stefan
[1]:
https://docs.ceph.com/en/latest/cephadm/troubleshooting/#gathering-log-files
4:50 a.m.
yup cephadm and orch was used to set all this up.
Current state of things:
ceph osd tree shows
33 hdd 1.84698 osd.33 destroyed 0 1.00000
cephadm logs --name osd.33 --fsid xx-xx-xx-xx
along with the systemctl stuff I already saw, showed me new things such as
ceph-osd[1645438]: did not load config file, using default settings.
ceph-osd[1645438]: 2021-03-18T14:31:32.990-0700 7f8bf14e3bc0 -1 parse_file: filesystem
error: cannot get file size: No such file or directory
This suggested to me that I needed to copy over /etc/ceph/ceph.conf to the OSD node.
which I did.
I then also copied over the admin key and generated a fresh bootstrap-osd key with it,
just for good measure, with
ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring
I had saved the previous output of ceph-volume lvm list
and on the OSD node, ran
ceph-volume lvm prepare --data xxxx --block.db xxxx
But it says osd is already prepared.
I tried an activate... it tells me
--> ceph-volume lvm activate successful for osd ID: 33
but now the cephadm logs output shows me
ceph-osd[1677135]: 2021-03-18T17:57:47.982-0700 7ff64593f700 -1 monclient(hunting):
handle_auth_bad_method server allowed_methods [2] but i only support [2]
Not the best error message :-}
Now what do I need to do?
----- Original Message -----
From: "Stefan Kooman" <stefan(a)bit.nl>
To: "Philip Brown" <pbrown(a)medata.com>om>, "ceph-users"
<ceph-users(a)ceph.io>
Sent: Thursday, March 18, 2021 2:04:09 PM
Subject: Re: [ceph-users] ceph octopus mysterious OSD crash
On 3/18/21 9:28 PM, Philip Brown wrote:
I've been banging on my ceph octopus test cluster
for a few days now.
8 nodes. each node has 2 SSDs and 8 HDDs.
They were all autoprovisioned so that each HDD gets an LVM slice of an SSD as a db
partition.
service_type: osd
service_id: osd_spec_default
placement:
host_pattern: '*'
data_devices:
rotational: 1
db_devices:
rotational: 0
things were going pretty good, until... yesterday.. i noticed TWO of the OSDs were
"down".
I went to check the logs, with
journalctl -u ceph-xxxx(a)osd.xxx
all it showed were a bunch of generic debug info, and the fact that it stopped.
and various automatic attempts to restart.
but no indication of what was wrong, and why the restarts KEEP failing.
It's a deployment made with cephadm? Looks like it as I see podman
messages. Are these all the log messages you can find on those OSDs?
I.e. have you tried to gather logs with cephadm logs [1].
Gr. Stefan
[1]:
https://docs.ceph.com/en/latest/cephadm/troubleshooting/#gathering-log-files
8:39 a.m.
On 3/19/21 2:20 AM, Philip Brown wrote:
yup cephadm and orch was used to set all this up.
Current state of things:
ceph osd tree shows
33 hdd 1.84698 osd.33 destroyed 0 1.00000
^^ Destroyed, ehh, this doesn't look good to me. Ceph thinks this OSD is
destroyed. Do you know what might have happened to osd.33? Did you
perform a "kill an OSD" while testing?
AFAIK you can't fix that anymore. You will have to remove it and redploy
it. Might even get a new osd.id.
cephadm logs --name osd.33 --fsid xx-xx-xx-xx
along with the systemctl stuff I already saw, showed me new things such as
ceph-osd[1645438]: did not load config file, using default settings.
ceph-osd[1645438]: 2021-03-18T14:31:32.990-0700 7f8bf14e3bc0 -1 parse_file: filesystem
error: cannot get file size: No such file or directory
This suggested to me that I needed to copy over /etc/ceph/ceph.conf to the OSD node.
which I did.
I then also copied over the admin key and generated a fresh bootstrap-osd key with it,
just for good measure, with
ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring
I had saved the previous output of ceph-volume lvm list
and on the OSD node, ran
ceph-volume lvm prepare --data xxxx --block.db xxxx
But it says osd is already prepared.
I tried an activate... it tells me
--> ceph-volume lvm activate successful for osd ID: 33
but now the cephadm logs output shows me
ceph-osd[1677135]: 2021-03-18T17:57:47.982-0700 7ff64593f700 -1 monclient(hunting):
handle_auth_bad_method server allowed_methods [2] but i only support [2]
Not the best error message :-}
Indeed, would be nice to have a references to [2]. But I think why you
get this is because of the destroyed OSD. I would use cephadm docu on
how to replace an osd. Does that exist? We add a large thread about this
"container" topic (see "[ceph-users] ceph-ansible in Pacific and
beyond?").
Now what do I need to do?
I would remove osd.33. Even manually editing crushmaps if need to
(should not be the case), and then redeploy this osd and wait for recovery.
If you have not manually "destroyed" this osd than either things work
differently in Octopus from things I have seen so far, my memory is
failing me, or some really weird stuff is happening and I would really
like to know what that is.
Wat version are you running? Do note that 15.2.10 has been released.
Gr. Stefan
6:23 p.m.
mkay.
Sooo... what's the new and nifty proper way to clean this up?
The outsider's view is,
"I should just be able to run 'ceph orch osd rm 33'"
but that returns
Unable to find OSDs: ['33']
----- Original Message -----
From: "Stefan Kooman" <stefan(a)bit.nl>
To: "Philip Brown" <pbrown(a)medata.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>
Sent: Thursday, March 18, 2021 10:09:28 PM
Subject: Re: [ceph-users] ceph octopus mysterious OSD crash
On 3/19/21 2:20 AM, Philip Brown wrote:
yup cephadm and orch was used to set all this up.
Current state of things:
ceph osd tree shows
33 hdd 1.84698 osd.33 destroyed 0 1.00000
^^ Destroyed, ehh, this doesn't look good to me. Ceph thinks this OSD is
destroyed. Do you know what might have happened to osd.33? Did you
perform a "kill an OSD" while testing?
AFAIK you can't fix that anymore. You will have to remove it and redploy
it. Might even get a new osd.id.
8:28 p.m.
On 3/19/21 3:53 PM, Philip Brown wrote:
mkay.
Sooo... what's the new and nifty proper way to clean this up?
The outsider's view is,
"I should just be able to run 'ceph orch osd rm 33'"
Can you spawn a cephadm shell and run: ceph osd rm 33?
And / or: ceph osd crush rm 33, or try to do it with cephadm. Does this
work: ceph orch osd crush rm 33?
Gr. Stefan
P.s. I'll have to install an octopus release with cephadm to get myself
up to speed here.
8:52 p.m.
I made *some* progress for cleanup.
I could already do "ceph osd rm 33" from my master. But doing the cleanup on the
actual OSD node was problematical.
ceph-volume lvm zap xxx
wasnt working properly.. because the device wasnt fully released.... because at the
regular OS level, it cant even SEE the VGs??
That caught me by surprise.
But doing cephadm shell let me see the vgs, remove it, and thus have the zap work.
so now we move on to reconstructing the hybrid OSD.
First off, by default, the cephadm shell did not have permission to create OSDs. so I had
to do
[ceph: root@dxxxx /]# ceph auth get client.bootstrap-osd >
/var/lib/ceph/bootstrap-osd/ceph.keyring
Unfortunately, since I had run the lvm zap on both the data /dev/sdX, AND the db lv
partition.. attempting to recreate
the OSD with
ceph-volume lvm prepare --data /dev/sdb --block.db /dev/ceph-xx-xx-xx/osd-db-xxxx
(the original db lvm on SSD, which still technically existed)
FAILED, because
--> blkid could not detect a PARTUUID for device: /dev/ceph-xxxx/osd-xxx
--> Was unable to complete a new OSD, will rollback changes
cmon.... just MAKE one for me???
:-(
Happily, i could grep for osd-db-specific-id-here in /var/log/ceph/ceph-volume.log and
found the exact original lvcreate syntax to remake it.
BUT....
lvm prepare once again complained about not detecting a PARTUUID.
I think there may be a command to do that, that is left out of the ceph-volume.log :(
So.. now what can I do?
----- Original Message -----
From: "Stefan Kooman" <stefan(a)bit.nl>
To: "Philip Brown" <pbrown(a)medata.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>
Sent: Friday, March 19, 2021 9:58:56 AM
Subject: Re: [ceph-users] ceph octopus mysterious OSD crash
On 3/19/21 3:53 PM, Philip Brown wrote:
mkay.
Sooo... what's the new and nifty proper way to clean this up?
The outsider's view is,
"I should just be able to run 'ceph orch osd rm 33'"
Can you spawn a cephadm shell and run: ceph osd rm 33?
And / or: ceph osd crush rm 33, or try to do it with cephadm. Does this
work: ceph orch osd crush rm 33?
Gr. Stefan
P.s. I'll have to install an octopus release with cephadm to get myself
up to speed here.
10:04 p.m.
On 3/19/21 6:22 PM, Philip Brown wrote:
I made *some* progress for cleanup.
I could already do "ceph osd rm 33" from my master. But doing the cleanup on
the actual OSD node was problematical.
ceph-volume lvm zap xxx
wasnt working properly.. because the device wasnt fully released.... because at the
regular OS level, it cant even SEE the VGs??
That caught me by surprise.
But doing cephadm shell let me see the vgs, remove it, and thus have the zap work.
so now we move on to reconstructing the hybrid OSD.
First off, by default, the cephadm shell did not have permission to create OSDs. so I had
to do
[ceph: root@dxxxx /]# ceph auth get client.bootstrap-osd >
/var/lib/ceph/bootstrap-osd/ceph.keyring
Unfortunately, since I had run the lvm zap on both the data /dev/sdX, AND the db lv
partition.. attempting to recreate
the OSD with
ceph-volume lvm prepare --data /dev/sdb --block.db /dev/ceph-xx-xx-xx/osd-db-xxxx
(the original db lvm on SSD, which still technically existed)
FAILED, because
--> blkid could not detect a PARTUUID for device: /dev/ceph-xxxx/osd-xxx
--> Was unable to complete a new OSD, will rollback changes
cmon.... just MAKE one for me???
:-(
Happily, i could grep for osd-db-specific-id-here in /var/log/ceph/ceph-volume.log and
found the exact original lvcreate syntax to remake it.
BUT....
lvm prepare once again complained about not detecting a PARTUUID.
I think there may be a command to do that, that is left out of the ceph-volume.log :(
So.. now what can I do?
Not to try to create OSDs by hand. That might be an old way to do it,
but with containers this is just a PITY. So, my best guess is that you
put the OSD in a state that cephadm expects it to be to be:
https://docs.ceph.com/en/latest/cephadm/osd/#deploy-osds
Make sure all the conditions are met:
The device must have no partitions.
The device must not have any LVM state.
The device must not be mounted.
The device must not contain a file system.
The device must not contain a Ceph BlueStore OSD.
The device must be larger than 5 GB.
So make sure everything is destroyed from this device.
ceph orch daemon add osd hostname:/dev/device
or
ceph orch apply osd --all-available-devices
I guess that should be the same device (no other unused disks).
And let cephadm just make it happen ... wishfull thinking at this point ;-).
Gr. Stefan
10:17 p.m.
Unfortunately, neither of those things will work.
because
ceph orch daemon add
does not have a syntax that lets me add an SSD as a journal to a HDD
and likewise
ceph orch apply osd --all-available-devices
will not do the right thing. both for mixed ssd/hdd.. but also, even though I have a
lovely deployspec that does
data_devices:
rotational: 1
db_devices:
rotational: 0
and works WONDERFULLY if you are giving it brand new unused SSDS and HDDS....
(it will actually carve up the SSD into lvns based on how many HDDS are present)...
I dont think it works when 7/8 devices are already configured, and the SSD is already
mostly sliced.
----- Original Message -----
From: "Stefan Kooman" <stefan(a)bit.nl>
To: "Philip Brown" <pbrown(a)medata.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>
Sent: Friday, March 19, 2021 11:34:58 AM
Subject: Re: [ceph-users] ceph octopus mysterious OSD crash
So make sure everything is destroyed from this device.
ceph orch daemon add osd hostname:/dev/device
or
ceph orch apply osd --all-available-devices
10:34 p.m.
On 3/19/21 7:47 PM, Philip Brown wrote:
I see.
I dont think it works when 7/8 devices are already configured, and the SSD is already
mostly sliced.
OK. If it is a test cluster you might just blow it all away. By doing
this you are simulating a "SSD" failure taking down all HDDs with it. It
sure isn't pretty. I would say the situation you ended up with is not a
corner case by any means. I am afraid I would really need to set up a
test cluster with cephadm to help you further at this point, besides the
suggestion above.
Gr. Stefan
11:41 p.m.
if we cant replace a drive on a node in a crash situation, without blowing away the entire
node....
seems to me ceph octopus fails the "test" part of the "test cluster"
:-/
I vaguely recall running into this "doesnt have PARTUUID" problem before.
THAT time, I did end up wiping the entire machine I think.
But for preparing for production use, I really need to have a better documented method.
I note that I cant even fall back to "ceph-disk". since that is no longer in the
distribution, it would seem.
That would be the "easy" way to deal with this... but it is not here.
----- Original Message -----
From: "Stefan Kooman" <stefan(a)bit.nl>
To: "Philip Brown" <pbrown(a)medata.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>
Sent: Friday, March 19, 2021 12:04:30 PM
Subject: Re: [ceph-users] ceph octopus mysterious OSD crash
On 3/19/21 7:47 PM, Philip Brown wrote:
I see.
I dont think it works when 7/8 devices are already configured, and the SSD is already
mostly sliced.
OK. If it is a test cluster you might just blow it all away. By doing
this you are simulating a "SSD" failure taking down all HDDs with it. It
sure isn't pretty. I would say the situation you ended up with is not a
corner case by any means. I am afraid I would really need to set up a
test cluster with cephadm to help you further at this point, besides the
suggestion above.
Gr. Stefan
20 Mar
20 Mar
12:48 a.m.
On 3/19/21 9:11 PM, Philip Brown wrote:
if we cant replace a drive on a node in a crash
situation, without blowing away the entire node....
seems to me ceph octopus fails the "test" part of the "test cluster"
:-/
I agree. This should not be necessary. And I'm sure there is, or there
will be found a solution to fix this issue. If you think it's a bug
please create an issue for it [1]. Do note however you do not need to
use cephadm / containers. You can still install ceph through regular
packages and configure by hand. Sure, the project is definitely moving
into the containerized deploy direction, and I guess it might be more
future proof to start a new cluster deployed with cephadm. Having said
that, I do read in the cephadm documentation that the following should
be possible:
data_devices:
paths:
- /dev/sdb
db_devices:
paths:
- /dev/sdc
What if you change your spec file to reflect the devices you want to use
and try again?
Gr. Stefan
[1]: https://tracker.ceph.com/
12:49 a.m.
I am quite sure that this case is covered by cephadm already. A few
months ago I tested it after a major rework of ceph-volume. I don’t
have any links right now. But I had a lab environment with multiple
OSDs per node with rocksDB on SSD and after wiping both HDD and DB LV
cephadm automatically redeployed the OSD according to my drive group
file.
Zitat von Stefan Kooman <stefan(a)bit.nl>nl>:
On 3/19/21 7:47 PM, Philip Brown wrote:
I see.
I dont think it works when 7/8 devices are already configured, and
the SSD is already mostly sliced.
OK. If it is a test cluster you might just blow it all away. By
doing this you are simulating a "SSD" failure taking down all HDDs
with it. It sure isn't pretty. I would say the situation you ended
up with is not a corner case by any means. I am afraid I would
really need to set up a test cluster with cephadm to help you
further at this point, besides the suggestion above.
Gr. Stefan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1:19 a.m.
New subject: [BULK] Re: Re: ceph octopus mysterious OSD crash
Wow.
My expectations have been adjusted. Thank you for detailing your experience, so I had
motivation to try again.
Explicit steps I took:
1. went into "cephadm shell" and did a vgremove on the HDD
2. ceph-volume zap /dev/(hdd)
3. lvremove (the matching old lv). This meant that the VG on the SSD had 25% space
available.
At this point, "ceph-volume inventory" shows the HDD as
"available=True", but the shared SSD as false.
4. on my actual admin node, "ceph orch apply osd -i osd.deployspec.yml"
and after a few minutes...
it DID actually pick up the disk and make the OSD.
(I had prevously "ceph osd rm"'d the id. so it used the prior ID)
SO... there's still the concern about why the thing mysteriosly crashed in the first
place :-/
(on TWO osd's!)
But at least I know how to rebuild a single disk.
----- Original Message -----
From: "Eugen Block" <eblock(a)nde.ag>
To: "Stefan Kooman" <stefan(a)bit.nl>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Philip Brown"
<pbrown(a)medata.com>
Sent: Friday, March 19, 2021 2:19:55 PM
Subject: [BULK] Re: [ceph-users] Re: ceph octopus mysterious OSD crash
I am quite sure that this case is covered by cephadm already. A few
months ago I tested it after a major rework of ceph-volume. I don’t
have any links right now. But I had a lab environment with multiple
OSDs per node with rocksDB on SSD and after wiping both HDD and DB LV
cephadm automatically redeployed the OSD according to my drive group
file.
Zitat von Stefan Kooman <stefan(a)bit.nl>nl>:
> On 3/19/21 7:47 PM, Philip Brown wrote:
>
> I see.
>
>>
>> I dont think it works when 7/8 devices are already configured, and
>> the SSD is already mostly sliced.
>
> OK. If it is a test cluster you might just blow it all away. By
> doing this you are simulating a "SSD" failure taking down all HDDs
> with it. It sure isn't pretty. I would say the situation you ended
> up with is not a corner case by any means. I am afraid I would
> really need to set up a test cluster with cephadm to help you
> further at this point, besides the suggestion above.
>
> Gr. Stefan
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:39 a.m.
Are you sure the OSD is with DB/WAL on SSD?
Tony
________________________________________
From: Philip Brown <pbrown(a)medata.com>
Sent: March 19, 2021 02:49 PM
To: Eugen Block
Cc: ceph-users
Subject: [ceph-users] Re: [BULK] Re: Re: ceph octopus mysterious OSD crash
Wow.
My expectations have been adjusted. Thank you for detailing your experience, so I had
motivation to try again.
Explicit steps I took:
1. went into "cephadm shell" and did a vgremove on the HDD
2. ceph-volume zap /dev/(hdd)
3. lvremove (the matching old lv). This meant that the VG on the SSD had 25% space
available.
At this point, "ceph-volume inventory" shows the HDD as
"available=True", but the shared SSD as false.
4. on my actual admin node, "ceph orch apply osd -i osd.deployspec.yml"
and after a few minutes...
it DID actually pick up the disk and make the OSD.
(I had prevously "ceph osd rm"'d the id. so it used the prior ID)
SO... there's still the concern about why the thing mysteriosly crashed in the first
place :-/
(on TWO osd's!)
But at least I know how to rebuild a single disk.
----- Original Message -----
From: "Eugen Block" <eblock(a)nde.ag>
To: "Stefan Kooman" <stefan(a)bit.nl>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Philip Brown"
<pbrown(a)medata.com>
Sent: Friday, March 19, 2021 2:19:55 PM
Subject: [BULK] Re: [ceph-users] Re: ceph octopus mysterious OSD crash
I am quite sure that this case is covered by cephadm already. A few
months ago I tested it after a major rework of ceph-volume. I don’t
have any links right now. But I had a lab environment with multiple
OSDs per node with rocksDB on SSD and after wiping both HDD and DB LV
cephadm automatically redeployed the OSD according to my drive group
file.
Zitat von Stefan Kooman <stefan(a)bit.nl>nl>:
On 3/19/21 7:47 PM, Philip Brown wrote:
I see.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
I dont think it works when 7/8 devices are already configured, and
the SSD is already mostly sliced.
OK. If it is a test cluster you might just blow it all away. By
doing this you are simulating a "SSD" failure taking down all HDDs
with it. It sure isn't pretty. I would say the situation you ended
up with is not a corner case by any means. I am afraid I would
really need to set up a test cluster with cephadm to help you
further at this point, besides the suggestion above.
Gr. Stefan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:52 a.m.
you mean, "are you sure the new one got created that way"?
yes, because it created a new lv on the ssd for it
but I admire your sysadmin paranoia so I'll double check just for you, with
ceph-volume lvm list :-)
yup.
separate block and db devs.
block is on HDD
db is on ssd
----- Original Message -----
From: "Tony Liu" <tonyliu0592(a)hotmail.com>
To: "Philip Brown" <pbrown(a)medata.com>om>, "Eugen Block"
<eblock(a)nde.ag>
Cc: "ceph-users" <ceph-users(a)ceph.io>
Sent: Friday, March 19, 2021 4:09:55 PM
Subject: Re: ceph octopus mysterious OSD crash
Are you sure the OSD is with DB/WAL on SSD?
Tony
________________________________________
From: Philip Brown <pbrown(a)medata.com>
Sent: March 19, 2021 02:49 PM
To: Eugen Block
Cc: ceph-users
Subject: [ceph-users] Re: [BULK] Re: Re: ceph octopus mysterious OSD crash
Wow.
My expectations have been adjusted. Thank you for detailing your experience, so I had
motivation to try again.
Explicit steps I took:
1. went into "cephadm shell" and did a vgremove on the HDD
2. ceph-volume zap /dev/(hdd)
3. lvremove (the matching old lv). This meant that the VG on the SSD had 25% space
available.
At this point, "ceph-volume inventory" shows the HDD as
"available=True", but the shared SSD as false.
4. on my actual admin node, "ceph orch apply osd -i osd.deployspec.yml"
and after a few minutes...
it DID actually pick up the disk and make the OSD.
(I had prevously "ceph osd rm"'d the id. so it used the prior ID)
SO... there's still the concern about why the thing mysteriosly crashed in the first
place :-/
(on TWO osd's!)
But at least I know how to rebuild a single disk.
----- Original Message -----
From: "Eugen Block" <eblock(a)nde.ag>
To: "Stefan Kooman" <stefan(a)bit.nl>
Cc: "ceph-users" <ceph-users(a)ceph.io>io>, "Philip Brown"
<pbrown(a)medata.com>
Sent: Friday, March 19, 2021 2:19:55 PM
Subject: [BULK] Re: [ceph-users] Re: ceph octopus mysterious OSD crash
I am quite sure that this case is covered by cephadm already. A few
months ago I tested it after a major rework of ceph-volume. I don’t
have any links right now. But I had a lab environment with multiple
OSDs per node with rocksDB on SSD and after wiping both HDD and DB LV
cephadm automatically redeployed the OSD according to my drive group
file.
Zitat von Stefan Kooman <stefan(a)bit.nl>nl>:
On 3/19/21 7:47 PM, Philip Brown wrote:
I see.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
I dont think it works when 7/8 devices are already configured, and
the SSD is already mostly sliced.
OK. If it is a test cluster you might just blow it all away. By
doing this you are simulating a "SSD" failure taking down all HDDs
with it. It sure isn't pretty. I would say the situation you ended
up with is not a corner case by any means. I am afraid I would
really need to set up a test cluster with cephadm to help you
further at this point, besides the suggestion above.
Gr. Stefan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1:27 a.m.
We also ran into a scenario in which I did exactly this, and it did
_not_ work. It created the OSD, but did not put the DB/WAL on the NVME
(didn't even create an LV). I'm wondering if there's some constraint
applied (haven't looked at code yet) that when the NVME already has
all but the one DB on it, it may not have the minimum space required
(even though it's plenty based on the specification).
Our service specification looks like this:
service_type: osd
service_id: osd_spec_test
placement:
host_pattern: '*'
data_devices:
rotational: 1
db_devices:
rotational: 0
db_slots: 12
It works fine when fed an empty machine, but I've yet to get it to
work when I've had an OSD fail, and I wipe out the LV for the DB and
OSD. I'll get a new OSD, but no DB. On one of our clusters, due to the
NVME sizing (800GB / 745.2G usable) + 24 OSDs the DBs (12 per NVME,
two NVMEs per server) end up being ~62.1G, so there's about 62.1G free
when we clear out the LV. I'm not sure why it doesn't 'do the right
thing' and using that when spinning up the replaced OSD.
I'm also curious what happens if two OSDs were to fail, you deleted
two DBs, then added one OSD back. Would Ceph be smart enough to see
the 12 slots per non-rotational in the osd specification and not
allocate a 124.2G DB/WAL to that single OSD, preserving enough space
for a second (for adding the second OSD later) - assuming this entire
process worked as designed?
David
On Fri, Mar 19, 2021 at 4:20 PM Eugen Block <eblock(a)nde.ag> wrote:
>
> I am quite sure that this case is covered by cephadm already. A few
> months ago I tested it after a major rework of ceph-volume. I don’t
> have any links right now. But I had a lab environment with multiple
> OSDs per node with rocksDB on SSD and after wiping both HDD and DB LV
> cephadm automatically redeployed the OSD according to my drive group
> file.
>
>
> Zitat von Stefan Kooman <stefan(a)bit.nl>nl>:
>
> > On 3/19/21 7:47 PM, Philip Brown wrote:
> >
> > I see.
> >
> >>
> >> I dont think it works when 7/8 devices are already configured, and
> >> the SSD is already mostly sliced.
> >
> > OK. If it is a test cluster you might just blow it all away. By
> > doing this you are simulating a "SSD" failure taking down all HDDs
> > with it. It sure isn't pretty. I would say the situation you ended
> > up with is not a corner case by any means. I am afraid I would
> > really need to set up a test cluster with cephadm to help you
> > further at this point, besides the suggestion above.
> >
> > Gr. Stefan
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
25 Mar
25 Mar
11:34 p.m.
As we wanted to verify this behavior with 15.2.10, we went ahead and
tested with a failed OSD. The drive was replaced, and we followed the
steps below (comments for clarity on our process) - this assumes you
have a service specification that will perform deployment once
matched:
# capture "db device" associated with OSD
ceph-volume list | less
# drain drive if possible, do this when planning replacement,
otherwise do once failure has occurred
ceph orch osd rm 391 --replace
# One drained (or if failure occurred), using "db device" path from
the ceph-volume list
lvremove /dev/ceph-blah/osd-db-blah
# monitor ceph for replacement
ceph -W cephadm
# once daemon has been deployed "TIMESTAMP mgr.cephXX.XXXXX [INF]
Deploying daemon osd.391 on cephXX", watch for rebalance to complete
ceph -s
--------------------
### consider increasing max_backfills if it's just a single drive replacement:
ceph config set osd osd_max_backfills 10
### if you do, after backfilling is complete:
ceph config rm osd osd_max_backfills
Following these steps, as soon as we completed the lvremove of the db
device in question, the OSD was rebuilt, and we verified a new
NVME-based db LV was created as per our specification:
service_type: osd
service_id: osd_spec_XXXXX
service_name: osd.osd_spec_XXXX
placement:
host_pattern: '*'
spec:
data_devices:
rotational: 1
db_devices:
rotational: 0
db_slots: 12
filter_logic: AND
objectstore: bluestore
Hope this helps out others in the future who need to deal with drive
replacements on cephadm/containerized deployments,
David
On Fri, Mar 19, 2021 at 4:57 PM David Orman <ormandj(a)corenode.com> wrote:
>
> We also ran into a scenario in which I did exactly this, and it did
> _not_ work. It created the OSD, but did not put the DB/WAL on the NVME
> (didn't even create an LV). I'm wondering if there's some constraint
> applied (haven't looked at code yet) that when the NVME already has
> all but the one DB on it, it may not have the minimum space required
> (even though it's plenty based on the specification).
>
> Our service specification looks like this:
>
> service_type: osd
> service_id: osd_spec_test
> placement:
> host_pattern: '*'
> data_devices:
> rotational: 1
> db_devices:
> rotational: 0
> db_slots: 12
>
> It works fine when fed an empty machine, but I've yet to get it to
> work when I've had an OSD fail, and I wipe out the LV for the DB and
> OSD. I'll get a new OSD, but no DB. On one of our clusters, due to the
> NVME sizing (800GB / 745.2G usable) + 24 OSDs the DBs (12 per NVME,
> two NVMEs per server) end up being ~62.1G, so there's about 62.1G free
> when we clear out the LV. I'm not sure why it doesn't 'do the right
> thing' and using that when spinning up the replaced OSD.
>
> I'm also curious what happens if two OSDs were to fail, you deleted
> two DBs, then added one OSD back. Would Ceph be smart enough to see
> the 12 slots per non-rotational in the osd specification and not
> allocate a 124.2G DB/WAL to that single OSD, preserving enough space
> for a second (for adding the second OSD later) - assuming this entire
> process worked as designed?
>
> David
>
> On Fri, Mar 19, 2021 at 4:20 PM Eugen Block <eblock(a)nde.ag> wrote:
> >
> > I am quite sure that this case is covered by cephadm already. A few
> > months ago I tested it after a major rework of ceph-volume. I don’t
> > have any links right now. But I had a lab environment with multiple
> > OSDs per node with rocksDB on SSD and after wiping both HDD and DB LV
> > cephadm automatically redeployed the OSD according to my drive group
> > file.
> >
> >
> > Zitat von Stefan Kooman <stefan(a)bit.nl>nl>:
> >
> > > On 3/19/21 7:47 PM, Philip Brown wrote:
> > >
> > > I see.
> > >
> > >>
> > >> I dont think it works when 7/8 devices are already configured, and
> > >> the SSD is already mostly sliced.
> > >
> > > OK. If it is a test cluster you might just blow it all away. By
> > > doing this you are simulating a "SSD" failure taking down all
HDDs
> > > with it. It sure isn't pretty. I would say the situation you ended
> > > up with is not a corner case by any means. I am afraid I would
> > > really need to set up a test cluster with cephadm to help you
> > > further at this point, besides the suggestion above.
> > >
> > > Gr. Stefan
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >
> >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
26 Mar
26 Mar
12:07 a.m.
DOhhh...
Read David's proceedure. And was surprised.
I thought the wording of the --replace flag was so blindingly obvious, I didnt think I
needed to read the docs. except apparently I do.
(ceph orch osd rm --replace)
"This follows the same procedure as the “Remove OSD” part with the exception that the
OSD is not permanently removed from the CRUSH hierarchy, but is assigned a ‘destroyed’
flag."
Oh come on, guys...
This is a flag to an *orchestration* suite.
Dont you think that when the user/admin uses a --replace flag, the orchestration suite
should.. ya know.. orchestrate a replacement?
otherwise, in my opinion, that flag really needs to be renamed to something else.
----- Original Message -----
From: "David Orman" <ormandj(a)corenode.com>
To: "Eugen Block" <eblock(a)nde.ag>
Cc: "Stefan Kooman" <stefan(a)bit.nl>nl>, "ceph-users"
<ceph-users(a)ceph.io>io>, "Philip Brown" <pbrown(a)medata.com>
Sent: Thursday, March 25, 2021 12:04:17 PM
Subject: Re: [ceph-users] Re: ceph octopus mysterious OSD crash
As we wanted to verify this behavior with 15.2.10, we went ahead and
tested with a failed OSD. The drive was replaced, and we followed the
steps below (comments for clarity on our process) - this assumes you
have a service specification that will perform deployment once
matched:
# capture "db device" associated with OSD
ceph-volume list | less
# drain drive if possible, do this when planning replacement,
otherwise do once failure has occurred
ceph orch osd rm 391 --replace
# One drained (or if failure occurred), using "db device" path from
the ceph-volume list
lvremove /dev/ceph-blah/osd-db-blah
# monitor ceph for replacement
ceph -W cephadm
# once daemon has been deployed "TIMESTAMP mgr.cephXX.XXXXX [INF]
Deploying daemon osd.391 on cephXX", watch for rebalance to complete
ceph -s
--------------------
### consider increasing max_backfills if it's just a single drive replacement:
ceph config set osd osd_max_backfills 10
### if you do, after backfilling is complete:
ceph config rm osd osd_max_backfills
Following these steps, as soon as we completed the lvremove of the db
device in question, the OSD was rebuilt, and we verified a new
NVME-based db LV was created as per our specification:
service_type: osd
service_id: osd_spec_XXXXX
service_name: osd.osd_spec_XXXX
placement:
host_pattern: '*'
spec:
data_devices:
rotational: 1
db_devices:
rotational: 0
db_slots: 12
filter_logic: AND
objectstore: bluestore
Hope this helps out others in the future who need to deal with drive
replacements on cephadm/containerized deployments,
David
On Fri, Mar 19, 2021 at 4:57 PM David Orman <ormandj(a)corenode.com> wrote:
>
> We also ran into a scenario in which I did exactly this, and it did
> _not_ work. It created the OSD, but did not put the DB/WAL on the NVME
> (didn't even create an LV). I'm wondering if there's some constraint
> applied (haven't looked at code yet) that when the NVME already has
> all but the one DB on it, it may not have the minimum space required
> (even though it's plenty based on the specification).
>
> Our service specification looks like this:
>
> service_type: osd
> service_id: osd_spec_test
> placement:
> host_pattern: '*'
> data_devices:
> rotational: 1
> db_devices:
> rotational: 0
> db_slots: 12
>
> It works fine when fed an empty machine, but I've yet to get it to
> work when I've had an OSD fail, and I wipe out the LV for the DB and
> OSD. I'll get a new OSD, but no DB. On one of our clusters, due to the
> NVME sizing (800GB / 745.2G usable) + 24 OSDs the DBs (12 per NVME,
> two NVMEs per server) end up being ~62.1G, so there's about 62.1G free
> when we clear out the LV. I'm not sure why it doesn't 'do the right
> thing' and using that when spinning up the replaced OSD.
>
> I'm also curious what happens if two OSDs were to fail, you deleted
> two DBs, then added one OSD back. Would Ceph be smart enough to see
> the 12 slots per non-rotational in the osd specification and not
> allocate a 124.2G DB/WAL to that single OSD, preserving enough space
> for a second (for adding the second OSD later) - assuming this entire
> process worked as designed?
>
> David
>
> On Fri, Mar 19, 2021 at 4:20 PM Eugen Block <eblock(a)nde.ag> wrote:
> >
> > I am quite sure that this case is covered by cephadm already. A few
> > months ago I tested it after a major rework of ceph-volume. I don’t
> > have any links right now. But I had a lab environment with multiple
> > OSDs per node with rocksDB on SSD and after wiping both HDD and DB LV
> > cephadm automatically redeployed the OSD according to my drive group
> > file.
> >
> >
> > Zitat von Stefan Kooman <stefan(a)bit.nl>nl>:
> >
> > > On 3/19/21 7:47 PM, Philip Brown wrote:
> > >
> > > I see.
> > >
> > >>
> > >> I dont think it works when 7/8 devices are already configured, and
> > >> the SSD is already mostly sliced.
> > >
> > > OK. If it is a test cluster you might just blow it all away. By
> > > doing this you are simulating a "SSD" failure taking down all
HDDs
> > > with it. It sure isn't pretty. I would say the situation you ended
> > > up with is not a corner case by any means. I am afraid I would
> > > really need to set up a test cluster with cephadm to help you
> > > further at this point, besides the suggestion above.
> > >
> > > Gr. Stefan
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >
> >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
19 Mar
19 Mar
1:48 a.m.
Use journalctl -xe (maybe with -S/-U if you want to filter) to find
the time period in which a restart attempt has happened, and see
what's logged at that period. If that's not helpful, then what you may
want to do is disable that service (systemctl disable blah) then get
the ExecStart out of it, then try running it by hand and seeing what
happens (the symlinked systemd unit will will refer to a unit.run file
in /var/lib/ceph that will have the actual podman cmd). If the pod
dies, you should still see it in podman ps -a and you can podman logs
on it to get the details. Then you can correct the issue then
re-enable the service and restart it properly to do the housekeeping.
Follow these directions at your own risk; make sure you understand the
ramifications of whatever you might be doing!
David
On Thu, Mar 18, 2021 at 3:29 PM Philip Brown <pbrown(a)medata.com> wrote:
>
> I've been banging on my ceph octopus test cluster for a few days now.
> 8 nodes. each node has 2 SSDs and 8 HDDs.
> They were all autoprovisioned so that each HDD gets an LVM slice of an SSD as a db
partition.
>
> service_type: osd
> service_id: osd_spec_default
> placement:
> host_pattern: '*'
> data_devices:
> rotational: 1
> db_devices:
> rotational: 0
>
>
> things were going pretty good, until... yesterday.. i noticed TWO of the OSDs were
"down".
>
> I went to check the logs, with
> journalctl -u ceph-xxxx(a)osd.xxx
>
> all it showed were a bunch of generic debug info, and the fact that it stopped.
> and various automatic attempts to restart.
> but no indication of what was wrong, and why the restarts KEEP failing.
>
>
> sample output:
>
>
> systemd[1]: Stopped Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00.
> systemd[1]: Starting Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00...
> bash[9340]: ceph-e51eb2fa-7f82-11eb-94d5-78e3b5148f00-osd.33-activate
> bash[9340]: WARNING: The same type, major and minor should not be used for multiple
devices.
> bash[9340]: WARNING: The same type, major and minor should not be used for multiple
devices.
> podman[9369]: 2021-03-07 16:00:15.543010794 -0800 PST m=+0.318475882 container
create
> podman[9369]: 2021-03-07 16:00:15.73461926 -0800 PST m=+0.510084288 container init
> .....
> bash[1611473]: --> ceph-volume lvm activate successful for osd ID: 33
> podman[1611501]: 2021-03-18 10:23:02.564242824 -0700 PDT m=+1.379793448 container
died
> bash[1611473]: ceph-xx-xx-xx-xx-osd.33
> bash[1611473]: WARNING: The same type, major and minor should not be used for
multiple devices.
> (repeat, repeat...)
> podman[1611615]: 2021-03-18 10:23:03.530992487 -0700 PDT m=+0.333130660 container
create
>
> ....
> systemd[1]: Started Ceph osd.33 for xx-xx-xx-xx
> systemd[1]: ceph-xx-xx-xx-xx(a)osd.33.service: main process exited, code=exited,
status=1/FAILURE
> bash[1611797]: ceph-xx-xx-xx-xx-osd.33-deactivate
>
> and eventually it just gives up.
>
> smartctl -a doesnt show any errors on the HDD
>
>
> dmesg doesnt show anything.
>
> So... what do I do?
>
>
>
>
>
> --
> Philip Brown| Sr. Linux System Administrator | Medata, Inc.
> 5 Peters Canyon Rd Suite 250
> Irvine CA 92606
> Office 714.918.1310| Fax 714.918.1325
> pbrown(a)medata.com| www.medata.com
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:26 a.m.
“podman logs ceph-xxxxxxx-osd-xxx” may contains additional logs.
在 2021年3月19日，04:29，Philip Brown
<pbrown(a)medata.com> 写道：
I've been banging on my ceph octopus test cluster for a few days now.
8 nodes. each node has 2 SSDs and 8 HDDs.
They were all autoprovisioned so that each HDD gets an LVM slice of an SSD as a db
partition.
service_type: osd
service_id: osd_spec_default
placement:
host_pattern: '*'
data_devices:
rotational: 1
db_devices:
rotational: 0
things were going pretty good, until... yesterday.. i noticed TWO of the OSDs were
"down".
I went to check the logs, with
journalctl -u ceph-xxxx(a)osd.xxx
all it showed were a bunch of generic debug info, and the fact that it stopped.
and various automatic attempts to restart.
but no indication of what was wrong, and why the restarts KEEP failing.
sample output:
systemd[1]: Stopped Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00.
systemd[1]: Starting Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00...
bash[9340]: ceph-e51eb2fa-7f82-11eb-94d5-78e3b5148f00-osd.33-activate
bash[9340]: WARNING: The same type, major and minor should not be used for multiple
devices.
bash[9340]: WARNING: The same type, major and minor should not be used for multiple
devices.
podman[9369]: 2021-03-07 16:00:15.543010794 -0800 PST m=+0.318475882 container create
podman[9369]: 2021-03-07 16:00:15.73461926 -0800 PST m=+0.510084288 container init
.....
bash[1611473]: --> ceph-volume lvm activate successful for osd ID: 33
podman[1611501]: 2021-03-18 10:23:02.564242824 -0700 PDT m=+1.379793448 container died
bash[1611473]: ceph-xx-xx-xx-xx-osd.33
bash[1611473]: WARNING: The same type, major and minor should not be used for multiple
devices.
(repeat, repeat...)
podman[1611615]: 2021-03-18 10:23:03.530992487 -0700 PDT m=+0.333130660 container
create
....
systemd[1]: Started Ceph osd.33 for xx-xx-xx-xx
systemd[1]: ceph-xx-xx-xx-xx(a)osd.33.service: main process exited, code=exited,
status=1/FAILURE
bash[1611797]: ceph-xx-xx-xx-xx-osd.33-deactivate
and eventually it just gives up.
smartctl -a doesnt show any errors on the HDD
dmesg doesnt show anything.
So... what do I do?
--
Philip Brown| Sr. Linux System Administrator | Medata, Inc.
5 Peters Canyon Rd Suite 250
Irvine CA 92606
Office 714.918.1310| Fax 714.918.1325
pbrown(a)medata.com|
https://apac01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.medat…
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:51 a.m.
Unfortunately, the pod wont stay up. So "podman logs" wont work for it.
it is not even visible with "podman ps -a"
----- Original Message -----
From: "胡 玮文" <huww98(a)outlook.com>
To: "Philip Brown" <pbrown(a)medata.com>
Cc: "ceph-users" <ceph-users(a)ceph.io>
Sent: Thursday, March 18, 2021 5:56:20 PM
Subject: Re: [ceph-users] ceph octopus mysterious OSD crash
“podman logs ceph-xxxxxxx-osd-xxx” may contains additional logs.
> 在 2021年3月19日，04:29，Philip Brown <pbrown(a)medata.com> 写道：
>
> I've been banging on my ceph octopus test cluster for a few days now.
> 8 nodes. each node has 2 SSDs and 8 HDDs.
> They were all autoprovisioned so that each HDD gets an LVM slice of an SSD as a db
partition.
>
> service_type: osd
> service_id: osd_spec_default
> placement:
> host_pattern: '*'
> data_devices:
> rotational: 1
> db_devices:
> rotational: 0
>
>
> things were going pretty good, until... yesterday.. i noticed TWO of the OSDs were
"down".
>
> I went to check the logs, with
> journalctl -u ceph-xxxx(a)osd.xxx
>
> all it showed were a bunch of generic debug info, and the fact that it stopped.
> and various automatic attempts to restart.
> but no indication of what was wrong, and why the restarts KEEP failing.
>
>
> sample output:
>
>
> systemd[1]: Stopped Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00.
> systemd[1]: Starting Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00...
> bash[9340]: ceph-e51eb2fa-7f82-11eb-94d5-78e3b5148f00-osd.33-activate
> bash[9340]: WARNING: The same type, major and minor should not be used for multiple
devices.
> bash[9340]: WARNING: The same type, major and minor should not be used for multiple
devices.
> podman[9369]: 2021-03-07 16:00:15.543010794 -0800 PST m=+0.318475882 container
create
> podman[9369]: 2021-03-07 16:00:15.73461926 -0800 PST m=+0.510084288 container init
> .....
> bash[1611473]: --> ceph-volume lvm activate successful for osd ID: 33
> podman[1611501]: 2021-03-18 10:23:02.564242824 -0700 PDT m=+1.379793448 container
died
> bash[1611473]: ceph-xx-xx-xx-xx-osd.33
> bash[1611473]: WARNING: The same type, major and minor should not be used for
multiple devices.
> (repeat, repeat...)
> podman[1611615]: 2021-03-18 10:23:03.530992487 -0700 PDT m=+0.333130660 container
create
>
> ....
> systemd[1]: Started Ceph osd.33 for xx-xx-xx-xx
> systemd[1]: ceph-xx-xx-xx-xx(a)osd.33.service: main process exited, code=exited,
status=1/FAILURE
> bash[1611797]: ceph-xx-xx-xx-xx-osd.33-deactivate
>
> and eventually it just gives up.
>
> smartctl -a doesnt show any errors on the HDD
>
>
> dmesg doesnt show anything.
>
> So... what do I do?
>
>
>
>
>
> --
> Philip Brown| Sr. Linux System Administrator | Medata, Inc.
> 5 Peters Canyon Rd Suite 250
> Irvine CA 92606
> Office 714.918.1310| Fax 714.918.1325
> pbrown(a)medata.com|
https://apac01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.medat…
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1147
days inactive
1154
days old
21 comments
6 participants
participants (6)
-
David Orman -
Eugen Block -
Philip Brown -
Stefan Kooman -
Tony Liu -
胡 玮文