26 Dec
2023
26 Dec
'23
10:45 a.m.
Hi community,
I am running ceph with block rbd with 6 nodes, erasure code 4+2 with
min_size of pool is 4.
When three osd is down, and an PG is state down, some pools is can't write
data, suppose three osd can't start and pg stuck in down state, how i can
delete or recreate pg to replace down pg or another way to allow pool to
write/read data?
Thanks for the community
*Tran Thanh Phong*
Email: tranphong079(a)gmail.com
Skype: tranphong079
26 Dec
26 Dec
12:15 p.m.
Den tis 26 dec. 2023 kl 08:45 skrev Phong Tran Thanh <tranphong079(a)gmail.com>om>:
Hi community,
I am running ceph with block rbd with 6 nodes, erasure code 4+2 with
min_size of pool is 4.
When three osd is down, and an PG is state down, some pools is can't write
data, suppose three osd can't start and pg stuck in down state, how i can
delete or recreate pg to replace down pg or another way to allow pool to
write/read data?
Depending on how the data is laid out in this pool, you might lose
more or less all data from it.
RBD images get split into pieces of 2 or 4M sizes, so that those
pieces end up on different PGs,
which in turn makes them end up on different OSDs and this allows for
load balancing over the'
whole cluster, but also means that if you lose some PGs on a 40G RBD
image (made up of 10k
pieces), chances are very high that the lost PG did contain one or
more of those 10k pieces.
So lost PGs would probably mean that every RBD image of decent sizes
will have holes in them,
and how this affects all the instances that mount the images will be
hard to tell.
If at all possible, try to use the offline OSD tools to try to get
this PG out of one of the bad OSDs.
https://hawkvelt.id.au/post/2022-4-5-ceph-pg-export-import/ might
help, to see how to run
the export + import commands.
If you can get it out, it can be injected (imported) into any other
running OSD and then replicas
will be recreated and moved to where they should be.
If you have disks to spare, make sure to do full copies of the broken
OSDs and work in the copies
instead, to maximize the chances of restoring your data.
If you are very sure that these three OSDs are never coming back, and
have marked the OSDs
as lost, then I guess
ceph pg force_create_pg <pgid>
would be the next step to have the cluster create empty PGs to replace
the lost ones, but I would
consider this only after trying all the possible options for repairing
at least one of the OSDs that held
the PGs that are missing.
--
May the most significant bit of your life be positive.
27 Dec
27 Dec
6:54 a.m.
Thank you for your knowledge. I have a question. Which pool is affected
when the PG is down, and how can I show it?
When a PG is down, is only one pool affected or are multiple pools affected?
Vào Th 3, 26 thg 12, 2023 vào lúc 16:15 Janne Johansson <
icepic.dz(a)gmail.com> đã viết:
Den tis 26 dec. 2023 kl 08:45 skrev Phong Tran Thanh
<
tranphong079(a)gmail.com>gt;:
--
Trân trọng,
----------------------------------------------------------------------------
*Tran Thanh Phong*
Email: tranphong079(a)gmail.com
Skype: tranphong079
Hi community,
I am running ceph with block rbd with 6 nodes, erasure code 4+2 with
min_size of pool is 4.
When three osd is down, and an PG is state down, some pools is can't
write
data, suppose three osd can't start and pg
stuck in down state, how i can
delete or recreate pg to replace down pg or another way to allow pool to
write/read data?
Depending on how the data is laid out in this pool, you might lose
more or less all data from it.
RBD images get split into pieces of 2 or 4M sizes, so that those
pieces end up on different PGs,
which in turn makes them end up on different OSDs and this allows for
load balancing over the'
whole cluster, but also means that if you lose some PGs on a 40G RBD
image (made up of 10k
pieces), chances are very high that the lost PG did contain one or
more of those 10k pieces.
So lost PGs would probably mean that every RBD image of decent sizes
will have holes in them,
and how this affects all the instances that mount the images will be
hard to tell.
If at all possible, try to use the offline OSD tools to try to get
this PG out of one of the bad OSDs.
https://hawkvelt.id.au/post/2022-4-5-ceph-pg-export-import/ might
help, to see how to run
the export + import commands.
If you can get it out, it can be injected (imported) into any other
running OSD and then replicas
will be recreated and moved to where they should be.
If you have disks to spare, make sure to do full copies of the broken
OSDs and work in the copies
instead, to maximize the chances of restoring your data.
If you are very sure that these three OSDs are never coming back, and
have marked the OSDs
as lost, then I guess
ceph pg force_create_pg <pgid>
would be the next step to have the cluster create empty PGs to replace
the lost ones, but I would
consider this only after trying all the possible options for repairing
at least one of the OSDs that held
the PGs that are missing.
--
May the most significant bit of your life be positive.
28 Dec
28 Dec
11:06 a.m.
On 27.12.2023 04:54, Phong Tran Thanh wrote:
Thank you for your knowledge. I have a question. Which
pool is affected
when the PG is down, and how can I show it?
When a PG is down, is only one pool affected or are multiple pools
affected?
If only 1 PG is down only 1 pool is affected.
The name of a PG is {pool-num}.{pg-id} and the pools number you find
with "ceph osd lspools".
ceph health detail
will show which PG is down and all other issues.
ceph pg ls
will show you all PG, their status and the OSD they are running on.
Some useful links
https://docs.ceph.com/en/quincy/rados/operations/monitoring-osd-pg/#monitor…
https://docs.ceph.com/en/quincy/rados/troubleshooting/troubleshooting-pg/
https://docs.ceph.com/en/latest/dev/placement-group/#user-visible-pg-states
--
Kai Stian Olstad
11:35 a.m.
Dear Kai Stian Olstad,
Thank you for your information. It's good knowledge for me.
Vào Th 5, 28 thg 12, 2023 vào lúc 15:06 Kai Stian Olstad <
ceph+list(a)olstad.com> đã viết:
On 27.12.2023 04:54, Phong Tran Thanh wrote:
--
Trân trọng,
----------------------------------------------------------------------------
*Tran Thanh Phong*
Email: tranphong079(a)gmail.com
Skype: tranphong079
Thank you for your knowledge. I have a question.
Which pool is affected
when the PG is down, and how can I show it?
When a PG is down, is only one pool affected or are multiple pools
affected?
If only 1 PG is down only 1 pool is affected.
The name of a PG is {pool-num}.{pg-id} and the pools number you find
with "ceph osd lspools".
ceph health detail
will show which PG is down and all other issues.
ceph pg ls
will show you all PG, their status and the OSD they are running on.
Some useful links
https://docs.ceph.com/en/quincy/rados/operations/monitoring-osd-pg/#monitor…
https://docs.ceph.com/en/quincy/rados/troubleshooting/troubleshooting-pg/
https://docs.ceph.com/en/latest/dev/placement-group/#user-visible-pg-states
--
Kai Stian Olstad
144
days inactive
146
days old
4 comments
3 participants
participants (3)
-
Janne Johansson -
Kai Stian Olstad -
Phong Tran Thanh