*Hello everyone,* I’m running a Ceph cluster used as an RGW backend, and I’m facing an issue with one particular placement group (PG). - Accessing objects from this PG is *extremely slow*. - Even running ceph pg <pg_id> takes a very long time. - The PG is currently *stuck in a degraded state*, so I’m unable to move it to other OSDs. Current ceph version is reef 18.2.7. Has anyone encountered a similar issue before or have any suggestions on how to troubleshoot and resolve it? Thanks in advance!
could it be an OSD not responding? I would usually do ceph tell osd.* version To test if you can connect with osds On Thu, Oct 2, 2025 at 12:09 PM Sa Pham <saphi070@gmail.com> wrote:
*Hello everyone,*
I’m running a Ceph cluster used as an RGW backend, and I’m facing an issue with one particular placement group (PG).
-
Accessing objects from this PG is *extremely slow*. -
Even running ceph pg <pg_id> takes a very long time. -
The PG is currently *stuck in a degraded state*, so I’m unable to move it to other OSDs.
Current ceph version is reef 18.2.7.
Has anyone encountered a similar issue before or have any suggestions on how to troubleshoot and resolve it?
Thanks in advance! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Joshua, No. OSD is still responding. # ceph tell osd.130 version { "version": "18.2.7-0-g6b0e988052e", "release": "reef", "release_type": "stable" } But the primary OSD which includes slow pg (18.773) will respond slower. detailed as below # ceph pg dump_stuck degraded PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 18.773 active+undersized+degraded+remapped+backfilling [302,150,138] 302 [130,101] 130 # time ceph tell osd.130 version { "version": "18.2.7-0-g6b0e988052e", "release": "reef", "release_type": "stable" } real 0m2.113s user 0m0.148s sys 0m0.036s # time ceph tell osd.101 version { "version": "18.2.7-0-g6b0e988052e", "release": "reef", "release_type": "stable" } real 0m0.192s user 0m0.152s sys 0m0.037s I don't know why. Regards, On Fri, Oct 3, 2025 at 4:21 AM Joshua Blanch <joshua.blanch@clyso.com> wrote:
could it be an OSD not responding?
I would usually do
ceph tell osd.* version
To test if you can connect with osds
On Thu, Oct 2, 2025 at 12:09 PM Sa Pham <saphi070@gmail.com> wrote:
*Hello everyone,*
I’m running a Ceph cluster used as an RGW backend, and I’m facing an issue with one particular placement group (PG).
-
Accessing objects from this PG is *extremely slow*. -
Even running ceph pg <pg_id> takes a very long time. -
The PG is currently *stuck in a degraded state*, so I’m unable to move it to other OSDs.
Current ceph version is reef 18.2.7.
Has anyone encountered a similar issue before or have any suggestions on how to troubleshoot and resolve it?
Thanks in advance! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582
Is it possible that this is a huge PG? What size does it have? But it could also be a faulty disk. Zitat von Sa Pham <saphi070@gmail.com>:
*Hello everyone,*
I’m running a Ceph cluster used as an RGW backend, and I’m facing an issue with one particular placement group (PG).
-
Accessing objects from this PG is *extremely slow*. -
Even running ceph pg <pg_id> takes a very long time. -
The PG is currently *stuck in a degraded state*, so I’m unable to move it to other OSDs.
Current ceph version is reef 18.2.7.
Has anyone encountered a similar issue before or have any suggestions on how to troubleshoot and resolve it?
Thanks in advance! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Eugen, This PG include: 254490 objects, size: 68095493667 bytes Regards, On Fri, Oct 3, 2025 at 9:10 PM Eugen Block <eblock@nde.ag> wrote:
Is it possible that this is a huge PG? What size does it have? But it could also be a faulty disk.
Zitat von Sa Pham <saphi070@gmail.com>:
*Hello everyone,*
I’m running a Ceph cluster used as an RGW backend, and I’m facing an issue with one particular placement group (PG).
-
Accessing objects from this PG is *extremely slow*. -
Even running ceph pg <pg_id> takes a very long time. -
The PG is currently *stuck in a degraded state*, so I’m unable to move it to other OSDs.
Current ceph version is reef 18.2.7.
Has anyone encountered a similar issue before or have any suggestions on how to troubleshoot and resolve it?
Thanks in advance! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582
Hi Sa, Regarding the output you provided, it appears that osd.302 is listed as UP but not ACTING for PG 18.773: PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 18.773 active+undersized+degraded+remapped+backfilling [302,150,138] 302 [130,101] 130 Have you already checked the logs for osd.302 and /var/log/messages for any I/O-related issues? Could you also try running 'ceph pg repeer 18.773'? If this is the only PG for which `osd.302` is not acting and the 'repeer' command does not resolve the issue, I would suggest attempting a deep-scrub on this PG. This might uncover errors that could potentially be fixed, either online or offline. Regards, Frédéric -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | frederic.nass@clyso.com Le lun. 6 oct. 2025 à 06:31, Sa Pham <saphi070@gmail.com> a écrit :
Hello Eugen,
This PG include: 254490 objects, size: 68095493667 bytes
Regards,
On Fri, Oct 3, 2025 at 9:10 PM Eugen Block <eblock@nde.ag> wrote:
Is it possible that this is a huge PG? What size does it have? But it could also be a faulty disk.
Zitat von Sa Pham <saphi070@gmail.com>:
*Hello everyone,*
I’m running a Ceph cluster used as an RGW backend, and I’m facing an issue with one particular placement group (PG).
-
Accessing objects from this PG is *extremely slow*. -
Even running ceph pg <pg_id> takes a very long time. -
The PG is currently *stuck in a degraded state*, so I’m unable to move it to other OSDs.
Current ceph version is reef 18.2.7.
Has anyone encountered a similar issue before or have any suggestions on how to troubleshoot and resolve it?
Thanks in advance! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frédéric, I tried to repeer and deep scrub, but it's not working. Have you already checked the logs for osd.302 and /var/log/messages for any I/O-related issues? => I checked , there is no I/O error/issue. Regards, On Mon, Oct 6, 2025 at 3:15 PM Frédéric Nass <frederic.nass@clyso.com> wrote:
Hi Sa,
Regarding the output you provided, it appears that osd.302 is listed as UP but not ACTING for PG 18.773:
PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 18.773 active+undersized+degraded+remapped+backfilling [302,150,138] 302 [130,101] 130
Have you already checked the logs for osd.302 and /var/log/messages for any I/O-related issues? Could you also try running 'ceph pg repeer 18.773'?
If this is the only PG for which `osd.302` is not acting and the 'repeer' command does not resolve the issue, I would suggest attempting a deep-scrub on this PG. This might uncover errors that could potentially be fixed, either online or offline.
Regards, Frédéric
-- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | frederic.nass@clyso.com
Le lun. 6 oct. 2025 à 06:31, Sa Pham <saphi070@gmail.com> a écrit :
Hello Eugen,
This PG include: 254490 objects, size: 68095493667 bytes
Regards,
On Fri, Oct 3, 2025 at 9:10 PM Eugen Block <eblock@nde.ag> wrote:
Is it possible that this is a huge PG? What size does it have? But it could also be a faulty disk.
Zitat von Sa Pham <saphi070@gmail.com>:
*Hello everyone,*
I’m running a Ceph cluster used as an RGW backend, and I’m facing an issue with one particular placement group (PG).
-
Accessing objects from this PG is *extremely slow*. -
Even running ceph pg <pg_id> takes a very long time. -
The PG is currently *stuck in a degraded state*, so I’m unable to move it to other OSDs.
Current ceph version is reef 18.2.7.
Has anyone encountered a similar issue before or have any suggestions on how to troubleshoot and resolve it?
Thanks in advance! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582
Could be an issue with the primary OSD which is now osd.130. Have you checked osd.130 for any errors? Maybe try restarting osd.130 and osd.302 one at a time and maybe 101 as well, waiting for ~all PGs to become active+clean between all restarts. Could you please share a ceph status? So we get a better view of the situation. Regards, Frédéric. -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | frederic.nass@clyso.com Le lun. 6 oct. 2025 à 14:19, Sa Pham <saphi070@gmail.com> a écrit :
Hi Frédéric,
I tried to repeer and deep scrub, but it's not working.
Have you already checked the logs for osd.302 and /var/log/messages for any I/O-related issues?
=> I checked , there is no I/O error/issue.
Regards,
On Mon, Oct 6, 2025 at 3:15 PM Frédéric Nass <frederic.nass@clyso.com> wrote:
Hi Sa,
Regarding the output you provided, it appears that osd.302 is listed as UP but not ACTING for PG 18.773:
PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 18.773 active+undersized+degraded+remapped+backfilling [302,150,138] 302 [130,101] 130
Have you already checked the logs for osd.302 and /var/log/messages for any I/O-related issues? Could you also try running 'ceph pg repeer 18.773'?
If this is the only PG for which `osd.302` is not acting and the 'repeer' command does not resolve the issue, I would suggest attempting a deep-scrub on this PG. This might uncover errors that could potentially be fixed, either online or offline.
Regards, Frédéric
-- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | frederic.nass@clyso.com
Le lun. 6 oct. 2025 à 06:31, Sa Pham <saphi070@gmail.com> a écrit :
Hello Eugen,
This PG include: 254490 objects, size: 68095493667 bytes
Regards,
On Fri, Oct 3, 2025 at 9:10 PM Eugen Block <eblock@nde.ag> wrote:
Is it possible that this is a huge PG? What size does it have? But it could also be a faulty disk.
Zitat von Sa Pham <saphi070@gmail.com>:
*Hello everyone,*
I’m running a Ceph cluster used as an RGW backend, and I’m facing an issue with one particular placement group (PG).
-
Accessing objects from this PG is *extremely slow*. -
Even running ceph pg <pg_id> takes a very long time. -
The PG is currently *stuck in a degraded state*, so I’m unable to move it to other OSDs.
Current ceph version is reef 18.2.7.
Has anyone encountered a similar issue before or have any suggestions on how to troubleshoot and resolve it?
Thanks in advance! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582
Hi Frédéric, I tried to restart every osd related to that PG many times but it didn't work. When the primary switch to OSD 101, we still have slow on OSD 101. So I think it’s not hardware issue Regards, On Mon, 6 Oct 2025 at 20:46 Frédéric Nass <frederic.nass@clyso.com> wrote:
Could be an issue with the primary OSD which is now osd.130. Have you checked osd.130 for any errors? Maybe try restarting osd.130 and osd.302 one at a time and maybe 101 as well, waiting for ~all PGs to become active+clean between all restarts.
Could you please share a ceph status? So we get a better view of the situation.
Regards, Frédéric.
-- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | frederic.nass@clyso.com
Le lun. 6 oct. 2025 à 14:19, Sa Pham <saphi070@gmail.com> a écrit :
Hi Frédéric,
I tried to repeer and deep scrub, but it's not working.
Have you already checked the logs for osd.302 and /var/log/messages for any I/O-related issues?
=> I checked , there is no I/O error/issue.
Regards,
On Mon, Oct 6, 2025 at 3:15 PM Frédéric Nass <frederic.nass@clyso.com> wrote:
Hi Sa,
Regarding the output you provided, it appears that osd.302 is listed as UP but not ACTING for PG 18.773:
PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 18.773 active+undersized+degraded+remapped+backfilling [302,150,138] 302 [130,101] 130
Have you already checked the logs for osd.302 and /var/log/messages for any I/O-related issues? Could you also try running 'ceph pg repeer 18.773'?
If this is the only PG for which `osd.302` is not acting and the 'repeer' command does not resolve the issue, I would suggest attempting a deep-scrub on this PG. This might uncover errors that could potentially be fixed, either online or offline.
Regards, Frédéric
-- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | frederic.nass@clyso.com
Le lun. 6 oct. 2025 à 06:31, Sa Pham <saphi070@gmail.com> a écrit :
Hello Eugen,
This PG include: 254490 objects, size: 68095493667 bytes
Regards,
On Fri, Oct 3, 2025 at 9:10 PM Eugen Block <eblock@nde.ag> wrote:
Is it possible that this is a huge PG? What size does it have? But it could also be a faulty disk.
Zitat von Sa Pham <saphi070@gmail.com>:
*Hello everyone,*
I’m running a Ceph cluster used as an RGW backend, and I’m facing an issue with one particular placement group (PG).
-
Accessing objects from this PG is *extremely slow*. -
Even running ceph pg <pg_id> takes a very long time. -
The PG is currently *stuck in a degraded state*, so I’m unable to move it to other OSDs.
Current ceph version is reef 18.2.7.
Has anyone encountered a similar issue before or have any suggestions on how to troubleshoot and resolve it?
Thanks in advance! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582
On Mon, Oct 6, 2025, 5:34 PM Sa Pham <saphi070@gmail.com> wrote:
Hi Frédéric,
I tried to restart every osd related to that PG many times but it didn't work.
When the primary switch to OSD 101, we still have slow on OSD 101. So I think it’s not hardware issue
Regards,
On Mon, 6 Oct 2025 at 20:46 Frédéric Nass <frederic.nass@clyso.com> wrote:
Could be an issue with the primary OSD which is now osd.130. Have you checked osd.130 for any errors? Maybe try restarting osd.130 and osd.302 one at a time and maybe 101 as well, waiting for ~all PGs to become active+clean between all restarts.
Could you please share a ceph status? So we get a better view of the situation.
Regards, Frédéric.
-- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | frederic.nass@clyso.com
Le lun. 6 oct. 2025 à 14:19, Sa Pham <saphi070@gmail.com> a écrit :
Hi Frédéric,
I tried to repeer and deep scrub, but it's not working.
Have you already checked the logs for osd.302 and /var/log/messages for any I/O-related issues?
=> I checked , there is no I/O error/issue.
Regards,
On Mon, Oct 6, 2025 at 3:15 PM Frédéric Nass <frederic.nass@clyso.com> wrote:
Hi Sa,
Regarding the output you provided, it appears that osd.302 is listed as UP but not ACTING for PG 18.773:
PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 18.773 active+undersized+degraded+remapped+backfilling [302,150,138] 302 [130,101] 130
Have you already checked the logs for osd.302 and /var/log/messages for any I/O-related issues? Could you also try running 'ceph pg repeer 18.773'?
If this is the only PG for which `osd.302` is not acting and the 'repeer' command does not resolve the issue, I would suggest attempting a deep-scrub on this PG. This might uncover errors that could potentially be fixed, either online or offline.
Regards, Frédéric
-- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | frederic.nass@clyso.com
Le lun. 6 oct. 2025 à 06:31, Sa Pham <saphi070@gmail.com> a écrit :
Hello Eugen,
This PG include: 254490 objects, size: 68095493667 bytes
Regards,
On Fri, Oct 3, 2025 at 9:10 PM Eugen Block <eblock@nde.ag> wrote:
Is it possible that this is a huge PG? What size does it have? But it could also be a faulty disk.
Zitat von Sa Pham <saphi070@gmail.com>:
> *Hello everyone,* > > I’m running a Ceph cluster used as an RGW backend, and I’m facing an issue > with one particular placement group (PG). > > > - > > Accessing objects from this PG is *extremely slow*. > - > > Even running ceph pg <pg_id> takes a very long time. > - > > The PG is currently *stuck in a degraded state*, so I’m unable to move > it to other OSDs. > > > Current ceph version is reef 18.2.7. > > Has anyone encountered a similar issue before or have any suggestions on > how to troubleshoot and resolve it? > > > Thanks in advance! > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, If it's a performance issue, particularly if it's manifesting as high CPU load, you can usually pinpoint what's going on based on which symbol(s) are hot according to `perf top -p <pid>`. If it's not CPU hot, `iostat` is worth a look to see if the kernel thinks the block device is busy. Barring either of those it gets a bit trickier to tease out, but first discern whether or not it's a resource issue and work backwards from there would be my advice. Cheers, Tyler
Hi Tyler, We do have a monitoring system visualized by grafana. I already checked CPU iowait and loadavg and CPU usage (user + system) as well, they are not high. So I don't think the problem is resource. Regards, On Tue, Oct 7, 2025 at 6:53 AM Tyler Stachecki <stachecki.tyler@gmail.com> wrote:
On Mon, Oct 6, 2025, 5:34 PM Sa Pham <saphi070@gmail.com> wrote:
Hi Frédéric,
I tried to restart every osd related to that PG many times but it didn't work.
When the primary switch to OSD 101, we still have slow on OSD 101. So I think it’s not hardware issue
Regards,
On Mon, 6 Oct 2025 at 20:46 Frédéric Nass <frederic.nass@clyso.com> wrote:
Could be an issue with the primary OSD which is now osd.130. Have you checked osd.130 for any errors? Maybe try restarting osd.130 and osd.302 one at a time and maybe 101 as well, waiting for ~all PGs to become active+clean between all restarts.
Could you please share a ceph status? So we get a better view of the situation.
Regards, Frédéric.
-- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | frederic.nass@clyso.com
Le lun. 6 oct. 2025 à 14:19, Sa Pham <saphi070@gmail.com> a écrit :
Hi Frédéric,
I tried to repeer and deep scrub, but it's not working.
Have you already checked the logs for osd.302 and /var/log/messages for any I/O-related issues?
=> I checked , there is no I/O error/issue.
Regards,
On Mon, Oct 6, 2025 at 3:15 PM Frédéric Nass <frederic.nass@clyso.com> wrote:
Hi Sa,
Regarding the output you provided, it appears that osd.302 is listed as UP but not ACTING for PG 18.773:
PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 18.773 active+undersized+degraded+remapped+backfilling [302,150,138] 302 [130,101] 130
Have you already checked the logs for osd.302 and /var/log/messages for any I/O-related issues? Could you also try running 'ceph pg repeer 18.773'?
If this is the only PG for which `osd.302` is not acting and the 'repeer' command does not resolve the issue, I would suggest attempting a deep-scrub on this PG. This might uncover errors that could potentially be fixed, either online or offline.
Regards, Frédéric
-- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | frederic.nass@clyso.com
Le lun. 6 oct. 2025 à 06:31, Sa Pham <saphi070@gmail.com> a écrit :
Hello Eugen,
This PG include: 254490 objects, size: 68095493667 bytes
Regards,
On Fri, Oct 3, 2025 at 9:10 PM Eugen Block <eblock@nde.ag> wrote:
> Is it possible that this is a huge PG? What size does it have? But it > could also be a faulty disk. > > > Zitat von Sa Pham <saphi070@gmail.com>: > > > *Hello everyone,* > > > > I’m running a Ceph cluster used as an RGW backend, and I’m facing an > issue > > with one particular placement group (PG). > > > > > > - > > > > Accessing objects from this PG is *extremely slow*. > > - > > > > Even running ceph pg <pg_id> takes a very long time. > > - > > > > The PG is currently *stuck in a degraded state*, so I’m unable to move > > it to other OSDs. > > > > > > Current ceph version is reef 18.2.7. > > > > Has anyone encountered a similar issue before or have any suggestions on > > how to troubleshoot and resolve it? > > > > > > Thanks in advance! > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-leave@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io >
-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi,
If it's a performance issue, particularly if it's manifesting as high CPU load, you can usually pinpoint what's going on based on which symbol(s) are hot according to `perf top -p <pid>`.
If it's not CPU hot, `iostat` is worth a look to see if the kernel thinks the block device is busy.
Barring either of those it gets a bit trickier to tease out, but first discern whether or not it's a resource issue and work backwards from there would be my advice.
Cheers, Tyler
-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582
Hello all, yesterday, We exported this pg from one OSD and imported to three others OSD and using pg-upmap to map to those acting OSDs ceph osd pg-upmap 18.773 11 52 111 But the pg is still slow when accessed (ceph pg 18.773 query) and the object in this PG is still Slow. The scrub process cannot finish. I tried to use ceph-objectstore-tool to repair the PG but it did not work. I don't know what I should do now? please advise. Regards, On Tue, Oct 7, 2025 at 6:57 AM Sa Pham <saphi070@gmail.com> wrote:
Hi Tyler,
We do have a monitoring system visualized by grafana. I already checked CPU iowait and loadavg and CPU usage (user + system) as well, they are not high.
So I don't think the problem is resource.
Regards,
On Tue, Oct 7, 2025 at 6:53 AM Tyler Stachecki <stachecki.tyler@gmail.com> wrote:
On Mon, Oct 6, 2025, 5:34 PM Sa Pham <saphi070@gmail.com> wrote:
Hi Frédéric,
I tried to restart every osd related to that PG many times but it didn't work.
When the primary switch to OSD 101, we still have slow on OSD 101. So I think it’s not hardware issue
Regards,
On Mon, 6 Oct 2025 at 20:46 Frédéric Nass <frederic.nass@clyso.com> wrote:
Could be an issue with the primary OSD which is now osd.130. Have you checked osd.130 for any errors? Maybe try restarting osd.130 and osd.302 one at a time and maybe 101 as well, waiting for ~all PGs to become active+clean between all restarts.
Could you please share a ceph status? So we get a better view of the situation.
Regards, Frédéric.
-- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | frederic.nass@clyso.com
Le lun. 6 oct. 2025 à 14:19, Sa Pham <saphi070@gmail.com> a écrit :
Hi Frédéric,
I tried to repeer and deep scrub, but it's not working.
Have you already checked the logs for osd.302 and /var/log/messages for any I/O-related issues?
=> I checked , there is no I/O error/issue.
Regards,
On Mon, Oct 6, 2025 at 3:15 PM Frédéric Nass <frederic.nass@clyso.com
wrote:
Hi Sa,
Regarding the output you provided, it appears that osd.302 is listed as UP but not ACTING for PG 18.773:
PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 18.773 active+undersized+degraded+remapped+backfilling [302,150,138] 302 [130,101] 130
Have you already checked the logs for osd.302 and /var/log/messages for any I/O-related issues? Could you also try running 'ceph pg repeer 18.773'?
If this is the only PG for which `osd.302` is not acting and the 'repeer' command does not resolve the issue, I would suggest attempting a deep-scrub on this PG. This might uncover errors that could potentially be fixed, either online or offline.
Regards, Frédéric
-- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | frederic.nass@clyso.com
Le lun. 6 oct. 2025 à 06:31, Sa Pham <saphi070@gmail.com> a écrit :
> Hello Eugen, > > > This PG include: 254490 objects, size: 68095493667 bytes > > > Regards, > > On Fri, Oct 3, 2025 at 9:10 PM Eugen Block <eblock@nde.ag> wrote: > > > Is it possible that this is a huge PG? What size does it have? But it > > could also be a faulty disk. > > > > > > Zitat von Sa Pham <saphi070@gmail.com>: > > > > > *Hello everyone,* > > > > > > I’m running a Ceph cluster used as an RGW backend, and I’m facing an > > issue > > > with one particular placement group (PG). > > > > > > > > > - > > > > > > Accessing objects from this PG is *extremely slow*. > > > - > > > > > > Even running ceph pg <pg_id> takes a very long time. > > > - > > > > > > The PG is currently *stuck in a degraded state*, so I’m unable > to move > > > it to other OSDs. > > > > > > > > > Current ceph version is reef 18.2.7. > > > > > > Has anyone encountered a similar issue before or have any > suggestions on > > > how to troubleshoot and resolve it? > > > > > > > > > Thanks in advance! > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users@ceph.io > > > To unsubscribe send an email to ceph-users-leave@ceph.io > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-leave@ceph.io > > > > > -- > Sa Pham Dang > Skype: great_bn > Phone/Telegram: 0986.849.582 > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io >
-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi,
If it's a performance issue, particularly if it's manifesting as high CPU load, you can usually pinpoint what's going on based on which symbol(s) are hot according to `perf top -p <pid>`.
If it's not CPU hot, `iostat` is worth a look to see if the kernel thinks the block device is busy.
Barring either of those it gets a bit trickier to tease out, but first discern whether or not it's a resource issue and work backwards from there would be my advice.
Cheers, Tyler
-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582
-- Sa Pham Dang Skype: great_bn Phone/Telegram: 0986.849.582
participants (5)
-
Eugen Block
-
Frédéric Nass
-
Joshua Blanch
-
Sa Pham
-
Tyler Stachecki