20 Feb
2024
20 Feb
'24
3:43 p.m.
Please don't drop the list from your response.
The first question coming to mind is, why do you have a cache-tier if
all your pools are on nvme decices anyway? I don't see any benefit here.
Did you try the suggested workaround and disable the cache-tier?
Zitat von Cedric <yipikai7(a)gmail.com>om>:
Thanks Eugen, see attached infos.
Some more details:
- commands that actually hangs: ceph balancer status ; rbd -p vms ls ;
rados -p vms_cache cache-flush-evict-all
- all scrub running on vms_caches pgs are stall / start in a loop
without actually doing anything
- all io are 0 both from ceph status or iostat on nodes
On Tue, Feb 20, 2024 at 10:00 AM Eugen Block <eblock(a)nde.ag> wrote:
>
> Hi,
>
> some more details would be helpful, for example what's the pool size
> of the cache pool? Did you issue a PG split before or during the
> upgrade? This thread [1] deals with the same problem, the described
> workaround was to set hit_set_count to 0 and disable the cache layer
> until that is resolved. Afterwards you could enable the cache layer
> again. But keep in mind that the code for cache tier is entirely
> removed in Reef (IIRC).
>
> Regards,
> Eugen
>
> [1]
>
https://ceph-users.ceph.narkive.com/zChyOq5D/ceph-strange-issue-after-addin…
>
> Zitat von Cedric <yipikai7(a)gmail.com>om>:
>
> > Hello,
> >
> > Following an upgrade from Nautilus (14.2.22) to Pacific (16.2.13), we
> > encounter an issue with a cache pool becoming completely stuck,
> > relevant messages below:
> >
> > pg xx.x has invalid (post-split) stats; must scrub before tier agent
> > can activate
> >
> > In OSD logs, scrubs are starting in a loop without succeeding for all
> > pg of this pool.
> >
> > What we already tried without luck so far:
> >
> > - shutdown / restart OSD
> > - rebalance pg between OSD
> > - raise the memory on OSD
> > - repeer PG
> >
> > Any idea what is causing this? any help will be greatly appreciated
> >
> > Thanks
> >
> > Cédric
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
20 Feb
20 Feb
5:19 p.m.
New subject: Scrub stuck and 'pg has invalid (post-split) stat'
Hi,
A bit of history might help to understand why we have the cache tier.
We run openstack on top ceph since many years now (started with mimic, then an upgrade to
nautilus (years 2 ago) and today and upgrade to pacific). At the beginning of the setup,
we used to have a mix of hdd+ssd devices in HCI mode for openstack nova. After the upgrade
to nautilus, we made a hardware refresh with brand new NVME devices. And transitionned
from mixed devices to nvme. But we were never able to evict all the data from the
vms_cache pools (even with being aggressive with the eviction; the last resort would have
been to stop all the virtual instances, and that was not an option for our customers), so
we decided to move on and set cache-mode proxy and serve data with only nvme since then.
And it's been like this for 1 years and a half.
But today, after the upgrade, the situation is that we cannot query any stats (with ceph
pg x.x query), rados query hangs, scrub hangs even though all PGs are
"active+clean". and there is no client activity reported by the cluster.
Recovery, and rebalance. Also some other commands hangs, ie: "ceph balancer
status".
--------------
bash-4.2$ ceph -s
cluster:
id: <fsid>
health: HEALTH_WARN
mon is allowing insecure global_id reclaim
noscrub,nodeep-scrub,nosnaptrim flag(s) set
18432 slow ops, oldest one blocked for 7626 sec, daemons
[osd.0,osd.1,osd.10,osd.11,osd.112,osd.113,osd.118,osd.119,osd.120,osd.122]... have slow
ops.
services:
mon: 3 daemons, quorum mon1,mon2,mon3(age 36m)
mgr: bm9612541(active, since 39m)
osd: 72 osds: 72 up (since 97m), 72 in (since 9h)
flags noscrub,nodeep-scrub,nosnaptrim
data:
pools: 8 pools, 2409 pgs
objects: 14.64M objects, 92 TiB
usage: 276 TiB used, 143 TiB / 419 TiB avail
pgs: 2409 active+clean
26 Feb
26 Feb
12:57 p.m.
New subject: Scrub stuck and 'pg has invalid (post-split) stat'
Hi,
thanks for the context. Was there any progress over the weekend? The
hanging commands seem to be MGR related, and there's only one in your
cluster according to your output. Can you deploy a second one
manually, then adopt it with cephadm? Can you add 'ceph versions' as
well?
Zitat von florian.leduc(a)socgen.com:
Hi,
A bit of history might help to understand why we have the cache tier.
We run openstack on top ceph since many years now (started with
mimic, then an upgrade to nautilus (years 2 ago) and today and
upgrade to pacific). At the beginning of the setup, we used to have
a mix of hdd+ssd devices in HCI mode for openstack nova. After the
upgrade to nautilus, we made a hardware refresh with brand new NVME
devices. And transitionned from mixed devices to nvme. But we were
never able to evict all the data from the vms_cache pools (even with
being aggressive with the eviction; the last resort would have been
to stop all the virtual instances, and that was not an option for
our customers), so we decided to move on and set cache-mode proxy
and serve data with only nvme since then. And it's been like this
for 1 years and a half.
But today, after the upgrade, the situation is that we cannot query
any stats (with ceph pg x.x query), rados query hangs, scrub hangs
even though all PGs are "active+clean". and there is no client
activity reported by the cluster. Recovery, and rebalance. Also some
other commands hangs, ie: "ceph balancer status".
--------------
bash-4.2$ ceph -s
cluster:
id: <fsid>
health: HEALTH_WARN
mon is allowing insecure global_id reclaim
noscrub,nodeep-scrub,nosnaptrim flag(s) set
18432 slow ops, oldest one blocked for 7626 sec, daemons
[osd.0,osd.1,osd.10,osd.11,osd.112,osd.113,osd.118,osd.119,osd.120,osd.122]... have slow
ops.
services:
mon: 3 daemons, quorum mon1,mon2,mon3(age 36m)
mgr: bm9612541(active, since 39m)
osd: 72 osds: 72 up (since 97m), 72 in (since 9h)
flags noscrub,nodeep-scrub,nosnaptrim
data:
pools: 8 pools, 2409 pgs
objects: 14.64M objects, 92 TiB
usage: 276 TiB used, 143 TiB / 419 TiB avail
pgs: 2409 active+clean
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
28 Feb
28 Feb
10:38 p.m.
New subject: Scrub stuck and 'pg has invalid (post-split) stat'
Hello,
Sorry for the late reply, so yes we finally find a solution, which was to split apart the
cache_pool on dedicated OSD. It had the effect to clear off slow ops and allow the cluster
to serves clients again, after 5 days of lock down, hopefully the majority of VM resume
well, thanks to the virtio driver that does not seems to have any timeout.
It seems that at least one of the main culprit was to store both cold and hot data pool on
same OSD (which in the end totally make sens), maybe some others actions engaged also had
an effect, we are still trying to trouble shoot the root of slow ops, weirdly it was the
5th cluster upgraded and all as almost the same configuration, but this one handles 5x
time more workload.
In the hope it could help.
Cédric
On 26 Feb 2024, at 10:57, Eugen Block
<eblock(a)nde.ag> wrote:
Hi,
thanks for the context. Was there any progress over the weekend? The hanging commands
seem to be MGR related, and there's only one in your cluster according to your output.
Can you deploy a second one manually, then adopt it with cephadm? Can you add 'ceph
versions' as well?
Zitat von florian.leduc(a)socgen.com:
Hi,
A bit of history might help to understand why we have the cache tier.
We run openstack on top ceph since many years now (started with mimic, then an upgrade to
nautilus (years 2 ago) and today and upgrade to pacific). At the beginning of the setup,
we used to have a mix of hdd+ssd devices in HCI mode for openstack nova. After the upgrade
to nautilus, we made a hardware refresh with brand new NVME devices. And transitionned
from mixed devices to nvme. But we were never able to evict all the data from the
vms_cache pools (even with being aggressive with the eviction; the last resort would have
been to stop all the virtual instances, and that was not an option for our customers), so
we decided to move on and set cache-mode proxy and serve data with only nvme since then.
And it's been like this for 1 years and a half.
But today, after the upgrade, the situation is that we cannot query any stats (with ceph
pg x.x query), rados query hangs, scrub hangs even though all PGs are
"active+clean". and there is no client activity reported by the cluster.
Recovery, and rebalance. Also some other commands hangs, ie: "ceph balancer
status".
--------------
bash-4.2$ ceph -s
cluster:
id: <fsid>
health: HEALTH_WARN
mon is allowing insecure global_id reclaim
noscrub,nodeep-scrub,nosnaptrim flag(s) set
18432 slow ops, oldest one blocked for 7626 sec, daemons
[osd.0,osd.1,osd.10,osd.11,osd.112,osd.113,osd.118,osd.119,osd.120,osd.122]... have slow
ops.
services:
mon: 3 daemons, quorum mon1,mon2,mon3(age 36m)
mgr: bm9612541(active, since 39m)
osd: 72 osds: 72 up (since 97m), 72 in (since 9h)
flags noscrub,nodeep-scrub,nosnaptrim
data:
pools: 8 pools, 2409 pgs
objects: 14.64M objects, 92 TiB
usage: 276 TiB used, 143 TiB / 419 TiB avail
pgs: 2409 active+clean
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:50 p.m.
New subject: Scrub stuck and 'pg has invalid (post-split) stat'
Hi,
great that you found a solution. Maybe that also helps to get rid of
the cache-tier entirely?
Zitat von Cedric <yipikai7(a)gmail.com>om>:
Hello,
Sorry for the late reply, so yes we finally find a solution, which
was to split apart the cache_pool on dedicated OSD. It had the
effect to clear off slow ops and allow the cluster to serves clients
again, after 5 days of lock down, hopefully the majority of VM
resume well, thanks to the virtio driver that does not seems to have
any timeout.
It seems that at least one of the main culprit was to store both
cold and hot data pool on same OSD (which in the end totally make
sens), maybe some others actions engaged also had an effect, we are
still trying to trouble shoot the root of slow ops, weirdly it was
the 5th cluster upgraded and all as almost the same configuration,
but this one handles 5x time more workload.
In the hope it could help.
Cédric
> On 26 Feb 2024, at 10:57, Eugen Block <eblock(a)nde.ag> wrote:
>
> Hi,
>
> thanks for the context. Was there any progress over the weekend?
> The hanging commands seem to be MGR related, and there's only one
> in your cluster according to your output. Can you deploy a second
> one manually, then adopt it with cephadm? Can you add 'ceph
> versions' as well?
>
>
> Zitat von florian.leduc(a)socgen.com:
>
>> Hi,
>> A bit of history might help to understand why we have the cache tier.
>>
>> We run openstack on top ceph since many years now (started with
>> mimic, then an upgrade to nautilus (years 2 ago) and today and
>> upgrade to pacific). At the beginning of the setup, we used to
>> have a mix of hdd+ssd devices in HCI mode for openstack nova.
>> After the upgrade to nautilus, we made a hardware refresh with
>> brand new NVME devices. And transitionned from mixed devices to
>> nvme. But we were never able to evict all the data from the
>> vms_cache pools (even with being aggressive with the eviction; the
>> last resort would have been to stop all the virtual instances, and
>> that was not an option for our customers), so we decided to move
>> on and set cache-mode proxy and serve data with only nvme since
>> then. And it's been like this for 1 years and a half.
>>
>> But today, after the upgrade, the situation is that we cannot
>> query any stats (with ceph pg x.x query), rados query hangs, scrub
>> hangs even though all PGs are "active+clean". and there is no
>> client activity reported by the cluster. Recovery, and rebalance.
>> Also some other commands hangs, ie: "ceph balancer status".
>>
>> --------------
>> bash-4.2$ ceph -s
>> cluster:
>> id: <fsid>
>> health: HEALTH_WARN
>> mon is allowing insecure global_id reclaim
>> noscrub,nodeep-scrub,nosnaptrim flag(s) set
>> 18432 slow ops, oldest one blocked for 7626 sec,
>> daemons
>> [osd.0,osd.1,osd.10,osd.11,osd.112,osd.113,osd.118,osd.119,osd.120,osd.122]...
have slow
>> ops.
>>
>> services:
>> mon: 3 daemons, quorum mon1,mon2,mon3(age 36m)
>> mgr: bm9612541(active, since 39m)
>> osd: 72 osds: 72 up (since 97m), 72 in (since 9h)
>> flags noscrub,nodeep-scrub,nosnaptrim
>>
>> data:
>> pools: 8 pools, 2409 pgs
>> objects: 14.64M objects, 92 TiB
>> usage: 276 TiB used, 143 TiB / 419 TiB avail
>> pgs: 2409 active+clean
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1 Mar
1 Mar
3:31 p.m.
New subject: Scrub stuck and 'pg has invalid (post-split) stat'
Not really, as unfortunately the cache eviction fails for some rbd
objects that still hace some "lock", right now we need to understand
why the eviction fails on these objects, and find a solution to have
the cache eviction fully working. I will provide more information
later on.
If you have any pointers, well they will be greatly appreciated.
Cheers
On Wed, Feb 28, 2024 at 9:50 PM Eugen Block <eblock(a)nde.ag> wrote:
>
> Hi,
>
> great that you found a solution. Maybe that also helps to get rid of
> the cache-tier entirely?
>
> Zitat von Cedric <yipikai7(a)gmail.com>om>:
>
> > Hello,
> >
> > Sorry for the late reply, so yes we finally find a solution, which
> > was to split apart the cache_pool on dedicated OSD. It had the
> > effect to clear off slow ops and allow the cluster to serves clients
> > again, after 5 days of lock down, hopefully the majority of VM
> > resume well, thanks to the virtio driver that does not seems to have
> > any timeout.
> >
> > It seems that at least one of the main culprit was to store both
> > cold and hot data pool on same OSD (which in the end totally make
> > sens), maybe some others actions engaged also had an effect, we are
> > still trying to trouble shoot the root of slow ops, weirdly it was
> > the 5th cluster upgraded and all as almost the same configuration,
> > but this one handles 5x time more workload.
> >
> > In the hope it could help.
> >
> > Cédric
> >
> >> On 26 Feb 2024, at 10:57, Eugen Block <eblock(a)nde.ag> wrote:
> >>
> >> Hi,
> >>
> >> thanks for the context. Was there any progress over the weekend?
> >> The hanging commands seem to be MGR related, and there's only one
> >> in your cluster according to your output. Can you deploy a second
> >> one manually, then adopt it with cephadm? Can you add 'ceph
> >> versions' as well?
> >>
> >>
> >> Zitat von florian.leduc(a)socgen.com:
> >>
> >>> Hi,
> >>> A bit of history might help to understand why we have the cache tier.
> >>>
> >>> We run openstack on top ceph since many years now (started with
> >>> mimic, then an upgrade to nautilus (years 2 ago) and today and
> >>> upgrade to pacific). At the beginning of the setup, we used to
> >>> have a mix of hdd+ssd devices in HCI mode for openstack nova.
> >>> After the upgrade to nautilus, we made a hardware refresh with
> >>> brand new NVME devices. And transitionned from mixed devices to
> >>> nvme. But we were never able to evict all the data from the
> >>> vms_cache pools (even with being aggressive with the eviction; the
> >>> last resort would have been to stop all the virtual instances, and
> >>> that was not an option for our customers), so we decided to move
> >>> on and set cache-mode proxy and serve data with only nvme since
> >>> then. And it's been like this for 1 years and a half.
> >>>
> >>> But today, after the upgrade, the situation is that we cannot
> >>> query any stats (with ceph pg x.x query), rados query hangs, scrub
> >>> hangs even though all PGs are "active+clean". and there is no
> >>> client activity reported by the cluster. Recovery, and rebalance.
> >>> Also some other commands hangs, ie: "ceph balancer status".
> >>>
> >>> --------------
> >>> bash-4.2$ ceph -s
> >>> cluster:
> >>> id: <fsid>
> >>> health: HEALTH_WARN
> >>> mon is allowing insecure global_id reclaim
> >>> noscrub,nodeep-scrub,nosnaptrim flag(s) set
> >>> 18432 slow ops, oldest one blocked for 7626 sec,
> >>> daemons
> >>>
[osd.0,osd.1,osd.10,osd.11,osd.112,osd.113,osd.118,osd.119,osd.120,osd.122]... have slow
> >>> ops.
> >>>
> >>> services:
> >>> mon: 3 daemons, quorum mon1,mon2,mon3(age 36m)
> >>> mgr: bm9612541(active, since 39m)
> >>> osd: 72 osds: 72 up (since 97m), 72 in (since 9h)
> >>> flags noscrub,nodeep-scrub,nosnaptrim
> >>>
> >>> data:
> >>> pools: 8 pools, 2409 pgs
> >>> objects: 14.64M objects, 92 TiB
> >>> usage: 276 TiB used, 143 TiB / 419 TiB avail
> >>> pgs: 2409 active+clean
> >>> _______________________________________________
> >>> ceph-users mailing list -- ceph-users(a)ceph.io
> >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >>
> >>
> >> _______________________________________________
> >> ceph-users mailing list -- ceph-users(a)ceph.io
> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
>
79
days inactive
89
days old
5 comments
3 participants
participants (3)
-
Cedric -
Eugen Block -
florian.leduc＠socgen.com