28 May
2020
28 May
'20
2:39 p.m.
Hi,
I had configured a cache tier with max object counts 500k. But no evict
happens when the object counts hit the configured maximum.
Anyone experienced this issue? What should I do?
$ ceph health detail
HEALTH_WARN 1 cache pools at or near target size
CACHE_POOL_NEAR_FULL 1 cache pools at or near target size
cache pool 'cached-hdd-cache' with 887.11k objects at/near target max
500k objects
$ ceph df | grep -e "POOL\|cached-hdd"
POOLS:
POOL ID STORED OBJECTS USED %USED
MAX AVAIL
cached-hdd 24 1.4 TiB 1.52M 1.4 TiB 0.60
78 TiB
cached-hdd-cache 25 842 GiB 887.14k 842 GiB 15.97
1.4 TiB
$ ceph osd pool get cached-hdd-cache all
size: 3
min_size: 1
pg_num: 128
pgp_num: 128
crush_rule: nvme-repl-rule
hashpspool: true
nodelete: false
nopgchange: false
nosizechange: false
write_fadvise_dontneed: false
noscrub: false
nodeep-scrub: false
hit_set_type: bloom
hit_set_period: 1200
hit_set_count: 4
hit_set_fpp: 0.05
use_gmt_hitset: 1
target_max_objects: 500000
target_max_bytes: 1099511627776
cache_target_dirty_ratio: 0
cache_target_dirty_high_ratio: 0.7
cache_target_full_ratio: 0.9
cache_min_flush_age: 0
cache_min_evict_age: 0
min_read_recency_for_promote: 1
min_write_recency_for_promote: 1
fast_read: 0
hit_set_grade_decay_rate: 20
hit_set_search_last_n: 1
pg_autoscale_mode: warn
Regs,
Icy
28 May
28 May
2:55 p.m.
I don't see a cache_mode enabled on the pool, did you set one?
Zitat von icy chan <icy.kf.chan(a)gmail.com>om>:
Hi,
I had configured a cache tier with max object counts 500k. But no evict
happens when the object counts hit the configured maximum.
Anyone experienced this issue? What should I do?
$ ceph health detail
HEALTH_WARN 1 cache pools at or near target size
CACHE_POOL_NEAR_FULL 1 cache pools at or near target size
cache pool 'cached-hdd-cache' with 887.11k objects at/near target max
500k objects
$ ceph df | grep -e "POOL\|cached-hdd"
POOLS:
POOL ID STORED OBJECTS USED %USED
MAX AVAIL
cached-hdd 24 1.4 TiB 1.52M 1.4 TiB 0.60
78 TiB
cached-hdd-cache 25 842 GiB 887.14k 842 GiB 15.97
1.4 TiB
$ ceph osd pool get cached-hdd-cache all
size: 3
min_size: 1
pg_num: 128
pgp_num: 128
crush_rule: nvme-repl-rule
hashpspool: true
nodelete: false
nopgchange: false
nosizechange: false
write_fadvise_dontneed: false
noscrub: false
nodeep-scrub: false
hit_set_type: bloom
hit_set_period: 1200
hit_set_count: 4
hit_set_fpp: 0.05
use_gmt_hitset: 1
target_max_objects: 500000
target_max_bytes: 1099511627776
cache_target_dirty_ratio: 0
cache_target_dirty_high_ratio: 0.7
cache_target_full_ratio: 0.9
cache_min_flush_age: 0
cache_min_evict_age: 0
min_read_recency_for_promote: 1
min_write_recency_for_promote: 1
fast_read: 0
hit_set_grade_decay_rate: 20
hit_set_search_last_n: 1
pg_autoscale_mode: warn
Regs,
Icy
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
29 May
29 May
8:40 a.m.
Hi Eugen,
Sorry for the missing information. "cached-hdd-cache" is the overlay tier
of "cached-hdd" and configured in "readproxy" mode.
$ ceph osd dump | grep cached-hdd
pool 24 'cached-hdd' replicated size 3 min_size 2 crush_rule 1 object_hash
rjenkins pg_num 512 pgp_num 512 autoscale_mode warn last_change 27242 lfor
22992/22992/22992 flags hashpspool,selfmanaged_snaps tiers 25 read_tier 25
write_tier 25 stripe_width 0 application rbd
pool 25 'cached-hdd-cache' replicated size 3 min_size 1 crush_rule 3
object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change
30376 lfor 22992/30306/30304 flags
hashpspool,incomplete_clones,selfmanaged_snaps tier_of 24 cache_mode
readproxy target_bytes 1099511627776 target_objects 500000 hit_set
bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 1200s x4
decay_rate 20 search_last_n 1 min_read_recency_for_promote 1
min_write_recency_for_promote 1 stripe_width 0
Regs,
Icy
On Thu, 28 May 2020 at 18:25, Eugen Block <eblock(a)nde.ag> wrote:
I don't see a cache_mode enabled on the pool, did
you set one?
Zitat von icy chan <icy.kf.chan(a)gmail.com>om>:
Hi,
I had configured a cache tier with max object counts 500k. But no evict
happens when the object counts hit the configured maximum.
Anyone experienced this issue? What should I do?
$ ceph health detail
HEALTH_WARN 1 cache pools at or near target size
CACHE_POOL_NEAR_FULL 1 cache pools at or near target size
cache pool 'cached-hdd-cache' with 887.11k objects at/near target max
500k objects
$ ceph df | grep -e "POOL\|cached-hdd"
POOLS:
POOL ID STORED OBJECTS USED %USED
MAX AVAIL
cached-hdd 24 1.4 TiB 1.52M 1.4 TiB 0.60
78 TiB
cached-hdd-cache 25 842 GiB 887.14k 842 GiB 15.97
1.4 TiB
$ ceph osd pool get cached-hdd-cache all
size: 3
min_size: 1
pg_num: 128
pgp_num: 128
crush_rule: nvme-repl-rule
hashpspool: true
nodelete: false
nopgchange: false
nosizechange: false
write_fadvise_dontneed: false
noscrub: false
nodeep-scrub: false
hit_set_type: bloom
hit_set_period: 1200
hit_set_count: 4
hit_set_fpp: 0.05
use_gmt_hitset: 1
target_max_objects: 500000
target_max_bytes: 1099511627776
cache_target_dirty_ratio: 0
cache_target_dirty_high_ratio: 0.7
cache_target_full_ratio: 0.9
cache_min_flush_age: 0
cache_min_evict_age: 0
min_read_recency_for_promote: 1
min_write_recency_for_promote: 1
fast_read: 0
hit_set_grade_decay_rate: 20
hit_set_search_last_n: 1
pg_autoscale_mode: warn
Regs,
Icy
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:54 p.m.
Can you manually flush/evict the cache? Maybe reduce max_target_bytes
and max_target_objects to see if that triggers anything. We use
cache_mode writeback, maybe give that a try?
I don't see many differences between our and your cache tier config,
except for cache_mode and we don't have a max_target_objects limit
set, only max_target_bytes. Is it possible that none of the objects
are getting marked dirty?
Do you see "dirty" entries in ceph df detail?
Zitat von icy chan <icy.kf.chan(a)gmail.com>om>:
Hi Eugen,
Sorry for the missing information. "cached-hdd-cache" is the overlay tier
of "cached-hdd" and configured in "readproxy" mode.
$ ceph osd dump | grep cached-hdd
pool 24 'cached-hdd' replicated size 3 min_size 2 crush_rule 1 object_hash
rjenkins pg_num 512 pgp_num 512 autoscale_mode warn last_change 27242 lfor
22992/22992/22992 flags hashpspool,selfmanaged_snaps tiers 25 read_tier 25
write_tier 25 stripe_width 0 application rbd
pool 25 'cached-hdd-cache' replicated size 3 min_size 1 crush_rule 3
object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change
30376 lfor 22992/30306/30304 flags
hashpspool,incomplete_clones,selfmanaged_snaps tier_of 24 cache_mode
readproxy target_bytes 1099511627776 target_objects 500000 hit_set
bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 1200s x4
decay_rate 20 search_last_n 1 min_read_recency_for_promote 1
min_write_recency_for_promote 1 stripe_width 0
Regs,
Icy
On Thu, 28 May 2020 at 18:25, Eugen Block <eblock(a)nde.ag> wrote:
> I don't see a cache_mode enabled on the pool, did you set one?
>
>
>
> Zitat von icy chan <icy.kf.chan(a)gmail.com>om>:
>
> > Hi,
> >
> > I had configured a cache tier with max object counts 500k. But no evict
> > happens when the object counts hit the configured maximum.
> > Anyone experienced this issue? What should I do?
> >
> > $ ceph health detail
> > HEALTH_WARN 1 cache pools at or near target size
> > CACHE_POOL_NEAR_FULL 1 cache pools at or near target size
> > cache pool 'cached-hdd-cache' with 887.11k objects at/near target
max
> > 500k objects
> >
> > $ ceph df | grep -e "POOL\|cached-hdd"
> > POOLS:
> > POOL ID STORED OBJECTS USED %USED
> > MAX AVAIL
> > cached-hdd 24 1.4 TiB 1.52M 1.4 TiB 0.60
> > 78 TiB
> > cached-hdd-cache 25 842 GiB 887.14k 842 GiB 15.97
> > 1.4 TiB
> >
> > $ ceph osd pool get cached-hdd-cache all
> > size: 3
> > min_size: 1
> > pg_num: 128
> > pgp_num: 128
> > crush_rule: nvme-repl-rule
> > hashpspool: true
> > nodelete: false
> > nopgchange: false
> > nosizechange: false
> > write_fadvise_dontneed: false
> > noscrub: false
> > nodeep-scrub: false
> > hit_set_type: bloom
> > hit_set_period: 1200
> > hit_set_count: 4
> > hit_set_fpp: 0.05
> > use_gmt_hitset: 1
> > target_max_objects: 500000
> > target_max_bytes: 1099511627776
> > cache_target_dirty_ratio: 0
> > cache_target_dirty_high_ratio: 0.7
> > cache_target_full_ratio: 0.9
> > cache_min_flush_age: 0
> > cache_min_evict_age: 0
> > min_read_recency_for_promote: 1
> > min_write_recency_for_promote: 1
> > fast_read: 0
> > hit_set_grade_decay_rate: 20
> > hit_set_search_last_n: 1
> > pg_autoscale_mode: warn
> >
> >
> > Regs,
> > Icy
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
1 Jun
1 Jun
8:37 a.m.
Hi Eugen,
I think I found the root cause. The actual object numbers are below 500k
that I configured. But there are some ghost objects from "ceph df"
triggers the alarm.
I will keep monitor it and post result here.
$ rados -p cached-hdd-cache ls | wc -l
444654
$ ceph df | grep -e "POOL\|cached-hdd"
POOLS:
POOL ID STORED OBJECTS USED %USED
MAX AVAIL
cached-hdd 24 1.4 TiB 1.52M 4.3 TiB 1.79
78 TiB
cached-hdd-cache 25 399 GiB 890.80k 798 GiB 14.21
2.4 TiB
This similar to one previous case from me:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/JUSZC6UNOT…
Regs,
Icy
On Fri, 29 May 2020 at 20:24, Eugen Block <eblock(a)nde.ag> wrote:
Can you manually flush/evict the cache? Maybe reduce
max_target_bytes
and max_target_objects to see if that triggers anything. We use
cache_mode writeback, maybe give that a try?
I don't see many differences between our and your cache tier config,
except for cache_mode and we don't have a max_target_objects limit
set, only max_target_bytes. Is it possible that none of the objects
are getting marked dirty?
Do you see "dirty" entries in ceph df detail?
Zitat von icy chan <icy.kf.chan(a)gmail.com>om>:
Hi Eugen,
Sorry for the missing information. "cached-hdd-cache" is the overlay tier
of "cached-hdd" and configured in "readproxy" mode.
$ ceph osd dump | grep cached-hdd
pool 24 'cached-hdd' replicated size 3 min_size 2 crush_rule 1
object_hash
rjenkins pg_num 512 pgp_num 512 autoscale_mode
warn last_change 27242
lfor
22992/22992/22992 flags
hashpspool,selfmanaged_snaps tiers 25 read_tier
25
write_tier 25 stripe_width 0 application rbd
pool 25 'cached-hdd-cache' replicated size 3 min_size 1 crush_rule 3
object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode warn
last_change
30376 lfor 22992/30306/30304 flags
hashpspool,incomplete_clones,selfmanaged_snaps tier_of 24 cache_mode
readproxy target_bytes 1099511627776 target_objects 500000 hit_set
bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 1200s x4
decay_rate 20 search_last_n 1 min_read_recency_for_promote 1
min_write_recency_for_promote 1 stripe_width 0
Regs,
Icy
On Thu, 28 May 2020 at 18:25, Eugen Block <eblock(a)nde.ag> wrote:
> I don't see a cache_mode enabled on the pool, did you set one?
>
>
>
> Zitat von icy chan <icy.kf.chan(a)gmail.com>om>:
>
> > Hi,
> >
> > I had configured a cache tier with max object counts 500k. But no
evict
> > happens when the object counts hit the
configured maximum.
> > Anyone experienced this issue? What should I do?
> >
> > $ ceph health detail
> > HEALTH_WARN 1 cache pools at or near target size
> > CACHE_POOL_NEAR_FULL 1 cache pools at or near target size
> > cache pool 'cached-hdd-cache' with 887.11k objects at/near target
max
> > 500k objects
> >
> > $ ceph df | grep -e "POOL\|cached-hdd"
> > POOLS:
> > POOL ID STORED OBJECTS USED
%USED
> > MAX AVAIL
> > cached-hdd 24 1.4 TiB 1.52M 1.4 TiB
0.60
> > 78 TiB
> > cached-hdd-cache 25 842 GiB 887.14k 842 GiB
15.97
> > 1.4 TiB
> >
> > $ ceph osd pool get cached-hdd-cache all
> > size: 3
> > min_size: 1
> > pg_num: 128
> > pgp_num: 128
> > crush_rule: nvme-repl-rule
> > hashpspool: true
> > nodelete: false
> > nopgchange: false
> > nosizechange: false
> > write_fadvise_dontneed: false
> > noscrub: false
> > nodeep-scrub: false
> > hit_set_type: bloom
> > hit_set_period: 1200
> > hit_set_count: 4
> > hit_set_fpp: 0.05
> > use_gmt_hitset: 1
> > target_max_objects: 500000
> > target_max_bytes: 1099511627776
> > cache_target_dirty_ratio: 0
> > cache_target_dirty_high_ratio: 0.7
> > cache_target_full_ratio: 0.9
> > cache_min_flush_age: 0
> > cache_min_evict_age: 0
> > min_read_recency_for_promote: 1
> > min_write_recency_for_promote: 1
> > fast_read: 0
> > hit_set_grade_decay_rate: 20
> > hit_set_search_last_n: 1
> > pg_autoscale_mode: warn
> >
> >
> > Regs,
> > Icy
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
1442
days inactive
1446
days old
4 comments
2 participants
participants (2)
-
Eugen Block -
icy chan