19 May
2020
19 May
'20
11:52 p.m.
Hi,
I'm using Nautilus and I'm using the whole cluster mainly for a single
bucket in RadosGW.
There is a lot of data in this bucket (Petabyte scale) and I don't want to
waste all of SSD on it.
Is there anyway to automatically set some aging threshold for this data and
e.g. move any data older than a month to HDD OSDs?
Does anyone has experience with this:
Pool Placement and Storage Classes:
https://docs.ceph.com/docs/master/radosgw/placement/
But something automatic would be much better for me in this case.
Any help would be appreciated.
Thanks a lot,
Khodayar
11:30 a.m.
Tiered pools should be able to do this for you.
It has been dis-encouraged as a performance gain (ie, the reverse when you
have spin drives and want to put a ssd pool in front of it to get ssd perf
but hdd price/storage) in some cases, but if you do it for migrations it
should probably be worth it, given that a slight perf hit on the backend
drives would be acceptable to be able to save money compared to all-ssd
pools I guess.
https://docs.ceph.com/docs/nautilus/rados/operations/cache-tiering/
Den ons 20 maj 2020 kl 11:12 skrev Khodayar Doustar <doustar(a)rayanexon.ir>ir>:
Anyone knows anything about this?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
May the most significant bit of your life be positive.
11:57 a.m.
Hi Khodayar,
Setting placement policies is probably not what you're looking for.
I've used placement policies successfully to separate an HDD pool from an
SSD pool. However, this policy only applies to new data if it is set. You
would have to read it out and write it back in at the s3 level using your
new policy.
Also, I think (but not sure) that policies are applied at the bucket level,
so you would need a second bucket.
Cheers,
Tom
On Tue, May 19, 2020 at 11:54 PM Khodayar Doustar <doustar(a)rayanexon.ir>
wrote:
Hi,
I'm using Nautilus and I'm using the whole cluster mainly for a single
bucket in RadosGW.
There is a lot of data in this bucket (Petabyte scale) and I don't want to
waste all of SSD on it.
Is there anyway to automatically set some aging threshold for this data and
e.g. move any data older than a month to HDD OSDs?
Does anyone has experience with this:
Pool Placement and Storage Classes:
https://docs.ceph.com/docs/master/radosgw/placement/
But something automatic would be much better for me in this case.
Any help would be appreciated.
Thanks a lot,
Khodayar
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Thomas Bennett
Storage Engineer at SARAO
1:15 p.m.
Thomas,
Yes, you are correct. I would have to move objects manually between (more
than one) buckets if I use "Pool placements and Storage classes"
So you have successfully used this method and it was OK? I may be forced to
use this method because clients needs more features than mere cache
tiering.
Have you ever used S3 Browser "bucket lifecycle rules" and those "storage
classes"?
Thanks,
Khodayar
On Wed, May 20, 2020 at 11:57 AM Thomas Bennett <thomas(a)ska.ac.za> wrote:
Hi Khodayar,
Setting placement policies is probably not what you're looking for.
I've used placement policies successfully to separate an HDD pool from an
SSD pool. However, this policy only applies to new data if it is set. You
would have to read it out and write it back in at the s3 level using your
new policy.
Also, I think (but not sure) that policies are applied at the bucket
level, so you would need a second bucket.
Cheers,
Tom
On Tue, May 19, 2020 at 11:54 PM Khodayar Doustar <doustar(a)rayanexon.ir>
wrote:
Hi,
I'm using Nautilus and I'm using the whole cluster mainly for a single
bucket in RadosGW.
There is a lot of data in this bucket (Petabyte scale) and I don't want to
waste all of SSD on it.
Is there anyway to automatically set some aging threshold for this data
and
e.g. move any data older than a month to HDD OSDs?
Does anyone has experience with this:
Pool Placement and Storage Classes:
https://docs.ceph.com/docs/master/radosgw/placement/
But something automatic would be much better for me in this case.
Any help would be appreciated.
Thanks a lot,
Khodayar
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Thomas Bennett
Storage Engineer at SARAO
3:55 p.m.
Hi Khodayar,
Yes, you are correct. I would have to move objects manually between (more
than one) buckets if I use "Pool placements and
Storage classes"
So you have successfully used this method and it was OK?
After we set up the new placement rule in the zone and zonegroups we
modified users configuration to set the placement to use. It's worked well
so far :). I'm happy to share how we set up the zones and zonegroups if you
need that information. But I followed what I could find in the ceph docs.
I may be forced to use this method because clients
needs more features
than mere cache tiering.
Have you ever used S3 Browser "bucket lifecycle rules" and those "storage
classes"?
We use s3 life cycle for data expiration but not for data transition. It
looks like transitions are supported in Nautilus
https://docs.ceph.com/docs/master/radosgw/placement/#storage-classes.
Cheers,
Tom
1430
days inactive
1431
days old
5 comments
3 participants
participants (3)
-
Janne Johansson -
Khodayar Doustar -
Thomas Bennett