21 Jan
2021
21 Jan
'21
4:42 a.m.
Hi,
What limits are there on the "reasonable size" of an rbd?
E.g. when I try to create a 1 PB rbd with default 4 MiB objects on my
octopus cluster:
$ rbd create --size 1P --data-pool rbd.ec rbd.meta/fs
2021-01-20T18:19:35.799+1100 7f47a99253c0 -1 librbd::image::CreateRequest:
validate_layout: image size not compatible with object
...which somes from:
== src/librbd/image/CreateRequest.cc
bool validate_layout(CephContext *cct, uint64_t size, file_layout_t &layout) {
if (!librbd::ObjectMap<>::is_compatible(layout, size)) {
lderr(cct) << "image size not compatible with object map" <<
dendl;
return false;
}
== src/librbd/ObjectMap.cc
template <typename I>
bool ObjectMap<I>::is_compatible(const file_layout_t& layout, uint64_t size)
{
uint64_t object_count = Striper::get_num_objects(layout, size);
return (object_count <= cls::rbd::MAX_OBJECT_MAP_OBJECT_COUNT);
}
== src/cls/rbd/cls_rbd_types.h
static const uint32_t MAX_OBJECT_MAP_OBJECT_COUNT = 256000000;
For 4 MiB objects that object count equates to just over 976 TB.
Is there any particular reason for that MAX_OBJECT_MAP_OBJECT_COUNT, or it
just "this is crazy large, if you're trying to go over this you're doing
something wrong, rethink your life..."?
Yes, I realise I can increase the size of the objects to get a larger rbd,
or drop the object-map support (and the fast-diff that goes along with
it).
I'm SO glad I found this limit now rather than starting on a smaller rbd
and a finding the limit when I tried to grow the rbd underneath a rapidly
filling filesystem.
What else should I know?
Background: I currently have nearly 0.5 PB on XFS (on lvm / raid6) and ZFS
that I'm looking to move over to ceph. XFS is a requirement, for the
reflinking (sadly not yet available in CephFS: https://tracker.ceph.com/issues/1680).
The recommendation for XFS is to start larger, on a thin-provisioned store
(hello rbd!), rather than start smaller and grow as needed - e.g. see the
thread surrounding:
https://www.spinics.net/lists/linux-xfs/msg20099.html
Rather than a single large rbd, should I be looking at multiple smaller
rbds linked together using lvm or somesuch? What are the tradeoffs?
And whilst we're here... for an rbd with the data on an erasure-coded
pool, how do you calculate the amount of rbd metadata required if/when the
rbd data is fully allocated?
Cheers,
Chris
21 Jan
21 Jan
9:57 a.m.
Hi,
Am 21.01.21 um 05:42 schrieb Chris Dunlop:
Is there any particular reason for that
MAX_OBJECT_MAP_OBJECT_COUNT, or
it just "this is crazy large, if you're trying to go over this you're
doing something wrong, rethink your life..."?
IMHO the limit is there because of the way deletion of RBDs work. "rbd
rm" has to look for every object, not only the ones that were really
created. This would make deleting a very very large RBD take a very very
long time.
Rather than a single large rbd, should I be looking at
multiple smaller
rbds linked together using lvm or somesuch? What are the tradeoffs?
IMHO there are no tradeoffs, there could even be benefits creating a
volume group with multiple physical volumes on RBD as the requests can
be bettere parallelized (i.e. virtio-single SCSI controller for qemu).
Regards
--
Robert Sander
Heinlein Support GmbH
Schwedter Str. 8/9b, 10119 Berlin
http://www.heinlein-support.de
Tel: 030 / 405051-43
Fax: 030 / 405051-19
Zwangsangaben lt. §35a GmbHG:
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
2:15 p.m.
Does Ceph now supports volume group of RBDs? From which version if any?
regards,
samuel
huxiaoyu(a)horebdata.cn
From: Robert Sander
Date: 2021-01-21 10:57
To: ceph-users
Subject: [ceph-users] Re: Large rbd
Hi,
Am 21.01.21 um 05:42 schrieb Chris Dunlop:
Is there any particular reason for that
MAX_OBJECT_MAP_OBJECT_COUNT, or
it just "this is crazy large, if you're trying to go over this you're
doing something wrong, rethink your life..."?
IMHO the limit is there because of the way deletion of RBDs work. "rbd
rm" has to look for every object, not only the ones that were really
created. This would make deleting a very very large RBD take a very very
long time.
Rather than a single large rbd, should I be looking at
multiple smaller
rbds linked together using lvm or somesuch? What are the tradeoffs?
IMHO there are no tradeoffs, there could even be benefits creating a
volume group with multiple physical volumes on RBD as the requests can
be bettere parallelized (i.e. virtio-single SCSI controller for qemu).
Regards
--
Robert Sander
Heinlein Support GmbH
Schwedter Str. 8/9b, 10119 Berlin
http://www.heinlein-support.de
Tel: 030 / 405051-43
Fax: 030 / 405051-19
Zwangsangaben lt. §35a GmbHG:
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:38 p.m.
Hi,
I think what's being suggested here is to create a good old LVM VG in a
virtualized guest, from multiple RBDs, each accessed as a separate
VirtIO SCSI device.
As each storage device in the LVM VG has its own queues at the VirtIO /
QEMU / RBD interface levels, that would allow for greater parallel
performance.
Loris Cuoghi
On 21/01/21 15:15, huxiaoyu(a)horebdata.cn wrote:
> Does Ceph now supports volume group of RBDs? From which version if any?
>
> regards,
>
> samuel
>
>
>
> huxiaoyu(a)horebdata.cn
>
> From: Robert Sander
> Date: 2021-01-21 10:57
> To: ceph-users
> Subject: [ceph-users] Re: Large rbd
> Hi,
>
> Am 21.01.21 um 05:42 schrieb Chris Dunlop:
>
>> Is there any particular reason for that MAX_OBJECT_MAP_OBJECT_COUNT, or
>> it just "this is crazy large, if you're trying to go over this
you're
>> doing something wrong, rethink your life..."?
>
> IMHO the limit is there because of the way deletion of RBDs work. "rbd
> rm" has to look for every object, not only the ones that were really
> created. This would make deleting a very very large RBD take a very very
> long time.
>
>> Rather than a single large rbd, should I be looking at multiple smaller
>> rbds linked together using lvm or somesuch? What are the tradeoffs?
>
> IMHO there are no tradeoffs, there could even be benefits creating a
> volume group with multiple physical volumes on RBD as the requests can
> be bettere parallelized (i.e. virtio-single SCSI controller for qemu).
>
> Regards
2:46 p.m.
I've always been curious about this. Does anyone have any experience
spanning an LVM VG over multiple RBDs?
3:31 p.m.
On Thu, Jan 21, 2021 at 9:47 AM John Petrini <jpetrini(a)coredial.com> wrote:
I've always been curious about this. Does anyone
have any experience
spanning an LVM VG over multiple RBDs?
I do on RHEL, it works very well. Each RDB device has some inherent IO
limitations, but using multiple in parallel works quite well. We never
were able to use fancy striping for this purpose, but LVM is simple, well
tested, and does not require any extra considerations. We currently just
use spanned volumes, not striped (although this would be interesting), and
allow parallelization to take place via database itself - the parallel IOs
are not necessarily aligned on RBD boundaries, but for our purposes, and
using all SSD OSDs this is enough.
Alex Gorbachev
iss-integration.com
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:33 p.m.
We do it in production, haven't benchmarked it though if that's what
you're aiming for. General consensus when we started with it was that it
allowed for greater performance (we use librbd kvm).
--
David Majchrzak
CTO
Oderland Webbhotell AB
Östra Hamngatan 50B, 411 09 Göteborg, SWEDEN
Den 2021-01-21 kl. 15:46, skrev John Petrini:
> I've always been curious about this. Does anyone have any experience
> spanning an LVM VG over multiple RBDs?
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:18 p.m.
On Thu, Jan 21, 2021 at 10:57:49AM +0100, Robert Sander wrote:
That's a good point, I hadn't considered potential i/o bandwidth benefits.
Thanks,
Chris
Hi,
Am 21.01.21 um 05:42 schrieb Chris Dunlop:
I wouldn't have though the ceph designers would have put in a hard limit like
that just to protect people from a long time to delete.
The removal time may well be a consideration for some but it's not a
significant issue in this case as the filesystem is intended to last for years
(the XFS and ZFS it's meant to replace have been around for maybe a decade).
That said, it does take a while. For a 967T rbd (the largest possible w/
default 4M objects) with a small amount written to it (maybe 4T):
$ rbd info rbd.meta/fs
rbd image 'fs':
size 976 TiB in 255852544 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 8126791dce2ad3
data_pool: rbd.ec.data
block_name_prefix: rbd_data.22.8126791dce2ad3
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten,
data-pool
op_features:
flags:
create_timestamp: Thu Jan 21 14:03:38 2021
access_timestamp: Thu Jan 21 14:03:38 2021
modify_timestamp: Thu Jan 21 14:03:38 2021
$ time rbd remove rbd.meta/fs
real 117m31.183s
user 116m56.895s
sys 0m2.101s
The issue is the number of objects. For instance, the same size rbd (967T) but
created with "--object-size 16M":
$ rbd info rbd.meta/fs
rbd image 'fs':
size 976 TiB in 63963136 objects
order 24 (16 MiB objects)
...
$ time rbd remove rbd.meta/fs
real 7m23.326s
user 6m45.201s
sys 0m1.272s
I don't know if the amount written affects the rbd removal time.
Is there any particular reason for that
MAX_OBJECT_MAP_OBJECT_COUNT, or
it just "this is crazy large, if you're trying to go over this you're
doing something wrong, rethink your life..."?
IMHO the limit is there because of the way deletion of RBDs work. "rbd
rm" has to look for every object, not only the ones that were really
created. This would make deleting a very very large RBD take a very very
long time. Rather than a
single large rbd, should I be looking at multiple smaller
rbds linked together using lvm or somesuch? What are the tradeoffs?
IMHO there are no tradeoffs, there could even be benefits creating a
volume group with multiple physical volumes on RBD as the requests can
be bettere parallelized (i.e. virtio-single SCSI controller for qemu). 
22 Jan
22 Jan
12:52 a.m.
On Thu, Jan 21, 2021 at 6:18 PM Chris Dunlop <chris(a)onthe.net.au> wrote:
That's a good point, I hadn't considered potential i/o bandwidth benefits.
Thanks,
Chris
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Jason
On Thu, Jan 21, 2021 at 10:57:49AM +0100, Robert Sander wrote:
You are free to disable the object-map when creating large images by
specifying the image-features -- or you can increase the object size
from its default 4MiB allocation size (which is honestly really no
different from QCOW2 switching increasing the backing cluster size as
the image grows larger).
The issue is that the size for the object-map for a 1PiB image w/ 4MiB
objects is going to be 268,435,456 backing objects which will require
64MiB of memory to store. It also just so happens that Ceph has a
hard-limit on the maximum object size of around 90MiB if I recall
correctly.
Hi,
Am 21.01.21 um 05:42 schrieb Chris Dunlop:
I wouldn't have though the ceph designers would have put in a hard limit like
that just to protect people from a long time to delete. Is there any particular reason for that
MAX_OBJECT_MAP_OBJECT_COUNT, or
it just "this is crazy large, if you're trying to go over this you're
doing something wrong, rethink your life..."?
IMHO the limit is there because of the way deletion of RBDs work. "rbd
rm" has to look for every object, not only the ones that were really
created. This would make deleting a very very large RBD take a very very
long time. The removal time may well be a consideration for some
but it's not a
significant issue in this case as the filesystem is intended to last for years
(the XFS and ZFS it's meant to replace have been around for maybe a decade).
That said, it does take a while. For a 967T rbd (the largest possible w/
default 4M objects) with a small amount written to it (maybe 4T):
$ rbd info rbd.meta/fs
rbd image 'fs':
size 976 TiB in 255852544 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 8126791dce2ad3
data_pool: rbd.ec.data
block_name_prefix: rbd_data.22.8126791dce2ad3
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten,
data-pool
op_features:
flags:
create_timestamp: Thu Jan 21 14:03:38 2021
access_timestamp: Thu Jan 21 14:03:38 2021
modify_timestamp: Thu Jan 21 14:03:38 2021
$ time rbd remove rbd.meta/fs
real 117m31.183s
user 116m56.895s
sys 0m2.101s
The issue is the number of objects. For instance, the same size rbd (967T) but
created with "--object-size 16M":
$ rbd info rbd.meta/fs
rbd image 'fs':
size 976 TiB in 63963136 objects
order 24 (16 MiB objects)
...
$ time rbd remove rbd.meta/fs
real 7m23.326s
user 6m45.201s
sys 0m1.272s
I don't know if the amount written affects the rbd removal time.
When the object-map is enabled, only written data extents need to be
deleted. W/o the object-map, it would need to issue deletes against
all possible objects.
Rather than a single large rbd, should I be looking at
multiple smaller
rbds linked together using lvm or somesuch? What are the tradeoffs?
IMHO there are no tradeoffs, there could even be benefits creating a
volume group with multiple physical volumes on RBD as the requests can
be bettere parallelized (i.e. virtio-single SCSI controller for qemu).
1:17 a.m.
On Thu, Jan 21, 2021 at 07:52:00PM -0500, Jason Dillaman wrote:
How does the lack of an object-map affect trimming? That's a very
important factor for a large thin volume like this.
Thanks,
Chris
On Thu, Jan 21, 2021 at 6:18 PM Chris Dunlop
<chris(a)onthe.net.au> wrote:
Is the whole object-map memory all held "in-core" the whole time, or is it
retrieved / freed as needed (for a busy filesystem that might mean it's
in-core practically the whole time, but for a quite filesystem maybe not
so much).
In this case the server has 192G RAM so 64MiB doesn't sound so scary.
By the way - does that 64MiB also match approx how much fast/replicated
storage I'll need if the data is on an ec volume?
I'm looking at 16M or larger objects, however I'm concerned about
fragmentation and how that might affect the "thinness" of the volume. It
seems with larger objects, and in the face of file removals in the upper
filesystem, with many files less than the object size, there's far more
chance for many objects to be significantly (but not completely) empty,
blowing out the actual storage used compared to the logical storage used
in the upper filesystem. Trimming won't help for partially allocated
objects.
Actually, maybe that's a good reason to not create a humongous fs in the
first place.
The XFS devs seem comfortable with growing a fs "a bit", e.g. 2-5 times
original size, but seemingly at about 10 times it's getting a bit dodgy.
So, creating a smaller fs in the first place (with the expectation it may
grow to 2-5 times original) means the fs itself will be encouraged to
reuse the space in larger objects rather than spreading itself out and
leaving a large number of partially filled objects.
On Thu, Jan 21, 2021 at 10:57:49AM +0100, Robert
Sander wrote:
You are free to disable the object-map when creating large images by
specifying the image-features -- or you can increase the object size
from its default 4MiB allocation size (which is honestly really no
different from QCOW2 switching increasing the backing cluster size as
the image grows larger).
The issue is that the size for the object-map for a 1PiB image w/ 4MiB
objects is going to be 268,435,456 backing objects which will require
64MiB of memory to store. It also just so happens that Ceph has a
hard-limit on the maximum object size of around 90MiB if I recall
correctly. Am 21.01.21 um 05:42 schrieb Chris Dunlop:
I wouldn't have though the ceph designers would have put in a hard limit like
that just to protect people from a long time to delete. Is there any particular reason for that
MAX_OBJECT_MAP_OBJECT_COUNT, or
it just "this is crazy large, if you're trying to go over this you're
doing something wrong, rethink your life..."?
IMHO the limit is there because of the way deletion of RBDs work. "rbd
rm" has to look for every object, not only the ones that were really
created. This would make deleting a very very large RBD take a very very
long time. I don't
know if the amount written affects the rbd removal time.
When the object-map is enabled, only written data extents need to be
deleted. W/o the object-map, it would need to issue deletes against
all possible objects.
1197
days inactive
1198
days old
9 comments
8 participants
participants (8)
-
Alex Gorbachev -
Chris Dunlop -
David Majchrzak, ODERLAND Webbhotell AB -
huxiaoyu＠horebdata.cn -
Jason Dillaman -
John Petrini -
Loris Cuoghi -
Robert Sander