30 Jan
2024
30 Jan
'24
2:02 a.m.
Hello,
I have a question regarding the default pool of a cephfs.
According to the docs it is recommended to use a fast ssd replicated
pool as default pool for cephfs. I'm asking what are the space
requirements for storing the inode backtrace information?
Let's say I have a 85 TiB replicated ssd pool (hot data) and as 3 PiB EC
data pool (cold data).
Does it make sense to create a third pool as default pool which only
holds the inode backtrace information (what would be a good size), or is
it OK to use the ssd pool as default pool?
Thanks
Dietmar
31 Jan
31 Jan
9:53 a.m.
The docs recommend a fast SSD pool for the CephFS *metadata*, but the
default data pool can be more flexible. The backtraces are relatively
small — it's an encoded version of the path an inode is located at,
plus the RADOS hobject, which is probably more of the space usage. So
it should fit fine in your SSD pool, but if all the cephfs file data
is living in the hard drive pool I'd just set it up there.
-Greg
On Tue, Jan 30, 2024 at 2:03 AM Dietmar Rieder
<dietmar.rieder(a)i-med.ac.at> wrote:
Hello,
I have a question regarding the default pool of a cephfs.
According to the docs it is recommended to use a fast ssd replicated
pool as default pool for cephfs. I'm asking what are the space
requirements for storing the inode backtrace information?
Let's say I have a 85 TiB replicated ssd pool (hot data) and as 3 PiB EC
data pool (cold data).
Does it make sense to create a third pool as default pool which only
holds the inode backtrace information (what would be a good size), or is
it OK to use the ssd pool as default pool?
Thanks
Dietmar
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:32 p.m.
New subject: [EXTERN] Re: cephfs inode backtrace information
On 1/31/24 18:53, Gregory Farnum wrote:
The docs recommend a fast SSD pool for the CephFS
*metadata*, but the
default data pool can be more flexible. The backtraces are relatively
small — it's an encoded version of the path an inode is located at,
plus the RADOS hobject, which is probably more of the space usage. So
it should fit fine in your SSD pool, but if all the cephfs file data
is living in the hard drive pool I'd just set it up there.
Right, I wrote *ssd* because our replicated pool is on SSD disks and the
docs say:
"....If erasure-coded pools are planned for file system data, it is
best to configure the default as a *replicated pool* to improve
small-object write and read performance when updating backtraces..."
so of course replicated HDD pool would also fit there, I assume.
Well we plan to use the 85 TiB rep ssd pool for user homes (default
pool) and the 3 PiB EC HDD pool for data, so most of the data will
eventually be stored on the EC pool, so I was thinking if the need for
storing all inode backtrace information on the default pool will
substantially fill up the pool where also the user homes are going to end.
Thanks
Dietmar
On Tue, Jan 30, 2024 at 2:03 AM Dietmar Rieder
<dietmar.rieder(a)i-med.ac.at> wrote:
Hello,
I have a question regarding the default pool of a cephfs.
According to the docs it is recommended to use a fast ssd replicated
pool as default pool for cephfs. I'm asking what are the space
requirements for storing the inode backtrace information?
Let's say I have a 85 TiB replicated ssd pool (hot data) and as 3 PiB EC
data pool (cold data).
Does it make sense to create a third pool as default pool which only
holds the inode backtrace information (what would be a good size), or is
it OK to use the ssd pool as default pool?
Thanks
Dietmar
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:13 a.m.
On Tue, Jan 30, 2024 at 5:03 AM Dietmar Rieder
<dietmar.rieder(a)i-med.ac.at> wrote:
Hello,
I have a question regarding the default pool of a cephfs.
According to the docs it is recommended to use a fast ssd replicated
pool as default pool for cephfs. I'm asking what are the space
requirements for storing the inode backtrace information?
The actual recommendation is to use a replicated pool for the default
data pool. Regular hard drives are fine for the storage device.
Let's say I have a 85 TiB replicated ssd pool (hot
data) and as 3 PiB EC
data pool (cold data).
Does it make sense to create a third pool as default pool which only
holds the inode backtrace information (what would be a good size), or is
it OK to use the ssd pool as default pool?
Assuming your 85 TiB rep ssd pool is the default data pool already, use that.
(I am curious why this question is asked now when the file system
already has a significant amount of data? Are you thinking about
recreating the fs?)
--
Patrick Donnelly, Ph.D.
He / Him / His
Red Hat Partner Engineer
IBM, Inc.
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
11:23 p.m.
New subject: [EXTERN] Re: cephfs inode backtrace information
On 1/31/24 20:13, Patrick Donnelly wrote:
yes, I was planning to uses the 85 TiB rep ssd pool as the default, I
was just not sure if it might get substantially filled by inode
backtrace information from files that will get stored in the 3 PiB EC pool.
On Tue, Jan 30, 2024 at 5:03 AM Dietmar Rieder
<dietmar.rieder(a)i-med.ac.at> wrote:
yes true that, I was saying ssd because my replicated pool is on ssds.
Hello,
I have a question regarding the default pool of a cephfs.
According to the docs it is recommended to use a fast ssd replicated
pool as default pool for cephfs. I'm asking what are the space
requirements for storing the inode backtrace information?
The actual recommendation is to use a replicated pool for the default
data pool. Regular hard drives are fine for the storage device.
Let's say
I have a 85 TiB replicated ssd pool (hot data) and as 3 PiB EC
data pool (cold data).
Does it make sense to create a third pool as default pool which only
holds the inode backtrace information (what would be a good size), or is
it OK to use the ssd pool as default pool?
Assuming your 85 TiB rep ssd pool is the default data pool already, use that.
(I am curious why this question is asked now when the
file system
already has a significant amount of data? Are you thinking about
recreating the fs?)
the system is being set up, so no data there yet, but we plan to use the
85 TiB rep pool for user homes and the 3 PiB EC pool for data. We use
ceph.dir.layout.pool settings to separate the storage devices/pool for
/home and /data.
Dietmar
1 Feb
1 Feb
7:02 a.m.
On 31/01/2024 20:13, Patrick Donnelly wrote:
On Tue, Jan 30, 2024 at 5:03 AM Dietmar Rieder
<dietmar.rieder(a)i-med.ac.at> wrote:
Hello,
Is there a rule of thumb for the space requirements of the default pool
(depending on the number of POSIX objects) ?
One of our CephFS clusters is configured with a replicated default pool,
but we find the space usage on that pool to be very high with the
(somewhat) moderate number of files:
[ceph: root@$NODE /]# ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 5.9 PiB 3.9 PiB 2.0 PiB 2.0 PiB 33.53
ssd 51 TiB 35 TiB 15 TiB 15 TiB 30.08
TOTAL 5.9 PiB 3.9 PiB 2.0 PiB 2.0 PiB 33.50
--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
device_health_metrics 2 1 733 MiB 664 2.1 GiB 0 9.9 TiB
cephfs_EC_data 3 8192 1.5 PiB 515.56M 1.9 PiB 34.22 2.9 PiB
cephfs_metadata 4 128 86 GiB 10.28M 259 GiB 0.84 9.9 TiB
cephfs_default 5 512 4.2 TiB 131.17M 13 TiB 29.87 9.9 TiB
[...]
According to our statistics, there are about 132 million files and
symlinks in the filesystem which is consistent with the number of
objects for the "cephfs_default" pool.
(same for the metadata pool and the ~10 million directories)
But 4.2 TiB stored (~32 kiB per object) seems high, is this overhead
expected ?
This is a Pacific cluster (16.2.14) if that matters.
Have a nice day,
Loïc.
--
| Loīc Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre |
Hello,
I have a question regarding the default pool of a cephfs.
According to the docs it is recommended to use a fast ssd replicated
pool as default pool for cephfs. I'm asking what are the space
requirements for storing the inode backtrace information?
The actual recommendation is to use a replicated pool for the default
data pool. Regular hard drives are fine for the storage device.
28 Mar
28 Mar
8:18 p.m.
Hi Loïc, I'm surprised by that high storage amount, my "default" pool uses
only ~512 Bytes per file, not ~32 KiB like in your pool. That's a 64x difference!
(See also my other response to the original post I just sent.)
I'm using Ceph 16.2.1.
2 Apr
2 Apr
1 a.m.
On 29/03/2024 04:18, Niklas Hambüchen wrote:
Hi Loïc, I'm surprised by that high storage
amount, my "default" pool uses only ~512 Bytes per file, not ~32 KiB like in
your pool. That's a 64x difference!
(See also my other response to the original post I just sent.)
I'm using Ceph 16.2.1.
Hello,
We actually traced the source of this issue: a configuration mistake
(data pool not set properly on a client directory).
The directories for this client had "a few" large (tens of GiB) files,
which were stored in the "default" pool and used up a lot of space.
With this client's data moved where they belong:
[ceph: root@NODE /]# ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 6.1 PiB 3.8 PiB 2.3 PiB 2.3 PiB 37.15
ssd 52 TiB 49 TiB 3.2 TiB 3.2 TiB 6.04
TOTAL 6.1 PiB 3.9 PiB 2.3 PiB 2.3 PiB 36.89
--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
device_health_metrics 2 1 710 MiB 664 2.1 GiB 0 15 TiB
cephfs_EC_data 3 8192 1.7 PiB 606.79M 2.1 PiB 38.13 2.8 PiB
cephfs_metadata 4 128 101 GiB 14.55M 304 GiB 0.64 15 TiB
cephfs_default 5 128 0 B 162.90M 0 B 0 15 TiB
[...]
So the "correct" stored value for the default pool should be 0 bytes.
Loïc.
--
| Loīc Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre |
28 Mar
28 Mar
8:27 p.m.
Hi Patrick,
The actual recommendation is to use a replicated pool
for the default data pool. Regular hard drives are fine for the storage device.
But would it not be a better recommendation to put the default pool on SSDs?
* If you expect to have many small files on CephFS on HDD, seeking can be a huge
bottleneck for balancing/recovery/scrubbing. E.g. on 500 million small files it easily
takes 2 months to recover. My understanding is that CephFS makes at least 2 RADOS objects
per file: One for the file contents, one for the inode info. By putting half of those
objects on SSD, one could reduce the seek bottleneck by 2x. And from my other post, each
such inode object only seems to be ~512 Bytes (though in Loïcs' post it's 32 KiB,
not sure why that is, maybe some different minimum object size?).
* If you do not expect to have many small files, inode information should be so small that
it doesn't hurt to put it on SSD either.
So it seems like a sane recommendation either way.
Related thread I posted:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/VKVENC3VP3L…
What do you think?
Would it make sense to recommend this in the Ceph docs for all filesystems, even non-EC
replicated HDD?
30 Mar
30 Mar
2:09 p.m.
My sense is that CephFS is designed for larger files, but as always, QLC NAND >>
spinners.
> On Mar 30, 2024, at 10:38 AM, Niklas Hambüchen <mail(a)nh2.me> wrote:
>
> Hi Patrick,
>
>> The actual recommendation is to use a replicated pool for the default data pool.
Regular hard drives are fine for the storage device.
>
> But would it not be a better recommendation to put the default pool on SSDs?
>
> * If you expect to have many small files on CephFS on HDD, seeking can be a huge
bottleneck for balancing/recovery/scrubbing. E.g. on 500 million small files it easily
takes 2 months to recover. My understanding is that CephFS makes at least 2 RADOS objects
per file: One for the file contents, one for the inode info. By putting half of those
objects on SSD, one could reduce the seek bottleneck by 2x. And from my other post, each
such inode object only seems to be ~512 Bytes (though in Loïcs' post it's 32 KiB,
not sure why that is, maybe some different minimum object size?).
>
> * If you do not expect to have many small files, inode information should be so small
that it doesn't hurt to put it on SSD either.
>
> So it seems like a sane recommendation either way.
>
> Related thread I posted:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/VKVENC3VP3L…
>
> What do you think?
>
> Would it make sense to recommend this in the Ceph docs for all filesystems, even
non-EC replicated HDD?
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
28 Mar
28 Mar
8:12 p.m.
Hi,
from my observation, Ceph uses ~512 Bytes per inode.
So what matters is not the TiB of your EC pool, but the number of bytes.
(Again, this is concluded from observation on Ceph 18.2.1, and the fact that it's
called "inodes"; I have not checked the code.)
Example from my cluster which has "data_ec" as EC 4+2, and "data"
ebing the "default" pool for inode backtrace information (from
https://docs.ceph.com/en/reef/cephfs/createfs/#creating-pools), and nothing else:
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
.mgr 1 1 203 MiB 26 609 MiB 90.00 5 GiB
data 2 32 0 B 112.23M 0 B 0 61 TiB
data_ec 3 168 124 TiB 115.30M 186 TiB 50.53 121 TiB
metadata 4 128 63 GiB 32.87k 189 GiB 90.00 5 GiB
The odd thing here is that the 112 M inodes count as 0 Bytes.
This messes up PG autoscaling, I filed an issue about that here:
https://tracker.ceph.com/issues/65199
I would probably create a pool just for this inode information, simply so that you can see
it separately, and easily migrate it to different storage if you change your mind.
I do not understand your "what would be a good size?" question, since if you
create multiple pools that use SSDs, your SSD OSDs will automatically be used and the
remaining space will be shared across all of your SSD pools anyway -- you do not have to
provision "separate" SSDs for making another SSD pool.
See also my related question:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/VKVENC3VP3L…
36
days inactive
99
days old
10 comments
6 participants
participants (6)
-
Anthony D'Atri -
Dietmar Rieder -
Gregory Farnum -
Loïc Tortay -
Niklas Hambüchen -
Patrick Donnelly