cephfs inode backtrace information
Hello, I have a question regarding the default pool of a cephfs. According to the docs it is recommended to use a fast ssd replicated pool as default pool for cephfs. I'm asking what are the space requirements for storing the inode backtrace information? Let's say I have a 85 TiB replicated ssd pool (hot data) and as 3 PiB EC data pool (cold data). Does it make sense to create a third pool as default pool which only holds the inode backtrace information (what would be a good size), or is it OK to use the ssd pool as default pool? Thanks Dietmar
The docs recommend a fast SSD pool for the CephFS *metadata*, but the default data pool can be more flexible. The backtraces are relatively small — it's an encoded version of the path an inode is located at, plus the RADOS hobject, which is probably more of the space usage. So it should fit fine in your SSD pool, but if all the cephfs file data is living in the hard drive pool I'd just set it up there. -Greg On Tue, Jan 30, 2024 at 2:03 AM Dietmar Rieder <dietmar.rieder@i-med.ac.at> wrote:
Hello,
I have a question regarding the default pool of a cephfs.
According to the docs it is recommended to use a fast ssd replicated pool as default pool for cephfs. I'm asking what are the space requirements for storing the inode backtrace information?
Let's say I have a 85 TiB replicated ssd pool (hot data) and as 3 PiB EC data pool (cold data).
Does it make sense to create a third pool as default pool which only holds the inode backtrace information (what would be a good size), or is it OK to use the ssd pool as default pool?
Thanks Dietmar
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 1/31/24 18:53, Gregory Farnum wrote:
The docs recommend a fast SSD pool for the CephFS *metadata*, but the default data pool can be more flexible. The backtraces are relatively small — it's an encoded version of the path an inode is located at, plus the RADOS hobject, which is probably more of the space usage. So it should fit fine in your SSD pool, but if all the cephfs file data is living in the hard drive pool I'd just set it up there.
Right, I wrote *ssd* because our replicated pool is on SSD disks and the docs say: "....If erasure-coded pools are planned for file system data, it is best to configure the default as a *replicated pool* to improve small-object write and read performance when updating backtraces..." so of course replicated HDD pool would also fit there, I assume. Well we plan to use the 85 TiB rep ssd pool for user homes (default pool) and the 3 PiB EC HDD pool for data, so most of the data will eventually be stored on the EC pool, so I was thinking if the need for storing all inode backtrace information on the default pool will substantially fill up the pool where also the user homes are going to end. Thanks Dietmar
On Tue, Jan 30, 2024 at 2:03 AM Dietmar Rieder <dietmar.rieder@i-med.ac.at> wrote:
Hello,
I have a question regarding the default pool of a cephfs.
According to the docs it is recommended to use a fast ssd replicated pool as default pool for cephfs. I'm asking what are the space requirements for storing the inode backtrace information?
Let's say I have a 85 TiB replicated ssd pool (hot data) and as 3 PiB EC data pool (cold data).
Does it make sense to create a third pool as default pool which only holds the inode backtrace information (what would be a good size), or is it OK to use the ssd pool as default pool?
Thanks Dietmar
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tue, Jan 30, 2024 at 5:03 AM Dietmar Rieder <dietmar.rieder@i-med.ac.at> wrote:
Hello,
I have a question regarding the default pool of a cephfs.
According to the docs it is recommended to use a fast ssd replicated pool as default pool for cephfs. I'm asking what are the space requirements for storing the inode backtrace information?
The actual recommendation is to use a replicated pool for the default data pool. Regular hard drives are fine for the storage device.
Let's say I have a 85 TiB replicated ssd pool (hot data) and as 3 PiB EC data pool (cold data).
Does it make sense to create a third pool as default pool which only holds the inode backtrace information (what would be a good size), or is it OK to use the ssd pool as default pool?
Assuming your 85 TiB rep ssd pool is the default data pool already, use that. (I am curious why this question is asked now when the file system already has a significant amount of data? Are you thinking about recreating the fs?) -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
On 1/31/24 20:13, Patrick Donnelly wrote:
On Tue, Jan 30, 2024 at 5:03 AM Dietmar Rieder <dietmar.rieder@i-med.ac.at> wrote:
Hello,
I have a question regarding the default pool of a cephfs.
According to the docs it is recommended to use a fast ssd replicated pool as default pool for cephfs. I'm asking what are the space requirements for storing the inode backtrace information?
The actual recommendation is to use a replicated pool for the default data pool. Regular hard drives are fine for the storage device.
yes true that, I was saying ssd because my replicated pool is on ssds.
Let's say I have a 85 TiB replicated ssd pool (hot data) and as 3 PiB EC data pool (cold data).
Does it make sense to create a third pool as default pool which only holds the inode backtrace information (what would be a good size), or is it OK to use the ssd pool as default pool?
Assuming your 85 TiB rep ssd pool is the default data pool already, use that.
yes, I was planning to uses the 85 TiB rep ssd pool as the default, I was just not sure if it might get substantially filled by inode backtrace information from files that will get stored in the 3 PiB EC pool.
(I am curious why this question is asked now when the file system already has a significant amount of data? Are you thinking about recreating the fs?)
the system is being set up, so no data there yet, but we plan to use the 85 TiB rep pool for user homes and the 3 PiB EC pool for data. We use ceph.dir.layout.pool settings to separate the storage devices/pool for /home and /data. Dietmar
On 31/01/2024 20:13, Patrick Donnelly wrote:
On Tue, Jan 30, 2024 at 5:03 AM Dietmar Rieder <dietmar.rieder@i-med.ac.at> wrote:
Hello,
I have a question regarding the default pool of a cephfs.
According to the docs it is recommended to use a fast ssd replicated pool as default pool for cephfs. I'm asking what are the space requirements for storing the inode backtrace information?
The actual recommendation is to use a replicated pool for the default data pool. Regular hard drives are fine for the storage device.
Hello, Is there a rule of thumb for the space requirements of the default pool (depending on the number of POSIX objects) ? One of our CephFS clusters is configured with a replicated default pool, but we find the space usage on that pool to be very high with the (somewhat) moderate number of files: [ceph: root@$NODE /]# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.9 PiB 3.9 PiB 2.0 PiB 2.0 PiB 33.53 ssd 51 TiB 35 TiB 15 TiB 15 TiB 30.08 TOTAL 5.9 PiB 3.9 PiB 2.0 PiB 2.0 PiB 33.50 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 2 1 733 MiB 664 2.1 GiB 0 9.9 TiB cephfs_EC_data 3 8192 1.5 PiB 515.56M 1.9 PiB 34.22 2.9 PiB cephfs_metadata 4 128 86 GiB 10.28M 259 GiB 0.84 9.9 TiB cephfs_default 5 512 4.2 TiB 131.17M 13 TiB 29.87 9.9 TiB [...] According to our statistics, there are about 132 million files and symlinks in the filesystem which is consistent with the number of objects for the "cephfs_default" pool. (same for the metadata pool and the ~10 million directories) But 4.2 TiB stored (~32 kiB per object) seems high, is this overhead expected ? This is a Pacific cluster (16.2.14) if that matters. Have a nice day, Loïc. -- | Loīc Tortay <tortay@cc.in2p3.fr> - IN2P3 Computing Centre |
Hi Loïc, I'm surprised by that high storage amount, my "default" pool uses only ~512 Bytes per file, not ~32 KiB like in your pool. That's a 64x difference! (See also my other response to the original post I just sent.) I'm using Ceph 16.2.1.
On 29/03/2024 04:18, Niklas Hambüchen wrote:
Hi Loïc, I'm surprised by that high storage amount, my "default" pool uses only ~512 Bytes per file, not ~32 KiB like in your pool. That's a 64x difference!
(See also my other response to the original post I just sent.)
I'm using Ceph 16.2.1.
Hello, We actually traced the source of this issue: a configuration mistake (data pool not set properly on a client directory). The directories for this client had "a few" large (tens of GiB) files, which were stored in the "default" pool and used up a lot of space. With this client's data moved where they belong: [ceph: root@NODE /]# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 6.1 PiB 3.8 PiB 2.3 PiB 2.3 PiB 37.15 ssd 52 TiB 49 TiB 3.2 TiB 3.2 TiB 6.04 TOTAL 6.1 PiB 3.9 PiB 2.3 PiB 2.3 PiB 36.89 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 2 1 710 MiB 664 2.1 GiB 0 15 TiB cephfs_EC_data 3 8192 1.7 PiB 606.79M 2.1 PiB 38.13 2.8 PiB cephfs_metadata 4 128 101 GiB 14.55M 304 GiB 0.64 15 TiB cephfs_default 5 128 0 B 162.90M 0 B 0 15 TiB [...] So the "correct" stored value for the default pool should be 0 bytes. Loïc. -- | Loīc Tortay <tortay@cc.in2p3.fr> - IN2P3 Computing Centre |
Hi Patrick,
The actual recommendation is to use a replicated pool for the default data pool. Regular hard drives are fine for the storage device.
But would it not be a better recommendation to put the default pool on SSDs? * If you expect to have many small files on CephFS on HDD, seeking can be a huge bottleneck for balancing/recovery/scrubbing. E.g. on 500 million small files it easily takes 2 months to recover. My understanding is that CephFS makes at least 2 RADOS objects per file: One for the file contents, one for the inode info. By putting half of those objects on SSD, one could reduce the seek bottleneck by 2x. And from my other post, each such inode object only seems to be ~512 Bytes (though in Loïcs' post it's 32 KiB, not sure why that is, maybe some different minimum object size?). * If you do not expect to have many small files, inode information should be so small that it doesn't hurt to put it on SSD either. So it seems like a sane recommendation either way. Related thread I posted: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/VKVENC3VP3LC... What do you think? Would it make sense to recommend this in the Ceph docs for all filesystems, even non-EC replicated HDD?
My sense is that CephFS is designed for larger files, but as always, QLC NAND >> spinners.
On Mar 30, 2024, at 10:38 AM, Niklas Hambüchen <mail@nh2.me> wrote:
Hi Patrick,
The actual recommendation is to use a replicated pool for the default data pool. Regular hard drives are fine for the storage device.
But would it not be a better recommendation to put the default pool on SSDs?
* If you expect to have many small files on CephFS on HDD, seeking can be a huge bottleneck for balancing/recovery/scrubbing. E.g. on 500 million small files it easily takes 2 months to recover. My understanding is that CephFS makes at least 2 RADOS objects per file: One for the file contents, one for the inode info. By putting half of those objects on SSD, one could reduce the seek bottleneck by 2x. And from my other post, each such inode object only seems to be ~512 Bytes (though in Loïcs' post it's 32 KiB, not sure why that is, maybe some different minimum object size?).
* If you do not expect to have many small files, inode information should be so small that it doesn't hurt to put it on SSD either.
So it seems like a sane recommendation either way.
Related thread I posted: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/VKVENC3VP3LC...
What do you think?
Would it make sense to recommend this in the Ceph docs for all filesystems, even non-EC replicated HDD? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, from my observation, Ceph uses ~512 Bytes per inode. So what matters is not the TiB of your EC pool, but the number of bytes. (Again, this is concluded from observation on Ceph 18.2.1, and the fact that it's called "inodes"; I have not checked the code.) Example from my cluster which has "data_ec" as EC 4+2, and "data" ebing the "default" pool for inode backtrace information (from https://docs.ceph.com/en/reef/cephfs/createfs/#creating-pools), and nothing else: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 203 MiB 26 609 MiB 90.00 5 GiB data 2 32 0 B 112.23M 0 B 0 61 TiB data_ec 3 168 124 TiB 115.30M 186 TiB 50.53 121 TiB metadata 4 128 63 GiB 32.87k 189 GiB 90.00 5 GiB The odd thing here is that the 112 M inodes count as 0 Bytes. This messes up PG autoscaling, I filed an issue about that here: https://tracker.ceph.com/issues/65199 I would probably create a pool just for this inode information, simply so that you can see it separately, and easily migrate it to different storage if you change your mind. I do not understand your "what would be a good size?" question, since if you create multiple pools that use SSDs, your SSD OSDs will automatically be used and the remaining space will be shared across all of your SSD pools anyway -- you do not have to provision "separate" SSDs for making another SSD pool. See also my related question: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/VKVENC3VP3LC...
participants (6)
-
Anthony D'Atri
-
Dietmar Rieder
-
Gregory Farnum
-
Loïc Tortay
-
Niklas Hambüchen
-
Patrick Donnelly