DB sizing for lots of large files
Hi, Sorry to bother you all. It’s a home server setup. Three nodes (ODROID-H2+ with 32GB RAM and dual 2.5Gbit NICs), two 14TB 7200rpm SATA drives and an Optane 118GB NVMe in each node (OS boots from eMMC). Only CephFS, I'm anticipating having 50-200K files when the 50TB (4+2 EC) is full. I'm trying to address the issue of really big OSD's, not my words, a Redditor: "When you write an object to a drive with collocated db and raw space, the disk has to read/write to both sections before acking a write. That's a lot to ask a 7200 disk to handle gracefully. I believe Red Hat only supports up to 8TB because of performance concerns with larger disks. I may be wrong, but once you are shuffling through 6-10TB of data I'd think those disks are gonna be bogged down in seek time." So I want to have my two DB's on the Optane to avoid the above, am I making sense? OK, so large files have lower metadata overhead than small files. This is for a media library this probably means super low overhead, one guy I spoke to had similar setup and for 48TB used he had a 2.6GB DB? Is there a rough CephFS calculation (each file uses x bytes of metadata), I think I should be safe with 30GB, now I read I should double that (you should allocate twice the size of the biggest layer to allow for compaction) but I only have 118GB and two OSDs so I will have to go for 59GB (or whatever will fit)? I'm thinking that I might not even use 3GB but to be safe I’ll make it 30GB, lets say it settles at 5GB, when compaction comes that means I will only need 20GB and therefore no spillover? I realise that if the Optane dies both OSD's go with it, do I have to configure anything special there (CRUSH would just handle it)? Being that this a home deployment will Ceph be OK with an occasional power outage, I mean if a bluray mkv gets corrupted ripping it again is an easy fix, also I will backup the clusters data once a month? Below are the commands I got from the website: Create the volume groups: $ vgcreate ceph-block-0 /dev/sda $ vgcreate ceph-block-1 /dev/sdb Create the logical volumes: $ lvcreate -l 100%FREE -n block-0 ceph-block-0 $ lvcreate -l 100%FREE -n block-1 ceph-block-1 Create db logical volumes (118GB Optane) $ vgcreate ceph-db-0 /dev/sdc $ lvcreate -L 59GB -n db-0 ceph-db-0 $ lvcreate -L 59GB -n db-1 ceph-db-0 Create the OSDs $ ceph-volume lvm create --bluestore --data ceph-block-0/block-0 --block.db ceph-db-0/db-0 $ ceph-volume lvm create --bluestore --data ceph-block-1/block-1 --block.db ceph-db-0/db-1 Thanks. Richard
Hi, On 11/26/20 12:45 PM, Richard Thornton wrote:
Hi,
Sorry to bother you all.
It’s a home server setup.
Three nodes (ODROID-H2+ with 32GB RAM and dual 2.5Gbit NICs), two 14TB 7200rpm SATA drives and an Optane 118GB NVMe in each node (OS boots from eMMC).
*snipsnap*
Is there a rough CephFS calculation (each file uses x bytes of metadata), I think I should be safe with 30GB, now I read I should double that (you should allocate twice the size of the biggest layer to allow for compaction) but I only have 118GB and two OSDs so I will have to go for 59GB (or whatever will fit)?
The recommended size of 30 GB is due to the level design of rocksdb; data is stored in different cache levels with increasing level sizes. 30 GB is a kind of sweet spot between 3 GB and 300 GB (too small / way too large for most use case). The recommendation for doubling the size for compaction is OK, but you will waste capacity most the time. In our cephfs instance we have ~ 115.000.000 files. Metadata is stored on 18 SSD based OSDs. About 30-35 GB raw capacity of the data is currently in use, almost exclusively for metadata, omap and other stuff. You might be able to scale this down to your use case. Our average file size approx. 5 MB, so you can also put a little bit on top in your case. If your working set (files accesses in a time span) is rather small, you also have the option to use the SSD for some block device caching layer like bcache or dmcache. In this setup the whole capacity will be used, and also data operations on the OSDs will benefit from the faster SSDs. Your failure domain will be the same; if the SSD dies your data disks will be useless. Otherwise I would recommend to use DB partitions of the recommended size (do not forget to include some extra space for the WAL), and use the remaining capacity for extra SSD based OSDs similar to our setup. This willensure that metadata access will be fast[tm]. Regards, Burkhard
Sorry, I replied to the wrong email thread before, so reposting this: I think it's time to start pointing out the the 3/30/300 logic not really holds any longer true post Octopus: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/CKRCB3HUR7U... Although I suppose in a way this makes it even harder to provide a sizing recommendation On Fri, 27 Nov 2020 at 04:49, Burkhard Linke < Burkhard.Linke@computational.bio.uni-giessen.de> wrote:
Hi,
On 11/26/20 12:45 PM, Richard Thornton wrote:
Hi,
Sorry to bother you all.
It’s a home server setup.
Three nodes (ODROID-H2+ with 32GB RAM and dual 2.5Gbit NICs), two 14TB 7200rpm SATA drives and an Optane 118GB NVMe in each node (OS boots from eMMC).
*snipsnap*
Is there a rough CephFS calculation (each file uses x bytes of metadata), I think I should be safe with 30GB, now I read I should double that (you should allocate twice the size of the biggest layer to allow for compaction) but I only have 118GB and two OSDs so I will have to go for 59GB (or whatever will fit)?
The recommended size of 30 GB is due to the level design of rocksdb; data is stored in different cache levels with increasing level sizes. 30 GB is a kind of sweet spot between 3 GB and 300 GB (too small / way too large for most use case). The recommendation for doubling the size for compaction is OK, but you will waste capacity most the time.
In our cephfs instance we have ~ 115.000.000 files. Metadata is stored on 18 SSD based OSDs. About 30-35 GB raw capacity of the data is currently in use, almost exclusively for metadata, omap and other stuff. You might be able to scale this down to your use case. Our average file size approx. 5 MB, so you can also put a little bit on top in your case.
If your working set (files accesses in a time span) is rather small, you also have the option to use the SSD for some block device caching layer like bcache or dmcache. In this setup the whole capacity will be used, and also data operations on the OSDs will benefit from the faster SSDs. Your failure domain will be the same; if the SSD dies your data disks will be useless.
Otherwise I would recommend to use DB partitions of the recommended size (do not forget to include some extra space for the WAL), and use the remaining capacity for extra SSD based OSDs similar to our setup. This willensure that metadata access will be fast[tm].
Regards,
Burkhard
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks everyone. So the 3/30/300GB restriction no longer exists in Octopus, so can I make it 10GB and it will use all 10GB? Is there a migration strategy that allows me to setup the DB on the OSD, see how much metadata my 25TB is using, make a partition on the Optane say quadruple the size and then move the DB to the Optane? Or maybe the best strategy would be to start with a small logical volume on the Optane, copy over my 25TB of existing data and extend it if required? The bluefs-bdev-migrate and bluefs-bdev-expand commnds seem to be the ticket. On 27 Nov 2020 at 6:19:06 am, Christian Wuerdig <christian.wuerdig@gmail.com> wrote:
Sorry, I replied to the wrong email thread before, so reposting this: I think it's time to start pointing out the the 3/30/300 logic not really holds any longer true post Octopus:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/CKRCB3HUR7U... Although I suppose in a way this makes it even harder to provide a sizing recommendation
On Fri, 27 Nov 2020 at 04:49, Burkhard Linke < Burkhard.Linke@computational.bio.uni-giessen.de> wrote:
Hi,
On 11/26/20 12:45 PM, Richard Thornton wrote:
Hi,
Sorry to bother you all.
It’s a home server setup.
Three nodes (ODROID-H2+ with 32GB RAM and dual 2.5Gbit NICs), two 14TB
7200rpm SATA drives and an Optane 118GB NVMe in each node (OS boots from
eMMC).
*snipsnap*
Is there a rough CephFS calculation (each file uses x bytes of
metadata), I
think I should be safe with 30GB, now I read I should double that (you
should allocate twice the size of the biggest layer to allow for
compaction) but I only have 118GB and two OSDs so I will have to go for
59GB (or whatever will fit)?
The recommended size of 30 GB is due to the level design of rocksdb;
data is stored in different cache levels with increasing level sizes. 30
GB is a kind of sweet spot between 3 GB and 300 GB (too small / way too
large for most use case). The recommendation for doubling the size for
compaction is OK, but you will waste capacity most the time.
In our cephfs instance we have ~ 115.000.000 files. Metadata is stored
on 18 SSD based OSDs. About 30-35 GB raw capacity of the data is
currently in use, almost exclusively for metadata, omap and other stuff.
You might be able to scale this down to your use case. Our average file
size approx. 5 MB, so you can also put a little bit on top in your case.
If your working set (files accesses in a time span) is rather small, you
also have the option to use the SSD for some block device caching layer
like bcache or dmcache. In this setup the whole capacity will be used,
and also data operations on the OSDs will benefit from the faster SSDs.
Your failure domain will be the same; if the SSD dies your data disks
will be useless.
Otherwise I would recommend to use DB partitions of the recommended size
(do not forget to include some extra space for the WAL), and use the
remaining capacity for extra SSD based OSDs similar to our setup. This
willensure that metadata access will be fast[tm].
Regards,
Burkhard
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Christian wrote “post Octopus”. The referenced code seems likely to appear in Pacific. We’ll see how it works out in practice. I suspect that provisioned space will automagically be used when an OSD starts under a future release, though the release notes may give us specific instructions, like we saw with the new stats reporting.
So the 3/30/300GB restriction no longer exists in Octopus, so can I make it 10GB and it will use all 10GB?
Is there a migration strategy that allows me to setup the DB on the OSD, see how much metadata my 25TB is using, make a partition on the Optane say quadruple the size and then move the DB to the Optane?
Or maybe the best strategy would be to start with a small logical volume on the Optane, copy over my 25TB of existing data and extend it if required?
The bluefs-bdev-migrate and bluefs-bdev-expand commnds seem to be the ticket.
On 27 Nov 2020 at 6:19:06 am, Christian Wuerdig <christian.wuerdig@gmail.com> wrote:
Sorry, I replied to the wrong email thread before, so reposting this: I think it's time to start pointing out the the 3/30/300 logic not really holds any longer true post Octopus:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/CKRCB3HUR7U... Although I suppose in a way this makes it even harder to provide a sizing recommendation
On Fri, 27 Nov 2020 at 04:49, Burkhard Linke < Burkhard.Linke@computational.bio.uni-giessen.de> wrote:
Hi,
On 11/26/20 12:45 PM, Richard Thornton wrote:
Hi,
Sorry to bother you all.
It’s a home server setup.
Three nodes (ODROID-H2+ with 32GB RAM and dual 2.5Gbit NICs), two 14TB
7200rpm SATA drives and an Optane 118GB NVMe in each node (OS boots from
eMMC).
*snipsnap*
Is there a rough CephFS calculation (each file uses x bytes of
metadata), I
think I should be safe with 30GB, now I read I should double that (you
should allocate twice the size of the biggest layer to allow for
compaction) but I only have 118GB and two OSDs so I will have to go for
59GB (or whatever will fit)?
The recommended size of 30 GB is due to the level design of rocksdb;
data is stored in different cache levels with increasing level sizes. 30
GB is a kind of sweet spot between 3 GB and 300 GB (too small / way too
large for most use case). The recommendation for doubling the size for
compaction is OK, but you will waste capacity most the time.
In our cephfs instance we have ~ 115.000.000 files. Metadata is stored
on 18 SSD based OSDs. About 30-35 GB raw capacity of the data is
currently in use, almost exclusively for metadata, omap and other stuff.
You might be able to scale this down to your use case. Our average file
size approx. 5 MB, so you can also put a little bit on top in your case.
If your working set (files accesses in a time span) is rather small, you
also have the option to use the SSD for some block device caching layer
like bcache or dmcache. In this setup the whole capacity will be used,
and also data operations on the OSDs will benefit from the faster SSDs.
Your failure domain will be the same; if the SSD dies your data disks
will be useless.
Otherwise I would recommend to use DB partitions of the recommended size
(do not forget to include some extra space for the WAL), and use the
remaining capacity for extra SSD based OSDs similar to our setup. This
willensure that metadata access will be fast[tm].
Regards,
Burkhard
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Also you don’t need to create the volume groups and logical volumes for the OSDs, ceph-volume can do that for you and handles raw disks, too. Check out the man page for all the options: https://docs.ceph.com/en/latest/man/8/ceph-volume/ Zitat von Richard Thornton <richie.thornton@gmail.com>:
Hi,
Sorry to bother you all.
It’s a home server setup.
Three nodes (ODROID-H2+ with 32GB RAM and dual 2.5Gbit NICs), two 14TB 7200rpm SATA drives and an Optane 118GB NVMe in each node (OS boots from eMMC).
Only CephFS, I'm anticipating having 50-200K files when the 50TB (4+2 EC) is full.
I'm trying to address the issue of really big OSD's, not my words, a Redditor:
"When you write an object to a drive with collocated db and raw space, the disk has to read/write to both sections before acking a write. That's a lot to ask a 7200 disk to handle gracefully. I believe Red Hat only supports up to 8TB because of performance concerns with larger disks. I may be wrong, but once you are shuffling through 6-10TB of data I'd think those disks are gonna be bogged down in seek time."
So I want to have my two DB's on the Optane to avoid the above, am I making sense?
OK, so large files have lower metadata overhead than small files. This is for a media library this probably means super low overhead, one guy I spoke to had similar setup and for 48TB used he had a 2.6GB DB?
Is there a rough CephFS calculation (each file uses x bytes of metadata), I think I should be safe with 30GB, now I read I should double that (you should allocate twice the size of the biggest layer to allow for compaction) but I only have 118GB and two OSDs so I will have to go for 59GB (or whatever will fit)?
I'm thinking that I might not even use 3GB but to be safe I’ll make it 30GB, lets say it settles at 5GB, when compaction comes that means I will only need 20GB and therefore no spillover?
I realise that if the Optane dies both OSD's go with it, do I have to configure anything special there (CRUSH would just handle it)?
Being that this a home deployment will Ceph be OK with an occasional power outage, I mean if a bluray mkv gets corrupted ripping it again is an easy fix, also I will backup the clusters data once a month?
Below are the commands I got from the website:
Create the volume groups: $ vgcreate ceph-block-0 /dev/sda $ vgcreate ceph-block-1 /dev/sdb
Create the logical volumes: $ lvcreate -l 100%FREE -n block-0 ceph-block-0 $ lvcreate -l 100%FREE -n block-1 ceph-block-1
Create db logical volumes (118GB Optane) $ vgcreate ceph-db-0 /dev/sdc $ lvcreate -L 59GB -n db-0 ceph-db-0 $ lvcreate -L 59GB -n db-1 ceph-db-0
Create the OSDs $ ceph-volume lvm create --bluestore --data ceph-block-0/block-0 --block.db ceph-db-0/db-0 $ ceph-volume lvm create --bluestore --data ceph-block-1/block-1 --block.db ceph-db-0/db-1
Thanks. Richard _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Anthony D'Atri
-
Burkhard Linke
-
Christian Wuerdig
-
Eugen Block
-
Richard Thornton