Is this a chassis with universal slots, or is that NVMe device maybe M.2 or rear-cage?
12 * HDD via LSI jbod + 1 PCI NVME.
All NVMe devices are PCI ;).
Now it's 1.6TB, for the production plan is to use 3.2TB.
`ceph df`
`ceph osd dump | grep pool` So we can see what's going on HDD and what's on NVMe.
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 703 TiB 587 TiB 116 TiB 116 TiB 16.51 TOTAL 703 TiB 587 TiB 116 TiB 116 TiB 16.51
--- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL default.rgw.meta 52 64 6.0 KiB 13 131 KiB 0 177 TiB .mgr 54 32 28 MiB 8 83 MiB 0 177 TiB .rgw.root 55 64 2.0 KiB 4 48 KiB 0 177 TiB default.rgw.control 56 64 0 B 8 0 B 0 177 TiB default.rgw.buckets.index 59 32 34 MiB 33 102 MiB 0 177 TiB default.rgw.log 63 32 3.6 KiB 209 408 KiB 0 177 TiB default.rgw.buckets.non-ec 65 32 44 MiB 40 133 MiB 0 177 TiB 4_2_EC 67 1024 71 TiB 18.61M 106 TiB 16.61 355 TiB
So *everything* is on the HDDs? I suggest disabling the pg autoscaler and adjusting your pg_num values. If I calculate correctly `ceph osd df` should show + / - 82 on each OSD. I would target double that number. As a start maybe raise buckets.index and buckets.non-ec to 256.
You also have the metadata pools used by RGW that ideally need to be on
NVME. Because you are using EC then there is the buckets.non-ec pool which is used to manage the OMAPS for the multipart uploads this is usually down at 8 PG’s and that will be limiting things as well.
This part is very interesting. Some time ago I asked a similar question here <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YPHI5MF2CBQ2C7KYOJFG32A7N5HF2BXS/#7I6UDHWUK23JCKZSK25VXTRQYEDRFCPY>. Conclusion was that the index is covered by a bluestore.
hmmm. Since you don’t have any pools on that single NVMe drive I gather that you’re using it for WAL+DB, and you’ll have ~132GB sliced per OSD? Interesting idea. You’ll want to watch the RocksDB usage on those OSDs to ensure you aren’t spilling onto the slow device.
Should we consider removing a few HDDs and replacing them with SSD to non-ec pool?
So now you have the question of do you have enough streams running in
parallel? Have you tried a benchmarking tool such as minio warp to see what it can achieve.
I think so, warp shows 1.6GiB/s for 20GB objects in 50 streams - acceptable.
Changing the bluestone_min_alloc_size would be the last thing I would even
consider. In fact I wouldn’t be changing it as you are in untested territory.
ACK! :)
It used to be 64KB. Back around Octopus / Pacific it changed to 4KB for both rotational and non-rotational devices, in part due to the space amp that RGW users with small objects experienced: https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUToPQjN8bDI/edit#gid=358760253 Bluestore Space Amplification Cheat Sheet docs.google.com In your case I don’t think it would hurt to rebuild all of your OSDs with a value of 64KB, but I don’t think with modern code it would buy you much.
Thanks!
On Mon, 27 May 2024 at 09:27, Darren Soothill <darren.soothill@croit.io> wrote:
So a few questions I have around this.
What is the network you have for this cluster?
Changing the bluestone_min_alloc_size would be the last thing I would even consider. In fact I wouldn’t be changing it as you are in untested territory.
The challenge with making these sort of things perform is to generate lots of parallel streams so what ever is doing the uploading needs to be doing parallel multipart uploads. There is no mention of the uploading code that is being used.
So with 7 Nodes each with 12 Disks and doing large files like this I would be expecting to see 50-70MB/s per useable HDD. By useable I mean if you are doing Replicas then you would divide the number of disks by the replica number or in your case with EC I would be diving the number of disks by the EC size and multiplying by the data part. So divide by 6 and multiply by 4.
So allowing for EC overhead you in theory could get beyond 2.8GBytes/s That is the theoretical disk limit I would be looking to exceed.
So now you have the question of do you have enough streams running in parallel? Have you tried a benchmarking tool such as minio warp to see what it can achieve.
You haven’t mentioned the number of PG’s you have for each of the pools in question. You need to ensure that every pool that is being used has more PG’s that the number of disks. If that’s not the case then individual disks could be slowing things down.
You also have the metadata pools used by RGW that ideally need to be on NVME.
Because you are using EC then there is the buckets.non-ec pool which is used to manage the OMAPS for the multipart uploads this is usually down at 8 PG’s and that will be limiting things as well.
Darren Soothill
Want a meeting with me: https://calendar.app.google/MUdgrLEa7jSba3du9
Looking for help with your Ceph cluster? Contact us at https://croit.io/
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io/ | YouTube: https://goo.gl/PGE1Bx
On 25 May 2024, at 14:56, Anthony D'Atri <aad@dreamsnake.net> wrote:
Hi Everyone,
I'm putting together a HDD cluster with an ECC pool dedicated to the backup environment. Traffic via s3. Version 18.2, 7 OSD nodes, 12 * 12TB HDD + 1NVME each,
QLC, man. QLC. That said, I hope you're going to use that single NVMe SSD for at least the index pool. Is this a chassis with universal slots, or is that NVMe device maybe M.2 or rear-cage?
Wondering if there is some general guidance for startup setup/tuning in regards to s3 object size.
Small objects are the devil of any object storage system.
Files are read from fast storage (SSD/NVME) and written to s3. Files sizes are 10MB-1TB, so it's not standard s3. traffic.
Nothing nonstandard about that, though your 1TB objects presumably are going to be MPU. Having the .buckets.non-ec pool on HDD with objects that large might be really slow to assemble them, you might need to increase timeouts but I'm speculating.
Backup for big files took hours to complete.
Spinners gotta spin. They're a false economy.
My first shot would be to increase default bluestore_min_alloc_size_hdd, to reduce the number of stored objects, but I'm not sure if it's a good direccion?
With that workload you *could* increase that to like 64KB, but I don't think it'd gain you much.
Any other parameters worth checking to support such a traffic pattern?
`ceph df` `ceph osd dump | grep pool`
So we can see what's going on HDD and what's on NVMe.
Thanks!
-- Łukasz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Łukasz Borek lukasz@borek.org.pl _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io