Hi Anthony Appreciate you taking the time to provide so much guidance All your messages in this mailing list are well documented and VERY helpful I have attached a text file with the output of the commands you mentioned You are right , there is no /dev/bluefs_/bluefsstore_bdev pointing to the NVME namespaces No specific reason to use separate WAL , just remembered it was recommended back in the "mimic" days ( yes, I am a bit "rusty") I am using 4+2 EC I have 2 separate NVMEs ( 1.6 TB each) dedicated to DB/WAL for HDD I want to use one for the 6 HDD that I currently have , and save the other for when I will be adding more HDDs The servers are SuperMicro SSG-641E with 2 x Intel GOLD 6530 ( 32 cores each) and 1TB RAM My plan was/is to use the 3 x 15TB NVME on each server for high performance pools (like metadata for cephfs or index for RGW) I have carved them in 3 ( 5 TB each) so I can deploy more PGs hence increase performance The SSD are meant to be used for RBD ( proxmox ) The HDD are meant to be used for archiving data using S3 and cephfs. I can redeploy - please clarify 2 things about osd spec file 1. can I use NVME namespaces ( created with nvme command - see below) or should I let ceph partition the NVME disk ? in either case, how do you specify WHICH device/NVME to use for DB/WAL ( as I have 2 identical ones and want to keep the second one for adding more HDD in the future ) nvme create-ns $device --nsze=$per_ns_blocks --ncap=$per_ns_blocks --flbas=0 --dps=0 nvme attach-ns $device --namespace-id=$i --controllers=`nvme id-ctrl $device 2. to deploy 3 different types of OSDs using the spec I have with the advice from you , I am guessing this is the correct spec service_type: osd service_id: hdd_osd crush_device_class: hdd_class placement: host_pattern: * spec: data_devices rotational: 1 db_devices: rotational: 0 size: 1000G:1600G filter_logic: AND objectstore: bluestore service_id: ssd_osd crush_device_class: ssd_class placement: host_pattern: * spec: data_devices rotational: 0 size: 6000G:8000G service_id: nvme_osd crush_device_class: nvme_class placement: host_pattern: * spec: data_devices rotational: 0 size: 4000G:5500G Many thanks Steven On Sun, 29 Jun 2025 at 17:45, Anthony D'Atri <aad@dreamsnake.net> wrote:
So you have NVMe SSD OSDs, SATA SSD OSDs, and HDD OSDs with offload onto NVNe SSDs.
Did you have a specific reason to explicitly specify wal_devices_. It’s usually fine to just run with the default WAL size, with the WAL colocated with the DB. And thus give your DB partitions a bit more space.
What are your use-cases for these three classes of OSDs? Looks like you have 42x 20T HDD OSDs, 63x NVMe OSDs, and 84x 7.6T SATA SSD OSDs? Apparently with the 15T SSDs divided into 3x OSDs each? How much CPU do you have on these nodes? Any specific reason to have chopped up the NVMe SSDs into thirds?
It looks to me as though your .mgr pool is using the default replicated_rule, which does not specify a device class. This will confound the balancer and if enabled the pg_autoscaler. I recommend changing the .mgr pool to use the CRUSH rule that the non-buckets.data pools use, which should be one that specifies 3-replication constrained to one of the SSD device classes. As it is the .mgr pool may be placed on any of the three device classes, which is trivial with respect to space, but confounds as I mentioned.
Or you could manually edit the CRUSH map and change the #0 replicated_rule to specify nvme_class but it sounds like you’re new to Ceph and I don’t want to frighten you with that process which unfortunately is still old-school. Changing the rule as I suggested will be much safer.
The numbers look like you have all of the RGW pools except buckets.data on the nvme_class SSDs, which is fine, but you won’t begin to use all their capacity, the index pool will maybe use 5-10% of the capacity used by your buckets.data pool over time, depending on your distribution of object sizes and the replication strategy of your buckets.data pool. Doing the math I’ll speculate that your buckets.data pool is using a … EC 5+2 profile? True? If so I might suggest rebuilding if/while you still can. There are distinct advantages to having EC K+M < the number of OSD nodes.
Hi,
Yes, I have separate NVME namespaces allocated for WAL and DB to each spinning disk
Namespaces, or partitions?
Does that mean I still have to hunt for the 8TB culprit ?
Okay so `ceph cf` shows 8.2 TB of raw space used on the hdd_class OSDs, that’s your concern, right?
Please share outputs of the following:
`ceph osd df` (showing a few of each device class) `ceph osd dump | grep pool` `ceph osd metadata NNNN | egrep /dev\|bluefs_\|bluestore_bdev` for at least one OSD of each device class. And run it yourself without specifying an OSD ID so it captures all, and see if all OSDs in each device class look the same. `ceph osd device ls-by-host ceph-host-1
It’s entirely possible that your WAL+DB aren’t actually offloaded to SSDs as you intended. Advanced OSD service specs can be tricky.
That’s my suspicion, that the WAL+DB are actually still on your HDDs. Which can be migrated in-situ, or you can nuke the site from orbit and redeploy.
A note about your OSD specs. Specifying the models as you’re doing is totally supported. But think about what happens if you add nodes in the future that have different drive SKUs, or you RMA a drive and they send you a different SKU as the replacement.
It’s usually more future-proof to use a size range in the spec for each osd service instead of `model`, with a bit of margin to account for base 2 units vs base 10 units.
Here’s an example that creates OSDs on SSDs between 490 and 1200 GB, this is on systems that have ~ 1TB nominal drives. The systems also have 2TB SATA SSDs that are used for WAL+DB offload, which are above the 1200GB limit specified so they aren’t matched.
service_type: osd service_id: dashboard-admin-1705602677615 service_name: osd.dashboard-admin-1705602677615 placement: host_pattern: * spec: data_devices: rotational: 0 size: 490G:1200G filter_logic: AND objectstore: bluestore
And here is a spec that matches any HDD larger than 18T and deploys OSDs on them without offload. This cluster has 20TB HDDs, so the range of 18+ TB matches both the SEAGATE_ST20000NM007H and SEAGATE_ST20000NM002D drives present.
service_type: osd service_id: cost_capacity service_name: osd.cost_capacity placement: host_pattern: noactuallyusedanymore spec: data_devices: rotational: 1 size: '18T:' filter_logic: AND objectstore: bluestore
Oh, and make sure that your HDDs and SSDs are all updated to the most recent firmware. If you have Dell chassis, run DSU on the nodes and update all firmware, but skip the OS drivers. If you have HP chassis, you can get firmware update scripts from their web site, but I suspect these aren’t HP. If anyone else, they’re likely generic drives and you can get firmware updaters from the mfgs respective web sites.
Then reboot nodes one at a time to effect the firmware, letting the cluster completely recover between each reboot.
If yes , what would be the most efficient way of finding out what takes the space ?
Apologies for sending pictures but we are operating in an air gapped environment
I used this spec file to create the OSDs
<image.png>
Here is the osd tree of one of the servers all the other 6 are similar
<image.png>
Steven
On Sun, 29 Jun 2025 at 14:25, Anthony D'Atri <aad@dreamsnake.net> wrote:
WAL by default rides along with the DB and rarely warrants a separate or larger allocation.
Since you say you’ve allocated DB space, does that mean that you have WAL+DB offloaded onto SSDs? If so they don’t contribute to the space used on the hdd device class.
On Jun 29, 2025, at 1:56 PM, Steven Vacaroaia <stef97@gmail.com> wrote:
Hi Janne
Thanks That make sense since I have allocated 196GB for DB and 5 GB for WALL for all 42 spinning OSDs Again, thanks Steveb
On Sun, 29 Jun 2025 at 12:02, Janne Johansson <icepic.dz@gmail.com> wrote:
Den sön 29 juni 2025 kl 17:22 skrev Steven Vacaroaia <stef97@gmail.com :
Hi,
I just built a new CEPH squid cluster with 7 nodes Since this is brand new, there is no actuall data on it except few test files in the S3 data.bucket
Why is "ceph -s" reporting 8 TB of used capacity ?
Because each OSD will have GBs of preallocated data for the RocksDB, write-ahead-logs and other structures, and this counts against "raw available space", even if you don't have objects of this size put into the pools, the creation of the DBs and other things happened at osd creation, or when the first object was made, and are there even if you delete the object later.
-- May the most significant bit of your life be positive.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io