Hi Anthony
Appreciate you taking the time to provide so much guidance
All your messages in this mailing list are well documented and VERY helpful
I have attached a text file with the output of the commands you mentioned
You are right , there is no /dev/bluefs_/bluefsstore_bdev pointing to the
NVME namespaces
No specific reason to use separate WAL , just remembered it was recommended
back in the "mimic" days ( yes, I am a bit "rusty")
I am using 4+2 EC
I have 2 separate NVMEs ( 1.6 TB each) dedicated to DB/WAL for HDD
I want to use one for the 6 HDD that I currently have , and save the other
for when I will be adding more HDDs
The servers are SuperMicro SSG-641E with 2 x Intel GOLD 6530 ( 32 cores
each) and 1TB RAM
My plan was/is to use the 3 x 15TB NVME on each server for high performance
pools (like metadata for cephfs or index for RGW)
I have carved them in 3 ( 5 TB each) so I can deploy more PGs hence
increase performance
The SSD are meant to be used for RBD ( proxmox )
The HDD are meant to be used for archiving data using S3 and cephfs.
I can redeploy - please clarify 2 things about osd spec file
1. can I use NVME namespaces ( created with nvme command - see below) or
should I let ceph partition the NVME disk ?
in either case, how do you specify WHICH device/NVME to use for
DB/WAL ( as I have 2 identical ones and want to keep the second one for
adding more HDD in the future )
nvme create-ns $device --nsze=$per_ns_blocks --ncap=$per_ns_blocks
--flbas=0 --dps=0
nvme attach-ns $device --namespace-id=$i --controllers=`nvme
id-ctrl $device
2. to deploy 3 different types of OSDs using the spec I have with the
advice from you , I am guessing this is the correct spec
service_type: osd
service_id: hdd_osd
crush_device_class: hdd_class
placement:
host_pattern: *
spec:
data_devices
rotational: 1
db_devices:
rotational: 0
size: 1000G:1600G
filter_logic: AND
objectstore: bluestore
service_id: ssd_osd
crush_device_class: ssd_class
placement:
host_pattern: *
spec:
data_devices
rotational: 0
size: 6000G:8000G
service_id: nvme_osd
crush_device_class: nvme_class
placement:
host_pattern: *
spec:
data_devices
rotational: 0
size: 4000G:5500G
Many thanks
Steven
On Sun, 29 Jun 2025 at 17:45, Anthony D'Atri <aad(a)dreamsnake.net> wrote:
> So you have NVMe SSD OSDs, SATA SSD OSDs, and HDD OSDs with offload onto
> NVNe SSDs.
>
> Did you have a specific reason to explicitly specify wal_devices_. It’s
> usually fine to just run with the default WAL size, with the WAL colocated
> with the DB. And thus give your DB partitions a bit more space.
>
> What are your use-cases for these three classes of OSDs? Looks like you
> have 42x 20T HDD OSDs, 63x NVMe OSDs, and 84x 7.6T SATA SSD OSDs?
> Apparently with the 15T SSDs divided into 3x OSDs each? How much CPU do
> you have on these nodes? Any specific reason to have chopped up the NVMe
> SSDs into thirds?
>
> It looks to me as though your .mgr pool is using the default
> replicated_rule, which does not specify a device class. This will confound
> the balancer and if enabled the pg_autoscaler.
> I recommend changing the .mgr pool to use the CRUSH rule that the
> non-buckets.data pools use, which should be one that specifies
> 3-replication constrained to one of the SSD device classes. As it is the
> .mgr pool may be placed on any of the three device classes, which is
> trivial with respect to space, but confounds as I mentioned.
>
> Or you could manually edit the CRUSH map and change the #0 replicated_rule
> to specify nvme_class but it sounds like you’re new to Ceph and I don’t
> want to frighten you with that process which unfortunately is still
> old-school. Changing the rule as I suggested will be much safer.
>
> The numbers look like you have all of the RGW pools except buckets.data on
> the nvme_class SSDs, which is fine, but you won’t begin to use all their
> capacity, the index pool will maybe use 5-10% of the capacity used by your
> buckets.data pool over time, depending on your distribution of object sizes
> and the replication strategy of your buckets.data pool. Doing the math
> I’ll speculate that your buckets.data pool is using a … EC 5+2 profile?
> True? If so I might suggest rebuilding if/while you still can. There are
> distinct advantages to having EC K+M < the number of OSD nodes.
>
>
>
>
> Hi,
>
> Yes, I have separate NVME namespaces allocated for WAL and DB to each
> spinning disk
>
>
> Namespaces, or partitions?
>
>
> Does that mean I still have to hunt for the 8TB culprit ?
>
>
> Okay so `ceph cf` shows 8.2 TB of raw space used on the hdd_class OSDs,
> that’s your concern, right?
>
> Please share outputs of the following:
>
> `ceph osd df` (showing a few of each device class)
> `ceph osd dump | grep pool`
> `ceph osd metadata NNNN | egrep /dev\|bluefs_\|bluestore_bdev` for at
> least one OSD of each device class. And run it yourself without specifying
> an OSD ID so it captures all, and see if all OSDs in each device class look
> the same.
> `ceph osd device ls-by-host ceph-host-1
>
> It’s entirely possible that your WAL+DB aren’t actually offloaded to SSDs
> as you intended. Advanced OSD service specs can be tricky.
>
> That’s my suspicion, that the WAL+DB are actually still on your HDDs.
> Which can be migrated in-situ, or you can nuke the site from orbit and
> redeploy.
>
> A note about your OSD specs. Specifying the models as you’re doing is
> totally supported. But think about what happens if you add nodes in the
> future that have different drive SKUs, or you RMA a drive and they send you
> a different SKU as the replacement.
>
> It’s usually more future-proof to use a size range in the spec for each
> osd service instead of `model`, with a bit of margin to account for base 2
> units vs base 10 units.
>
> Here’s an example that creates OSDs on SSDs between 490 and 1200 GB, this
> is on systems that have ~ 1TB nominal drives. The systems also have 2TB
> SATA SSDs that are used for WAL+DB offload, which are above the 1200GB
> limit specified so they aren’t matched.
>
> service_type: osd
> service_id: dashboard-admin-1705602677615
> service_name: osd.dashboard-admin-1705602677615
> placement:
> host_pattern: *
> spec:
> data_devices:
> rotational: 0
> size: 490G:1200G
> filter_logic: AND
> objectstore: bluestore
>
> And here is a spec that matches any HDD larger than 18T and deploys OSDs
> on them without offload. This cluster has 20TB HDDs, so the range of 18+
> TB matches both the SEAGATE_ST20000NM007H and SEAGATE_ST20000NM002D drives
> present.
>
> service_type: osd
> service_id: cost_capacity
> service_name: osd.cost_capacity
> placement:
> host_pattern: noactuallyusedanymore
> spec:
> data_devices:
> rotational: 1
> size: '18T:'
> filter_logic: AND
> objectstore: bluestore
>
> Oh, and make sure that your HDDs and SSDs are all updated to the most
> recent firmware. If you have Dell chassis, run DSU on the nodes and update
> all firmware, but skip the OS drivers. If you have HP chassis, you can get
> firmware update scripts from their web site, but I suspect these aren’t
> HP. If anyone else, they’re likely generic drives and you can get firmware
> updaters from the mfgs respective web sites.
>
> Then reboot nodes one at a time to effect the firmware, letting the
> cluster completely recover between each reboot.
>
>
>
>
> If yes , what would be the most efficient way of finding out what takes
> the space ?
>
> Apologies for sending pictures but we are operating in an air gapped
> environment
>
> I used this spec file to create the OSDs
>
> <image.png>
>
> Here is the osd tree of one of the servers
> all the other 6 are similar
>
> <image.png>
>
> Steven
>
>
> On Sun, 29 Jun 2025 at 14:25, Anthony D'Atri <aad(a)dreamsnake.net> wrote:
>
>> WAL by default rides along with the DB and rarely warrants a separate or
>> larger allocation.
>>
>> Since you say you’ve allocated DB space, does that mean that you have
>> WAL+DB offloaded onto SSDs? If so they don’t contribute to the space used
>> on the hdd device class.
>>
>>
>> > On Jun 29, 2025, at 1:56 PM, Steven Vacaroaia <stef97(a)gmail.com> wrote:
>> >
>> > Hi Janne
>> >
>> > Thanks
>> > That make sense since I have allocated 196GB for DB and 5 GB for WALL
>> for
>> > all 42 spinning OSDs
>> > Again, thanks
>> > Steveb
>> >
>> > On Sun, 29 Jun 2025 at 12:02, Janne Johansson <icepic.dz(a)gmail.com>
>> wrote:
>> >
>> >> Den sön 29 juni 2025 kl 17:22 skrev Steven Vacaroaia <stef97(a)gmail.com
>> >:
>> >>
>> >>> Hi,
>> >>>
>> >>> I just built a new CEPH squid cluster with 7 nodes
>> >>> Since this is brand new, there is no actuall data on it except few
>> test
>> >>> files in the S3 data.bucket
>> >>>
>> >>> Why is "ceph -s" reporting 8 TB of used capacity ?
>> >>>
>> >>
>> >> Because each OSD will have GBs of preallocated data for the RocksDB,
>> >> write-ahead-logs and other structures, and this counts against "raw
>> >> available space", even if you don't have objects of this size put into
>> the
>> >> pools, the creation of the DBs and other things happened at osd
>> creation,
>> >> or when the first object was made, and are there even if you delete the
>> >> object later.
>> >>
>> >> --
>> >> May the most significant bit of your life be positive.
>> >>
>> > _______________________________________________
>> > ceph-users mailing list -- ceph-users(a)ceph.io
>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>
>>
>