Recently deployed cluster showing 9Tb of raw usage without any load deployed - ceph-users

Work Ceph

4 Apr 4 Apr

1:29 a.m.

To add more information, in case that helps: ``` # ceph -s cluster: id: <Pool_UUID> health: HEALTH_OK .... task status: data: pools: 6 pools, 161 pgs objects: 223 objects, 7.0 KiB usage: 9.3 TiB used, 364 TiB / 373 TiB avail pgs: 161 active+clean # ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 373 TiB 364 TiB 9.3 TiB 9.3 TiB 2.50 TOTAL 373 TiB 364 TiB 9.3 TiB 9.3 TiB 2.50 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 0 B 0 0 B 0 115 TiB .rgw.root 2 32 3.6 KiB 8 1.5 MiB 0 115 TiB default.rgw.log 3 32 3.4 KiB 207 6 MiB 0 115 TiB default.rgw.control 4 32 0 B 8 0 B 0 115 TiB default.rgw.meta 5 32 0 B 0 0 B 0 115 TiB rbd 6 32 0 B 0 0 B 0 115 TiB ``` On Mon, Apr 3, 2023 at 10:25 PM Work Ceph <work.ceph.user.mailing(a)gmail.com> wrote:

...

Hello guys! We noticed an unexpected situation. In a recently deployed Ceph cluster we are seeing a raw usage, that is a bit odd. We have the following setup: We have a new cluster with 5 nodes with the following setup: - 128 GB of RAM - 2 cpus Intel(R) Intel Xeon Silver 4210R - 1 NVME of 2 TB for the rocks DB caching - 5 HDDs of 14TB - 1 NIC dual port of 25GiB in BOND mode. Right after deploying the Ceph cluster, we see a raw usage of about 9TiB. However, no load has been applied onto the cluster. Have you guys seen such a situation? Or, can you guys help understand it? We are using Ceph Octopus, and we have set the following configurations: ``` ceph_conf_overrides: global: osd pool default size: 3 osd pool default min size: 1 osd pool default pg autoscale mode: "warn" perf: true rocksdb perf: true mon: mon osd down out interval: 120 osd: bluestore min alloc size hdd: 65536 ``` Any tip or help on how to explain this situation is welcome!

Reply

Igor Fedotov

9:22 a.m.

Do you have standalone DB volumes for your OSD? If so then highly likely RAW usage is that high due to DB volumes space is considered as in-use one already. Could you please share "ceph osd df tree" output to prove that? Thanks, Igor On 4/4/2023 4:25 AM, Work Ceph wrote:

...

Hello guys! We noticed an unexpected situation. In a recently deployed Ceph cluster we are seeing a raw usage, that is a bit odd. We have the following setup: We have a new cluster with 5 nodes with the following setup: - 128 GB of RAM - 2 cpus Intel(R) Intel Xeon Silver 4210R - 1 NVME of 2 TB for the rocks DB caching - 5 HDDs of 14TB - 1 NIC dual port of 25GiB in BOND mode. Right after deploying the Ceph cluster, we see a raw usage of about 9TiB. However, no load has been applied onto the cluster. Have you guys seen such a situation? Or, can you guys help understand it? We are using Ceph Octopus, and we have set the following configurations: ``` ceph_conf_overrides: global: osd pool default size: 3 osd pool default min size: 1 osd pool default pg autoscale mode: "warn" perf: true rocksdb perf: true mon: mon osd down out interval: 120 osd: bluestore min alloc size hdd: 65536 ``` Any tip or help on how to explain this situation is welcome! _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

-- Igor Fedotov Ceph Lead Developer Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx

Reply

Work Ceph

11:18 a.m.

Thank you guys for your replies. The "used space" there is exactly that. It is the accounting for Rocks.DB and WAL. ``` RAW USED: The sum of USED space and the space allocated the db and wal BlueStore partitions. ``` There is one detail I do not understand. We are off-loading WAL and RocksDB to an NVME device; however, Ceph still seems to think that we use our data plane disks to store those elements. We have about 375TB (5 * 5 * 15) in HDD disks, and Ceph seems to be discounting from the usable space the volume (space) dedicated to WAL and Rocks.DB, which are applied into different disks; therefore, it shows as usable space 364 TB (after removing the space dedicated to WAL and Rocks.DB, which are in another device). Is that a bug of some sort? On Tue, Apr 4, 2023 at 6:31 AM Igor Fedotov <igor.fedotov(a)croit.io> wrote:

...

Please also note that total cluster size reported below as SIZE apparently includes DB volumes: # ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 373 TiB 364 TiB 9.3 TiB 9.3 TiB 2.50 On 4/4/2023 12:22 PM, Igor Fedotov wrote:

Do you have standalone DB volumes for your OSD? If so then highly likely RAW usage is that high due to DB volumes space is considered as in-use one already. Could you please share "ceph osd df tree" output to prove that? Thanks, Igor On 4/4/2023 4:25 AM, Work Ceph wrote:

Hello guys! We noticed an unexpected situation. In a recently deployed Ceph cluster we are seeing a raw usage, that is a bit odd. We have the following setup: We have a new cluster with 5 nodes with the following setup: - 128 GB of RAM - 2 cpus Intel(R) Intel Xeon Silver 4210R - 1 NVME of 2 TB for the rocks DB caching - 5 HDDs of 14TB - 1 NIC dual port of 25GiB in BOND mode. Right after deploying the Ceph cluster, we see a raw usage of about 9TiB. However, no load has been applied onto the cluster. Have you guys seen such a situation? Or, can you guys help understand it? We are using Ceph Octopus, and we have set the following configurations: ``` ceph_conf_overrides: global: osd pool default size: 3 osd pool default min size: 1 osd pool default pg autoscale mode: "warn" perf: true rocksdb perf: true mon: mon osd down out interval: 120 osd: bluestore min alloc size hdd: 65536 ``` Any tip or help on how to explain this situation is welcome! _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

-- Igor Fedotov Ceph Lead Developer Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx

Reply

Work Ceph

12:47 p.m.

The disks are 14.9, but the exact size of them does not matter much in this context. We figured out the issue. The raw used space accounts for the Rocks.DB and WAL space. Therefore, as we dedicated an NVME device in each host for them, Ceph is showing that space as used space already. It is funny though that it is still decrementing/discounting that space from the total space we have for HDDs when presenting the status. That seems to be a bug or incorrect information that is shown. On Tue, Apr 4, 2023 at 8:31 AM Igor Fedotov <igor.fedotov(a)croit.io> wrote:

...

Originally you mentioned 14TB HDDs not 15TB. Could this be a trick? If not - please share "ceph osd df tree" output? On 4/4/2023 2:18 PM, Work Ceph wrote: Thank you guys for your replies. The "used space" there is exactly that. It is the accounting for Rocks.DB and WAL. ``` RAW USED: The sum of USED space and the space allocated the db and wal BlueStore partitions. ``` There is one detail I do not understand. We are off-loading WAL and RocksDB to an NVME device; however, Ceph still seems to think that we use our data plane disks to store those elements. We have about 375TB (5 * 5 * 15) in HDD disks, and Ceph seems to be discounting from the usable space the volume (space) dedicated to WAL and Rocks.DB, which are applied into different disks; therefore, it shows as usable space 364 TB (after removing the space dedicated to WAL and Rocks.DB, which are in another device). Is that a bug of some sort? On Tue, Apr 4, 2023 at 6:31 AM Igor Fedotov <igor.fedotov(a)croit.io> wrote:

Please also note that total cluster size reported below as SIZE apparently includes DB volumes: # ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 373 TiB 364 TiB 9.3 TiB 9.3 TiB 2.50 On 4/4/2023 12:22 PM, Igor Fedotov wrote:

Do you have standalone DB volumes for your OSD? If so then highly likely RAW usage is that high due to DB volumes space is considered as in-use one already. Could you please share "ceph osd df tree" output to prove that? Thanks, Igor On 4/4/2023 4:25 AM, Work Ceph wrote: > Hello guys! > > > We noticed an unexpected situation. In a recently deployed Ceph > cluster we > are seeing a raw usage, that is a bit odd. We have the following setup: > > > We have a new cluster with 5 nodes with the following setup: > > - 128 GB of RAM > - 2 cpus Intel(R) Intel Xeon Silver 4210R > - 1 NVME of 2 TB for the rocks DB caching > - 5 HDDs of 14TB > - 1 NIC dual port of 25GiB in BOND mode. > > > Right after deploying the Ceph cluster, we see a raw usage of about > 9TiB. > However, no load has been applied onto the cluster. Have you guys > seen such > a situation? Or, can you guys help understand it? > > > We are using Ceph Octopus, and we have set the following

configurations:

``` ceph_conf_overrides: global: osd pool default size: 3 osd pool default min size: 1 osd pool default pg autoscale mode: "warn" perf: true rocksdb perf: true mon: mon osd down out interval: 120 osd: bluestore min alloc size hdd: 65536 ``` Any tip or help on how to explain this situation is welcome! _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

-- Igor Fedotov Ceph Lead Developer Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx --

Igor Fedotov Ceph Lead Developer Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx

Reply