Hello, My name is Mihai, and I have started using CEPH this mount for a HPC cluster. When was lunch in the production the available space shown was 80TB now is 16TB and I didn't do anything, while I'm having 12 OSD (SSD of 14TB): sudo ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 167.64825 root default -3 167.64825 host sto-core-hpc01 0 ssd 13.97069 osd.0 up 1.00000 1.00000 1 ssd 13.97069 osd.1 up 1.00000 1.00000 2 ssd 13.97069 osd.2 up 1.00000 1.00000 3 ssd 13.97069 osd.3 up 1.00000 1.00000 4 ssd 13.97069 osd.4 up 1.00000 1.00000 5 ssd 13.97069 osd.5 up 1.00000 1.00000 6 ssd 13.97069 osd.6 up 1.00000 1.00000 7 ssd 13.97069 osd.7 up 1.00000 1.00000 8 ssd 13.97069 osd.8 up 1.00000 1.00000 9 ssd 13.97069 osd.9 up 1.00000 1.00000 10 ssd 13.97069 osd.10 up 1.00000 1.00000 11 ssd 13.97069 osd.11 up 1.00000 1.00000 sudo ceph df detail --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 168 TiB 156 TiB 12 TiB 12 TiB 7.12 TOTAL 168 TiB 156 TiB 12 TiB 12 TiB 7.12 --- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR .mgr 1 1 705 KiB 705 KiB 0 B 2 1.4 MiB 1.4 MiB 0 B 0 8.1 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.meta 2 16 270 MiB 270 MiB 0 B 85.96k 270 MiB 270 MiB 0 B 0 16 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.data 3 129 12 TiB 12 TiB 0 B 3.73M 12 TiB 12 TiB 0 B 42.49 16 TiB N/A N/A N/A 0 B 0 B While on the client side I have this: $ df -h 10.18.31.1:6789:/ 21T 13T 8.1T 61% /data I don't know where it's gone all the space that was at the beginning. Someone has any hint? Best regards, Mihai
Look at `ceph osd df`. Is the balancer enabled?
On Mar 27, 2025, at 8:50 AM, Mihai Ciubancan <mihai.ciubancan@eli-np.ro> wrote:
Hello,
My name is Mihai, and I have started using CEPH this mount for a HPC cluster. When was lunch in the production the available space shown was 80TB now is 16TB and I didn't do anything, while I'm having 12 OSD (SSD of 14TB):
sudo ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 167.64825 root default -3 167.64825 host sto-core-hpc01 0 ssd 13.97069 osd.0 up 1.00000 1.00000 1 ssd 13.97069 osd.1 up 1.00000 1.00000 2 ssd 13.97069 osd.2 up 1.00000 1.00000 3 ssd 13.97069 osd.3 up 1.00000 1.00000 4 ssd 13.97069 osd.4 up 1.00000 1.00000 5 ssd 13.97069 osd.5 up 1.00000 1.00000 6 ssd 13.97069 osd.6 up 1.00000 1.00000 7 ssd 13.97069 osd.7 up 1.00000 1.00000 8 ssd 13.97069 osd.8 up 1.00000 1.00000 9 ssd 13.97069 osd.9 up 1.00000 1.00000 10 ssd 13.97069 osd.10 up 1.00000 1.00000 11 ssd 13.97069 osd.11 up 1.00000 1.00000
sudo ceph df detail --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 168 TiB 156 TiB 12 TiB 12 TiB 7.12 TOTAL 168 TiB 156 TiB 12 TiB 12 TiB 7.12
--- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR .mgr 1 1 705 KiB 705 KiB 0 B 2 1.4 MiB 1.4 MiB 0 B 0 8.1 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.meta 2 16 270 MiB 270 MiB 0 B 85.96k 270 MiB 270 MiB 0 B 0 16 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.data 3 129 12 TiB 12 TiB 0 B 3.73M 12 TiB 12 TiB 0 B 42.49 16 TiB N/A N/A N/A 0 B 0 B
While on the client side I have this:
$ df -h 10.18.31.1:6789:/ 21T 13T 8.1T 61% /data
I don't know where it's gone all the space that was at the beginning. Someone has any hint?
Best regards, Mihai _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Anthony, Thanks for the answer: The output of 'ceph osd df' is: ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB 27 GiB 156 TiB 7.13 1.00 - root default -3 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB 27 GiB 156 TiB 7.13 1.00 - host sto-core-hpc01 0 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.0 1 ssd 13.97069 1.00000 14 TiB 12 TiB 12 TiB 6 KiB 26 GiB 2.0 TiB 85.53 12.00 129 up osd.1 2 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.2 3 ssd 13.97069 1.00000 14 TiB 1.7 GiB 258 MiB 100 MiB 1.3 GiB 14 TiB 0.01 0.00 16 up osd.3 4 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.4 5 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.5 6 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.6 7 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.7 8 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB 63 MiB 14 TiB 0 0 1 up osd.8 9 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.9 10 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.10 11 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB 63 MiB 14 TiB 0 0 1 up osd.11 TOTAL 168 TiB 12 TiB 12 TiB 100 MiB 27 GiB 156 TiB 7.13 So all the date is on osd.1 But I have checked the balancer and seems active: ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.000368", "last_optimize_started": "Fri Mar 28 10:55:06 2025", "mode": "upmap", "no_optimization_needed": false, "optimize_result": "Some objects (0.500000) are degraded; try again later", "plans": [] } But the output of the commnad 'ceph config dump|grep balancer' gives me nothing. Best, Mihai On 2025-03-27 23:06, Anthony D'Atri wrote:
Look at `ceph osd df`. Is the balancer enabled?
On Mar 27, 2025, at 8:50 AM, Mihai Ciubancan <mihai.ciubancan@eli-np.ro> wrote:
Hello,
My name is Mihai, and I have started using CEPH this mount for a HPC cluster. When was lunch in the production the available space shown was 80TB now is 16TB and I didn't do anything, while I'm having 12 OSD (SSD of 14TB):
sudo ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 167.64825 root default -3 167.64825 host sto-core-hpc01 0 ssd 13.97069 osd.0 up 1.00000 1.00000 1 ssd 13.97069 osd.1 up 1.00000 1.00000 2 ssd 13.97069 osd.2 up 1.00000 1.00000 3 ssd 13.97069 osd.3 up 1.00000 1.00000 4 ssd 13.97069 osd.4 up 1.00000 1.00000 5 ssd 13.97069 osd.5 up 1.00000 1.00000 6 ssd 13.97069 osd.6 up 1.00000 1.00000 7 ssd 13.97069 osd.7 up 1.00000 1.00000 8 ssd 13.97069 osd.8 up 1.00000 1.00000 9 ssd 13.97069 osd.9 up 1.00000 1.00000 10 ssd 13.97069 osd.10 up 1.00000 1.00000 11 ssd 13.97069 osd.11 up 1.00000 1.00000
sudo ceph df detail --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 168 TiB 156 TiB 12 TiB 12 TiB 7.12 TOTAL 168 TiB 156 TiB 12 TiB 12 TiB 7.12
--- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR .mgr 1 1 705 KiB 705 KiB 0 B 2 1.4 MiB 1.4 MiB 0 B 0 8.1 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.meta 2 16 270 MiB 270 MiB 0 B 85.96k 270 MiB 270 MiB 0 B 0 16 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.data 3 129 12 TiB 12 TiB 0 B 3.73M 12 TiB 12 TiB 0 B 42.49 16 TiB N/A N/A N/A 0 B 0 B
While on the client side I have this:
$ df -h 10.18.31.1:6789:/ 21T 13T 8.1T 61% /data
I don't know where it's gone all the space that was at the beginning. Someone has any hint?
Best regards, Mihai _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Do you use size 1 for your data? You know that's bad, right? Please share 'ceph osd pool ls detail'. Also, it's recommended to use a power of 2 for pg_num, so you should decrease the pg_num for the pool cephfs.cephfs.data. Zitat von Mihai Ciubancan <mihai.ciubancan@eli-np.ro>:
Hi Anthony, Thanks for the answer:
The output of 'ceph osd df' is:
ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB 27 GiB 156 TiB 7.13 1.00 - root default -3 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB 27 GiB 156 TiB 7.13 1.00 - host sto-core-hpc01 0 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.0 1 ssd 13.97069 1.00000 14 TiB 12 TiB 12 TiB 6 KiB 26 GiB 2.0 TiB 85.53 12.00 129 up osd.1 2 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.2 3 ssd 13.97069 1.00000 14 TiB 1.7 GiB 258 MiB 100 MiB 1.3 GiB 14 TiB 0.01 0.00 16 up osd.3 4 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.4 5 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.5 6 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.6 7 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.7 8 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB 63 MiB 14 TiB 0 0 1 up osd.8 9 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.9 10 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.10 11 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB 63 MiB 14 TiB 0 0 1 up osd.11 TOTAL 168 TiB 12 TiB 12 TiB 100 MiB 27 GiB 156 TiB 7.13
So all the date is on osd.1
But I have checked the balancer and seems active: ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.000368", "last_optimize_started": "Fri Mar 28 10:55:06 2025", "mode": "upmap", "no_optimization_needed": false, "optimize_result": "Some objects (0.500000) are degraded; try again later", "plans": [] }
But the output of the commnad 'ceph config dump|grep balancer' gives me nothing.
Best, Mihai
On 2025-03-27 23:06, Anthony D'Atri wrote:
Look at `ceph osd df`. Is the balancer enabled?
On Mar 27, 2025, at 8:50 AM, Mihai Ciubancan <mihai.ciubancan@eli-np.ro> wrote:
Hello,
My name is Mihai, and I have started using CEPH this mount for a HPC cluster. When was lunch in the production the available space shown was 80TB now is 16TB and I didn't do anything, while I'm having 12 OSD (SSD of 14TB):
sudo ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 167.64825 root default -3 167.64825 host sto-core-hpc01 0 ssd 13.97069 osd.0 up 1.00000 1.00000 1 ssd 13.97069 osd.1 up 1.00000 1.00000 2 ssd 13.97069 osd.2 up 1.00000 1.00000 3 ssd 13.97069 osd.3 up 1.00000 1.00000 4 ssd 13.97069 osd.4 up 1.00000 1.00000 5 ssd 13.97069 osd.5 up 1.00000 1.00000 6 ssd 13.97069 osd.6 up 1.00000 1.00000 7 ssd 13.97069 osd.7 up 1.00000 1.00000 8 ssd 13.97069 osd.8 up 1.00000 1.00000 9 ssd 13.97069 osd.9 up 1.00000 1.00000 10 ssd 13.97069 osd.10 up 1.00000 1.00000 11 ssd 13.97069 osd.11 up 1.00000 1.00000
sudo ceph df detail --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 168 TiB 156 TiB 12 TiB 12 TiB 7.12 TOTAL 168 TiB 156 TiB 12 TiB 12 TiB 7.12
--- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR .mgr 1 1 705 KiB 705 KiB 0 B 2 1.4 MiB 1.4 MiB 0 B 0 8.1 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.meta 2 16 270 MiB 270 MiB 0 B 85.96k 270 MiB 270 MiB 0 B 0 16 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.data 3 129 12 TiB 12 TiB 0 B 3.73M 12 TiB 12 TiB 0 B 42.49 16 TiB N/A N/A N/A 0 B 0 B
While on the client side I have this:
$ df -h 10.18.31.1:6789:/ 21T 13T 8.1T 61% /data
I don't know where it's gone all the space that was at the beginning. Someone has any hint?
Best regards, Mihai _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugen, Thanks for your answer. Please find below the output of the command: ceph osd pool ls detail pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 323 flags hashpspool,nearfull stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.50 pool 2 'cephfs.cephfs.meta' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 1 pgp_num_target 16 autoscale_mode on last_change 295 lfor 0/0/54 flags hashpspool,nearfull stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 12.50 pool 3 'cephfs.cephfs.data' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 129 pgp_num 1 pg_num_target 512 pgp_num_target 512 autoscale_mode on last_change 326 lfor 0/0/54 flags hashpspool,nearfull,bulk stripe_width 0 application cephfs read_balance_score 12.50 How can I decrese the number of pg_num? Best, Mihai On 2025-03-28 13:19, Eugen Block wrote:
Do you use size 1 for your data? You know that's bad, right? Please share 'ceph osd pool ls detail'. Also, it's recommended to use a power of 2 for pg_num, so you should decrease the pg_num for the pool cephfs.cephfs.data.
Zitat von Mihai Ciubancan <mihai.ciubancan@eli-np.ro>:
Hi Anthony, Thanks for the answer:
The output of 'ceph osd df' is:
ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB 27 GiB 156 TiB 7.13 1.00 - root default -3 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB 27 GiB 156 TiB 7.13 1.00 - host sto-core-hpc01 0 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.0 1 ssd 13.97069 1.00000 14 TiB 12 TiB 12 TiB 6 KiB 26 GiB 2.0 TiB 85.53 12.00 129 up osd.1 2 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.2 3 ssd 13.97069 1.00000 14 TiB 1.7 GiB 258 MiB 100 MiB 1.3 GiB 14 TiB 0.01 0.00 16 up osd.3 4 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.4 5 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.5 6 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.6 7 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.7 8 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB 63 MiB 14 TiB 0 0 1 up osd.8 9 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.9 10 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.10 11 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB 63 MiB 14 TiB 0 0 1 up osd.11 TOTAL 168 TiB 12 TiB 12 TiB 100 MiB 27 GiB 156 TiB 7.13
So all the date is on osd.1
But I have checked the balancer and seems active: ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.000368", "last_optimize_started": "Fri Mar 28 10:55:06 2025", "mode": "upmap", "no_optimization_needed": false, "optimize_result": "Some objects (0.500000) are degraded; try again later", "plans": [] }
But the output of the commnad 'ceph config dump|grep balancer' gives me nothing.
Best, Mihai
On 2025-03-27 23:06, Anthony D'Atri wrote:
Look at `ceph osd df`. Is the balancer enabled?
On Mar 27, 2025, at 8:50 AM, Mihai Ciubancan <mihai.ciubancan@eli-np.ro> wrote:
Hello,
My name is Mihai, and I have started using CEPH this mount for a HPC cluster. When was lunch in the production the available space shown was 80TB now is 16TB and I didn't do anything, while I'm having 12 OSD (SSD of 14TB):
sudo ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 167.64825 root default -3 167.64825 host sto-core-hpc01 0 ssd 13.97069 osd.0 up 1.00000 1.00000 1 ssd 13.97069 osd.1 up 1.00000 1.00000 2 ssd 13.97069 osd.2 up 1.00000 1.00000 3 ssd 13.97069 osd.3 up 1.00000 1.00000 4 ssd 13.97069 osd.4 up 1.00000 1.00000 5 ssd 13.97069 osd.5 up 1.00000 1.00000 6 ssd 13.97069 osd.6 up 1.00000 1.00000 7 ssd 13.97069 osd.7 up 1.00000 1.00000 8 ssd 13.97069 osd.8 up 1.00000 1.00000 9 ssd 13.97069 osd.9 up 1.00000 1.00000 10 ssd 13.97069 osd.10 up 1.00000 1.00000 11 ssd 13.97069 osd.11 up 1.00000 1.00000
sudo ceph df detail --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 168 TiB 156 TiB 12 TiB 12 TiB 7.12 TOTAL 168 TiB 156 TiB 12 TiB 12 TiB 7.12
--- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR .mgr 1 1 705 KiB 705 KiB 0 B 2 1.4 MiB 1.4 MiB 0 B 0 8.1 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.meta 2 16 270 MiB 270 MiB 0 B 85.96k 270 MiB 270 MiB 0 B 0 16 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.data 3 129 12 TiB 12 TiB 0 B 3.73M 12 TiB 12 TiB 0 B 42.49 16 TiB N/A N/A N/A 0 B 0 B
While on the client side I have this:
$ df -h 10.18.31.1:6789:/ 21T 13T 8.1T 61% /data
I don't know where it's gone all the space that was at the beginning. Someone has any hint?
Best regards, Mihai _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
You have the autoscaler enabled, but it stuck changing the pg_num. The default replicated_rule (which you are using) requires as many hosts as your pool size is, so in your case 2. (If you value your data, don't use replicated pools with size 2.) You could make it work with only one host (as Anthony suggested with osd_crush_chooseleaf_type 0), but you don't any real resiliency. I recommend to reconsider your setup. Zitat von Mihai Ciubancan <mihai.ciubancan@eli-np.ro>:
Hi Eugen,
Thanks for your answer. Please find below the output of the command:
ceph osd pool ls detail pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 323 flags hashpspool,nearfull stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.50 pool 2 'cephfs.cephfs.meta' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 1 pgp_num_target 16 autoscale_mode on last_change 295 lfor 0/0/54 flags hashpspool,nearfull stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 12.50 pool 3 'cephfs.cephfs.data' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 129 pgp_num 1 pg_num_target 512 pgp_num_target 512 autoscale_mode on last_change 326 lfor 0/0/54 flags hashpspool,nearfull,bulk stripe_width 0 application cephfs read_balance_score 12.50
How can I decrese the number of pg_num?
Best, Mihai
On 2025-03-28 13:19, Eugen Block wrote:
Do you use size 1 for your data? You know that's bad, right? Please share 'ceph osd pool ls detail'. Also, it's recommended to use a power of 2 for pg_num, so you should decrease the pg_num for the pool cephfs.cephfs.data.
Zitat von Mihai Ciubancan <mihai.ciubancan@eli-np.ro>:
Hi Anthony, Thanks for the answer:
The output of 'ceph osd df' is:
ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB 27 GiB 156 TiB 7.13 1.00 - root default -3 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB 27 GiB 156 TiB 7.13 1.00 - host sto-core-hpc01 0 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.0 1 ssd 13.97069 1.00000 14 TiB 12 TiB 12 TiB 6 KiB 26 GiB 2.0 TiB 85.53 12.00 129 up osd.1 2 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.2 3 ssd 13.97069 1.00000 14 TiB 1.7 GiB 258 MiB 100 MiB 1.3 GiB 14 TiB 0.01 0.00 16 up osd.3 4 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.4 5 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.5 6 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.6 7 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.7 8 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB 63 MiB 14 TiB 0 0 1 up osd.8 9 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.9 10 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.10 11 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB 63 MiB 14 TiB 0 0 1 up osd.11 TOTAL 168 TiB 12 TiB 12 TiB 100 MiB 27 GiB 156 TiB 7.13
So all the date is on osd.1
But I have checked the balancer and seems active: ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.000368", "last_optimize_started": "Fri Mar 28 10:55:06 2025", "mode": "upmap", "no_optimization_needed": false, "optimize_result": "Some objects (0.500000) are degraded; try again later", "plans": [] }
But the output of the commnad 'ceph config dump|grep balancer' gives me nothing.
Best, Mihai
On 2025-03-27 23:06, Anthony D'Atri wrote:
Look at `ceph osd df`. Is the balancer enabled?
On Mar 27, 2025, at 8:50 AM, Mihai Ciubancan <mihai.ciubancan@eli-np.ro> wrote:
Hello,
My name is Mihai, and I have started using CEPH this mount for a HPC cluster. When was lunch in the production the available space shown was 80TB now is 16TB and I didn't do anything, while I'm having 12 OSD (SSD of 14TB):
sudo ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 167.64825 root default -3 167.64825 host sto-core-hpc01 0 ssd 13.97069 osd.0 up 1.00000 1.00000 1 ssd 13.97069 osd.1 up 1.00000 1.00000 2 ssd 13.97069 osd.2 up 1.00000 1.00000 3 ssd 13.97069 osd.3 up 1.00000 1.00000 4 ssd 13.97069 osd.4 up 1.00000 1.00000 5 ssd 13.97069 osd.5 up 1.00000 1.00000 6 ssd 13.97069 osd.6 up 1.00000 1.00000 7 ssd 13.97069 osd.7 up 1.00000 1.00000 8 ssd 13.97069 osd.8 up 1.00000 1.00000 9 ssd 13.97069 osd.9 up 1.00000 1.00000 10 ssd 13.97069 osd.10 up 1.00000 1.00000 11 ssd 13.97069 osd.11 up 1.00000 1.00000
sudo ceph df detail --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 168 TiB 156 TiB 12 TiB 12 TiB 7.12 TOTAL 168 TiB 156 TiB 12 TiB 12 TiB 7.12
--- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR .mgr 1 1 705 KiB 705 KiB 0 B 2 1.4 MiB 1.4 MiB 0 B 0 8.1 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.meta 2 16 270 MiB 270 MiB 0 B 85.96k 270 MiB 270 MiB 0 B 0 16 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.data 3 129 12 TiB 12 TiB 0 B 3.73M 12 TiB 12 TiB 0 B 42.49 16 TiB N/A N/A N/A 0 B 0 B
While on the client side I have this:
$ df -h 10.18.31.1:6789:/ 21T 13T 8.1T 61% /data
I don't know where it's gone all the space that was at the beginning. Someone has any hint?
Best regards, Mihai _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The problem is that I have only one host, and the host doesn't have a RAID controller...I have set as Anthone suggested, but nothing changed.Mihai -------- Original message --------From: Eugen Block <eblock@nde.ag> Date: 3/28/25 3:47 PM (GMT+02:00) To: Mihai Ciubancan <mihai.ciubancan@eli-np.ro> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: space size issue You have the autoscaler enabled, but it stuck changing the pg_num. The default replicated_rule (which you are using) requires as many hosts as your pool size is, so in your case 2. (If you value your data, don't use replicated pools with size 2.)You could make it work with only one host (as Anthony suggested with osd_crush_chooseleaf_type 0), but you don't any real resiliency. I recommend to reconsider your setup.Zitat von Mihai Ciubancan <mihai.ciubancan@eli-np.ro>:> Hi Eugen,>> Thanks for your answer. Please find below the output of the command:>> ceph osd pool ls detail> pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash > rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 323 flags > hashpspool,nearfull stripe_width 0 pg_num_max 32 pg_num_min 1 > application mgr read_balance_score 12.50> pool 2 'cephfs.cephfs.meta' replicated size 2 min_size 1 crush_rule > 0 object_hash rjenkins pg_num 16 pgp_num 1 pgp_num_target 16 > autoscale_mode on last_change 295 lfor 0/0/54 flags > hashpspool,nearfull stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 > recovery_priority 5 application cephfs read_balance_score 12.50> pool 3 'cephfs.cephfs.data' replicated size 2 min_size 1 crush_rule > 0 object_hash rjenkins pg_num 129 pgp_num 1 pg_num_target 512 > pgp_num_target 512 autoscale_mode on last_change 326 lfor 0/0/54 > flags hashpspool,nearfull,bulk stripe_width 0 application cephfs > read_balance_score 12.50>>> How can I decrese the number of pg_num?>> Best,> Mihai>>> On 2025-03-28 13:19, Eugen Block wrote:>> Do you use size 1 for your data? You know that's bad, right? Please>> share 'ceph osd pool ls detail'.>> Also, it's recommended to use a power of 2 for pg_num, so you should>> decrease the pg_num for the pool cephfs.cephfs.data.>>>> Zitat von Mihai Ciubancan <mihai.ciubancan@eli-np.ro>:>>>>> Hi Anthony,>>> Thanks for the answer:>>>>>> The output of 'ceph osd df' is:>>>>>> ceph osd df tree>>> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP >>> META AVAIL %USE VAR PGS STATUS TYPE NAME>>> -1 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB >>> 27 GiB 156 TiB 7.13 1.00 - root default>>> -3 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB >>> 27 GiB 156 TiB 7.13 1.00 - host >>> sto-core-hpc01>>> 0 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.0>>> 1 ssd 13.97069 1.00000 14 TiB 12 TiB 12 TiB 6 KiB >>> 26 GiB 2.0 TiB 85.53 12.00 129 up osd.1>>> 2 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.2>>> 3 ssd 13.97069 1.00000 14 TiB 1.7 GiB 258 MiB 100 MiB >>> 1.3 GiB 14 TiB 0.01 0.00 16 up osd.3>>> 4 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.4>>> 5 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.5>>> 6 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.6>>> 7 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.7>>> 8 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB >>> 63 MiB 14 TiB 0 0 1 up osd.8>>> 9 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.9>>> 10 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.10>>> 11 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB >>> 63 MiB 14 TiB 0 0 1 up osd.11>>> TOTAL 168 TiB 12 TiB 12 TiB 100 MiB >>> 27 GiB 156 TiB 7.13>>>>>> So all the date is on osd.1>>>>>> But I have checked the balancer and seems active:>>> ceph balancer status>>> {>>> "active": true,>>> "last_optimize_duration": "0:00:00.000368",>>> "last_optimize_started": "Fri Mar 28 10:55:06 2025",>>> "mode": "upmap",>>> "no_optimization_needed": false,>>> "optimize_result": "Some objects (0.500000) are degraded; try >>> again later",>>> "plans": []>>> }>>>>>> But the output of the commnad 'ceph config dump|grep balancer' >>> gives me nothing.>>>>>> Best,>>> Mihai>>>>>> On 2025-03-27 23:06, Anthony D'Atri wrote:>>>> Look at `ceph osd df`. Is the balancer enabled?>>>>>>>>> On Mar 27, 2025, at 8:50 AM, Mihai Ciubancan >>>>> <mihai.ciubancan@eli-np.ro> wrote:>>>>>>>>>> Hello,>>>>>>>>>> My name is Mihai, and I have started using CEPH this mount for a >>>>> HPC cluster.>>>>> When was lunch in the production the available space shown was >>>>> 80TB now is 16TB and I didn't do anything, while I'm having 12 >>>>> OSD (SSD of 14TB):>>>>>>>>>> sudo ceph osd tree>>>>> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF>>>>> -1 167.64825 root default>>>>> -3 167.64825 host sto-core-hpc01>>>>> 0 ssd 13.97069 osd.0 up 1.00000 1.00000>>>>> 1 ssd 13.97069 osd.1 up 1.00000 1.00000>>>>> 2 ssd 13.97069 osd.2 up 1.00000 1.00000>>>>> 3 ssd 13.97069 osd.3 up 1.00000 1.00000>>>>> 4 ssd 13.97069 osd.4 up 1.00000 1.00000>>>>> 5 ssd 13.97069 osd.5 up 1.00000 1.00000>>>>> 6 ssd 13.97069 osd.6 up 1.00000 1.00000>>>>> 7 ssd 13.97069 osd.7 up 1.00000 1.00000>>>>> 8 ssd 13.97069 osd.8 up 1.00000 1.00000>>>>> 9 ssd 13.97069 osd.9 up 1.00000 1.00000>>>>> 10 ssd 13.97069 osd.10 up 1.00000 1.00000>>>>> 11 ssd 13.97069 osd.11 up 1.00000 1.00000>>>>>>>>>> sudo ceph df detail>>>>> --- RAW STORAGE --->>>>> CLASS SIZE AVAIL USED RAW USED %RAW USED>>>>> ssd 168 TiB 156 TiB 12 TiB 12 TiB 7.12>>>>> TOTAL 168 TiB 156 TiB 12 TiB 12 TiB 7.12>>>>>>>>>> --- POOLS --->>>>> POOL ID PGS STORED (DATA) (OMAP) OBJECTS >>>>> USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA >>>>> BYTES DIRTY USED COMPR UNDER COMPR>>>>> .mgr 1 1 705 KiB 705 KiB 0 B 2 >>>>> 1.4 MiB 1.4 MiB 0 B 0 8.1 TiB N/A >>>>> N/A N/A 0 B 0 B>>>>> cephfs.cephfs.meta 2 16 270 MiB 270 MiB 0 B 85.96k >>>>> 270 MiB 270 MiB 0 B 0 16 TiB N/A >>>>> N/A N/A 0 B 0 B>>>>> cephfs.cephfs.data 3 129 12 TiB 12 TiB 0 B 3.73M >>>>> 12 TiB 12 TiB 0 B 42.49 16 TiB N/A >>>>> N/A N/A 0 B 0 B>>>>>>>>>> While on the client side I have this:>>>>>>>>>> $ df -h>>>>> 10.18.31.1:6789:/ 21T 13T 8.1T 61% /data>>>>>>>>>> I don't know where it's gone all the space that was at the beginning.>>>>> Someone has any hint?>>>>>>>>>> Best regards,>>>>> Mihai>>>>> _______________________________________________>>>>> ceph-users mailing list -- ceph-users@ceph.io>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io>>> _______________________________________________>>> ceph-users mailing list -- ceph-users@ceph.io>>> To unsubscribe send an email to ceph-users-leave@ceph.io>>>>>> _______________________________________________>> ceph-users mailing list -- ceph-users@ceph.io>> To unsubscribe send an email to ceph-users-leave@ceph.io_______________________________________________ceph-users mailing list -- ceph-users@ceph.ioTo unsubscribe send an email to ceph-users-leave@ceph.io
To get everything up to a working state, you will need to set your failure domain to "osd" instead of "host" in the default rule, and as it has been said before, pool size should be 3 and min_size 2. With that said, you will eventually need more hosts to get the most out of ceph. Den 2025-03-28 kl. 16:05, skrev mihai.ciubancan:
The problem is that I have only one host, and the host doesn't have a RAID controller...I have set as Anthone suggested, but nothing changed.Mihai -------- Original message --------From: Eugen Block <eblock@nde.ag> Date: 3/28/25 3:47 PM (GMT+02:00) To: Mihai Ciubancan <mihai.ciubancan@eli-np.ro> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: space size issue You have the autoscaler enabled, but it stuck changing the pg_num. The default replicated_rule (which you are using) requires as many hosts as your pool size is, so in your case 2. (If you value your data, don't use replicated pools with size 2.)You could make it work with only one host (as Anthony suggested with osd_crush_chooseleaf_type 0), but you don't any real resiliency. I recommend to reconsider your setup.Zitat von Mihai Ciubancan <mihai.ciubancan@eli-np.ro>:> Hi Eugen,>> Thanks for your answer. Please find below the output of the command:>> ceph osd pool ls detail> pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash > rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 323 flags > hashpspool,nearfull stripe_width 0 pg_num_max 32 pg_num_min 1 > application mgr read_balance_score 12.50> pool 2 'cephfs.cephfs.meta' replicated size 2 min_size 1 crush_rule > 0 object_hash rjenkins pg_num 16 pgp_num 1 pgp_num_target 16 > autoscale_mode on last_change 295 lfor 0/0/54 flags > hashpspool,nearfull stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 > recovery_priority 5 application cephfs read_balance_score 12.50> pool 3 'cephfs.cephfs.data' replicated size 2 min_size 1 crush_rule > 0 object_hash rjenkins pg_num 129 pgp_num 1 pg_num_target 512 > pgp_num_target 512 autoscale_mode on last_change 326 lfor 0/0/54 > flags hashpspool,nearfull,bulk stripe_width 0 application cephfs > read_balance_score 12.50>>> How can I decrese the number of pg_num?>> Best,> Mihai>>> On 2025-03-28 13:19, Eugen Block wrote:>> Do you use size 1 for your data? You know that's bad, right? Please>> share 'ceph osd pool ls detail'.>> Also, it's recommended to use a power of 2 for pg_num, so you should>> decrease the pg_num for the pool cephfs.cephfs.data.>>>> Zitat von Mihai Ciubancan <mihai.ciubancan@eli-np.ro>:>>>>> Hi Anthony,>>> Thanks for the answer:>>>>>> The output of 'ceph osd df' is:>>>>>> ceph osd df tree>>> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP >>> META AVAIL %USE VAR PGS STATUS TYPE NAME>>> -1 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB >>> 27 GiB 156 TiB 7.13 1.00 - root default>>> -3 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB >>> 27 GiB 156 TiB 7.13 1.00 - host >>> sto-core-hpc01>>> 0 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.0>>> 1 ssd 13.97069 1.00000 14 TiB 12 TiB 12 TiB 6 KiB >>> 26 GiB 2.0 TiB 85.53 12.00 129 up osd.1>>> 2 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.2>>> 3 ssd 13.97069 1.00000 14 TiB 1.7 GiB 258 MiB 100 MiB >>> 1.3 GiB 14 TiB 0.01 0.00 16 up osd.3>>> 4 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.4>>> 5 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.5>>> 6 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.6>>> 7 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.7>>> 8 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB >>> 63 MiB 14 TiB 0 0 1 up osd.8>>> 9 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.9>>> 10 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.10>>> 11 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB >>> 63 MiB 14 TiB 0 0 1 up osd.11>>> TOTAL 168 TiB 12 TiB 12 TiB 100 MiB >>> 27 GiB 156 TiB 7.13>>>>>> So all the date is on osd.1>>>>>> But I have checked the balancer and seems active:>>> ceph balancer status>>> {>>> "active": true,>>> "last_optimize_duration": "0:00:00.000368",>>> "last_optimize_started": "Fri Mar 28 10:55:06 2025",>>> "mode": "upmap",>>> "no_optimization_needed": false,>>> "optimize_result": "Some objects (0.500000) are degraded; try >>> again later",>>> "plans": []>>> }>>>>>> But the output of the commnad 'ceph config dump|grep balancer' >>> gives me nothing.>>>>>> Best,>>> Mihai>>>>>> On 2025-03-27 23:06, Anthony D'Atri wrote:>>>> Look at `ceph osd df`. Is the balancer enabled?>>>>>>>>> On Mar 27, 2025, at 8:50 AM, Mihai Ciubancan >>>>> <mihai.ciubancan@eli-np.ro> wrote:>>>>>>>>>> Hello,>>>>>>>>>> My name is Mihai, and I have started using CEPH this mount for a >>>>> HPC cluster.>>>>> When was lunch in the production the available space shown was >>>>> 80TB now is 16TB and I didn't do anything, while I'm having 12 >>>>> OSD (SSD of 14TB):>>>>>>>>>> sudo ceph osd tree>>>>> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF>>>>> -1 167.64825 root default>>>>> -3 167.64825 host sto-core-hpc01>>>>> 0 ssd 13.97069 osd.0 up 1.00000 1.00000>>>>> 1 ssd 13.97069 osd.1 up 1.00000 1.00000>>>>> 2 ssd 13.97069 osd.2 up 1.00000 1.00000>>>>> 3 ssd 13.97069 osd.3 up 1.00000 1.00000>>>>> 4 ssd 13.97069 osd.4 up 1.00000 1.00000>>>>> 5 ssd 13.97069 osd.5 up 1.00000 1.00000>>>>> 6 ssd 13.97069 osd.6 up 1.00000 1.00000>>>>> 7 ssd 13.97069 osd.7 up 1.00000 1.00000>>>>> 8 ssd 13.97069 osd.8 up 1.00000 1.00000>>>>> 9 ssd 13.97069 osd.9 up 1.00000 1.00000>>>>> 10 ssd 13.97069 osd.10 up 1.00000 1.00000>>>>> 11 ssd 13.97069 osd.11 up 1.00000 1.00000>>>>>>>>>> sudo ceph df detail>>>>> --- RAW STORAGE --->>>>> CLASS SIZE AVAIL USED RAW USED %RAW USED>>>>> ssd 168 TiB 156 TiB 12 TiB 12 TiB 7.12>>>>> TOTAL 168 TiB 156 TiB 12 TiB 12 TiB 7.12>>>>>>>>>> --- POOLS --->>>>> POOL ID PGS STORED (DATA) (OMAP) OBJECTS >>>>> USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA >>>>> BYTES DIRTY USED COMPR UNDER COMPR>>>>> .mgr 1 1 705 KiB 705 KiB 0 B 2 >>>>> 1.4 MiB 1.4 MiB 0 B 0 8.1 TiB N/A >>>>> N/A N/A 0 B 0 B>>>>> cephfs.cephfs.meta 2 16 270 MiB 270 MiB 0 B 85.96k >>>>> 270 MiB 270 MiB 0 B 0 16 TiB N/A >>>>> N/A N/A 0 B 0 B>>>>> cephfs.cephfs.data 3 129 12 TiB 12 TiB 0 B 3.73M >>>>> 12 TiB 12 TiB 0 B 42.49 16 TiB N/A >>>>> N/A N/A 0 B 0 B>>>>>>>>>> While on the client side I have this:>>>>>>>>>> $ df -h>>>>> 10.18.31.1:6789:/ 21T 13T 8.1T 61% /data>>>>>>>>>> I don't know where it's gone all the space that was at the beginning.>>>>> Someone has any hint?>>>>>>>>>> Best regards,>>>>> Mihai>>>>> _______________________________________________>>>>> ceph-users mailing list -- ceph-users@ceph.io>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io>>> _______________________________________________>>> ceph-users mailing list -- ceph-users@ceph.io>>> To unsubscribe send an email to ceph-users-leave@ceph.io>>>>>> _______________________________________________>> ceph-users mailing list -- ceph-users@ceph.io>> To unsubscribe send an email to ceph-users-leave@ceph.io_______________________________________________ceph-users mailing list -- ceph-users@ceph.ioTo unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, Thank you all for your useful advices! I choose ceph because I understood that is faster than NFS. But on the other hand as I don't think that I will extend the storage with new nodes soon probably I will go back to software RAID solution with NFS. So I will copy temporarily the data on another storage and reconfigured the machine I wish you all a nice week-end, Mihai On 2025-03-28 17:14, Peter Linder wrote:
To get everything up to a working state, you will need to set your failure domain to "osd" instead of "host" in the default rule, and as it has been said before, pool size should be 3 and min_size 2.
With that said, you will eventually need more hosts to get the most out of ceph.
Den 2025-03-28 kl. 16:05, skrev mihai.ciubancan:
The problem is that I have only one host, and the host doesn't have a RAID controller...I have set as Anthone suggested, but nothing changed.Mihai -------- Original message --------From: Eugen Block <eblock@nde.ag> Date: 3/28/25 3:47 PM (GMT+02:00) To: Mihai Ciubancan <mihai.ciubancan@eli-np.ro> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: space size issue You have the autoscaler enabled, but it stuck changing the pg_num. The default replicated_rule (which you are using) requires as many hosts as your pool size is, so in your case 2. (If you value your data, don't use replicated pools with size 2.)You could make it work with only one host (as Anthony suggested with osd_crush_chooseleaf_type 0), but you don't any real resiliency. I recommend to reconsider your setup.Zitat von Mihai Ciubancan <mihai.ciubancan@eli-np.ro>:> Hi Eugen,>> Thanks for your answer. Please find below the output of the command:>> ceph osd pool ls detail> pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash > rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 323 flags > hashpspool,nearfull stripe_width 0 pg_num_max 32 pg_num_min 1 > application mgr read_balance_score 12.50> pool 2 'cephfs.cephfs.meta' replicated size 2 min_size 1 crush_rule > 0 object_hash rjenkins pg_num 16 pgp_num 1 pgp_num_target 16 > autoscale_mode on last_change 295 lfor 0/0/54 flags > hashpspool,nearfull stripe_width 0 pg_autoscale_bias 4 pg_num_min 16
recovery_priority 5 application cephfs read_balance_score 12.50> pool 3 'cephfs.cephfs.data' replicated size 2 min_size 1 crush_rule > 0 object_hash rjenkins pg_num 129 pgp_num 1 pg_num_target 512 > pgp_num_target 512 autoscale_mode on last_change 326 lfor 0/0/54 > flags hashpspool,nearfull,bulk stripe_width 0 application cephfs > read_balance_score 12.50>>> How can I decrese the number of pg_num?>> Best,> Mihai>>> On 2025-03-28 13:19, Eugen Block wrote:>> Do you use size 1 for your data? You know that's bad, right? Please>> share 'ceph osd pool ls detail'.>> Also, it's recommended to use a power of 2 for pg_num, so you should>> decrease the pg_num for the pool cephfs.cephfs.data.>>>> Zitat von Mihai Ciubancan <mihai.ciubancan@eli-np.ro>:>>>>> Hi Anthony,>>> Thanks for the answer:>>>>>> The output of 'ceph osd df' is:>>>>>> ceph osd df tree>>> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP >>> META AVAIL %USE VAR PGS STATUS TYPE NAME>>> -1 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB >>> 27 GiB 156 TiB 7.13 1.00 - root default>>> -3 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB >>> 27 GiB 156 TiB 7.13 1.00 - host >>> sto-core-hpc01>>> 0 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.0>>> 1 ssd 13.97069 1.00000 14 TiB 12 TiB 12 TiB 6 KiB >>> 26 GiB 2.0 TiB 85.53 12.00 129 up osd.1>>> 2 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.2>>> 3 ssd 13.97069 1.00000 14 TiB 1.7 GiB 258 MiB 100 MiB >>> 1.3 GiB 14 TiB 0.01 0.00 16 up osd.3>>> 4 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.4>>> 5 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.5>>> 6 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.6>>> 7 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.7>>> 8 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB >>> 63 MiB 14 TiB 0 0 1 up osd.8>>> 9 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.9>>> 10 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.10>>> 11 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB >>> 63 MiB 14 TiB 0 0 1 up osd.11>>> TOTAL 168 TiB 12 TiB 12 TiB 100 MiB >>> 27 GiB 156 TiB 7.13>>>>>> So all the date is on osd.1>>>>>> But I have checked the balancer and seems active:>>> ceph balancer status>>> {>>> "active": true,>>> "last_optimize_duration": "0:00:00.000368",>>> "last_optimize_started": "Fri Mar 28 10:55:06 2025",>>> "mode": "upmap",>>> "no_optimization_needed": false,>>> "optimize_result": "Some objects (0.500000) are degraded; try >>> again later",>>> "plans": []>>> }>>>>>> But the output of the commnad 'ceph config dump|grep balancer' >>> gives me nothing.>>>>>> Best,>>> Mihai>>>>>> On 2025-03-27 23:06, Anthony D'Atri wrote:>>>> Look at `ceph osd df`. Is the balancer enabled?>>>>>>>>> On Mar 27, 2025, at 8:50 AM, Mihai Ciubancan >>>>> <mihai.ciubancan@eli-np.ro> wrote:>>>>>>>>>> Hello,>>>>>>>>>> My name is Mihai, and I have started using CEPH this mount for a >>>>> HPC cluster.>>>>> When was lunch in the production the available space shown was >>>>> 80TB now is 16TB and I didn't do anything, while I'm having 12 >>>>> OSD (SSD of 14TB):>>>>>>>>>> sudo ceph osd tree>>>>> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF>>>>> -1 167.64825 root default>>>>> -3 167.64825 host sto-core-hpc01>>>>> 0 ssd 13.97069 osd.0 up 1.00000 1.00000>>>>> 1 ssd 13.97069 osd.1 up 1.00000 1.00000>>>>> 2 ssd 13.97069 osd.2 up 1.00000 1.00000>>>>> 3 ssd 13.97069 osd.3 up 1.00000 1.00000>>>>> 4 ssd 13.97069 osd.4 up 1.00000 1.00000>>>>> 5 ssd 13.97069 osd.5 up 1.00000 1.00000>>>>> 6 ssd 13.97069 osd.6 up 1.00000 1.00000>>>>> 7 ssd 13.97069 osd.7 up 1.00000 1.00000>>>>> 8 ssd 13.97069 osd.8 up 1.00000 1.00000>>>>> 9 ssd 13.97069 osd.9 up 1.00000 1.00000>>>>> 10 ssd 13.97069 osd.10 up 1.00000 1.00000>>>>> 11 ssd 13.97069 osd.11 up 1.00000 1.00000>>>>>>>>>> sudo ceph df detail>>>>> --- RAW STORAGE --->>>>> CLASS SIZE AVAIL USED RAW USED %RAW USED>>>>> ssd 168 TiB 156 TiB 12 TiB 12 TiB 7.12>>>>> TOTAL 168 TiB 156 TiB 12 TiB 12 TiB 7.12>>>>>>>>>> --- POOLS --->>>>> POOL ID PGS STORED (DATA) (OMAP) OBJECTS >>>>> USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA >>>>> BYTES DIRTY USED COMPR UNDER COMPR>>>>> .mgr 1 1 705 KiB 705 KiB 0 B 2 >>>>> 1.4 MiB 1.4 MiB 0 B 0 8.1 TiB N/A >>>>> N/A N/A 0 B 0 B>>>>> cephfs.cephfs.meta 2 16 270 MiB 270 MiB 0 B 85.96k >>>>> 270 MiB 270 MiB 0 B 0 16 TiB N/A >>>>> N/A N/A 0 B 0 B>>>>> cephfs.cephfs.data 3 129 12 TiB 12 TiB 0 B 3.73M >>>>> 12 TiB 12 TiB 0 B 42.49 16 TiB N/A >>>>> N/A N/A 0 B 0 B>>>>>>>>>> While on the client side I have this:>>>>>>>>>> $ df -h>>>>> 10.18.31.1:6789:/ 21T 13T 8.1T 61% /data>>>>>>>>>> I don't know where it's gone all the space that was at the beginning.>>>>> Someone has any hint?>>>>>>>>>> Best regards,>>>>> Mihai>>>>> _______________________________________________>>>>> ceph-users mailing list -- ceph-users@ceph.io>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io>>> _______________________________________________>>> ceph-users mailing list -- ceph-users@ceph.io>>> To unsubscribe send an email to ceph-users-leave@ceph.io>>>>>> _______________________________________________>> ceph-users mailing list -- ceph-users@ceph.io>> To unsubscribe send an email to ceph-users-leave@ceph.io_______________________________________________ceph-users mailing list -- ceph-users@ceph.ioTo unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We're glad to have been of help. There is no One Size Fits All solution. For you, it seems that speed is more important than high availability. For me, it's HA+redundancy. Ceph has 3 ways to deliver data to remote clients: 1. As a direct ceph mount on the client. From experience, this is a pain when the clients hibernate. 2. As an internal ganesha NFS server running under ceph 3. As an independent ganesha NFS server using ceph as a backend. option 1 would in theory provide the fastest access, since the client directly talks to OSDs, but I've heard reports that NFS handles buffering better. It definitely handles client disconnects better. Also, option 1 means that you don't have to supply a fixed server IP, but NFS can employ keepalive to failover. Those are just the options I know of within ceph, and I don't have firm stats even on those, much less non-ceph solutions. At least you've learned a few things about ceph, even it it doesn't presently align with your needs, so there's that. Best Regards, Tim On 3/28/25 14:20, Mihai Ciubancan wrote:
Hello,
Thank you all for your useful advices!
I choose ceph because I understood that is faster than NFS. But on the other hand as I don't think that I will extend the storage with new nodes soon probably I will go back to software RAID solution with NFS. So I will copy temporarily the data on another storage and reconfigured the machine
I wish you all a nice week-end, Mihai
To get everything up to a working state, you will need to set your failure domain to "osd" instead of "host" in the default rule, and as it has been said before, pool size should be 3 and min_size 2.
With that said, you will eventually need more hosts to get the most out of ceph.
Den 2025-03-28 kl. 16:05, skrev mihai.ciubancan:
The problem is that I have only one host, and the host doesn't have a RAID controller...I have set as Anthone suggested, but nothing changed.Mihai -------- Original message --------From: Eugen Block <eblock@nde.ag> Date: 3/28/25 3:47 PM (GMT+02:00) To: Mihai Ciubancan <mihai.ciubancan@eli-np.ro> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: space size issue You have the autoscaler enabled, but it stuck changing the pg_num. The default replicated_rule (which you are using) requires as many hosts as your pool size is, so in your case 2. (If you value your data, don't use replicated pools with size 2.)You could make it work with only one host (as Anthony suggested with osd_crush_chooseleaf_type 0), but you don't any real resiliency. I recommend to reconsider your setup.Zitat von Mihai Ciubancan <mihai.ciubancan@eli-np.ro>:> Hi Eugen,>> Thanks for your answer. Please find below the output of the command:>> ceph osd pool ls detail> pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash > rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 323 flags > hashpspool,nearfull stripe_width 0 pg_num_max 32 pg_num_min 1 > application mgr read_balance_score 12.50> pool 2 'cephfs.cephfs.meta' replicated size 2 min_size 1 crush_rule > 0 object_hash rjenkins pg_num 16 pgp_num 1 pgp_num_target 16 > autoscale_mode on last_change 295 lfor 0/0/54 flags > hashpspool,nearfull stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 > recovery_priority 5 application cephfs read_balance_score 12.50> pool 3 'cephfs.cephfs.data' replicated size 2 min_size 1 crush_rule > 0 object_hash rjenkins pg_num 129 pgp_num 1 pg_num_target 512 > pgp_num_target 512 autoscale_mode on last_change 326 lfor 0/0/54 > flags hashpspool,nearfull,bulk stripe_width 0 application cephfs > read_balance_score 12.50>>> How can I decrese the number of pg_num?>> Best,> Mihai>>> On 2025-03-28 13:19, Eugen Block wrote:>> Do you use size 1 for your data? You know that's bad, right? Please>> share 'ceph osd pool ls detail'.>> Also, it's recommended to use a power of 2 for pg_num, so you should>> decrease the pg_num for the pool cephfs.cephfs.data.>>>> Zitat von Mihai Ciubancan <mihai.ciubancan@eli-np.ro>:>>>>> Hi Anthony,>>> Thanks for the answer:>>>>>> The output of 'ceph osd df' is:>>>>>> ceph osd df tree>>> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP >>> META AVAIL %USE VAR PGS STATUS TYPE NAME>>> -1 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB >>> 27 GiB 156 TiB 7.13 1.00 - root default>>> -3 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB >>> 27 GiB 156 TiB 7.13 1.00 - host >>> sto-core-hpc01>>> 0 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.0>>> 1 ssd 13.97069 1.00000 14 TiB 12 TiB 12 TiB 6 KiB >>> 26 GiB 2.0 TiB 85.53 12.00 129 up osd.1>>> 2 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.2>>> 3 ssd 13.97069 1.00000 14 TiB 1.7 GiB 258 MiB 100 MiB >>> 1.3 GiB 14 TiB 0.01 0.00 16 up osd.3>>> 4 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.4>>> 5 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.5>>> 6 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB
28 MiB 14 TiB 0 0 0 up osd.6>>> 7 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.7>>> 8 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB >>> 63 MiB 14 TiB 0 0 1 up osd.8>>> 9 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.9>>> 10 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB >>> 28 MiB 14 TiB 0 0 0 up osd.10>>> 11 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB >>> 63 MiB 14 TiB 0 0 1 up osd.11>>> TOTAL 168 TiB 12 TiB 12 TiB 100 MiB >>> 27 GiB 156 TiB 7.13>>>>>> So all the date is on osd.1>>>>>> But I have checked the balancer and seems active:>>> ceph balancer status>>> {>>> "active": true,>>> "last_optimize_duration": "0:00:00.000368",>>> "last_optimize_started": "Fri Mar 28 10:55:06 2025",>>> "mode": "upmap",>>> "no_optimization_needed": false,>>> "optimize_result": "Some objects (0.500000) are degraded; try >>> again later",>>> "plans": []>>> }>>>>>> But the output of the commnad 'ceph config dump|grep balancer' >>> gives me nothing.>>>>>> Best,>>> Mihai>>>>>> On 2025-03-27 23:06, Anthony D'Atri wrote:>>>> Look at `ceph osd df`. Is the balancer enabled?>>>>>>>>> On Mar 27, 2025, at 8:50 AM, Mihai Ciubancan >>>>> <mihai.ciubancan@eli-np.ro> wrote:>>>>>>>>>> Hello,>>>>>>>>>> My name is Mihai, and I have started using CEPH this mount for a >>>>> HPC cluster.>>>>> When was lunch in
N/A N/A 0 B 0 B>>>>>>>>>> While on the client side I have this:>>>>>>>>>> $ df -h>>>>> 10.18.31.1:6789:/ 21T 13T 8.1T 61% /data>>>>>>>>>> I don't know where it's gone all the space that was at the beginning.>>>>> Someone has any hint?>>>>>>>>>> Best regards,>>>>> Mihai>>>>> _______________________________________________>>>>> ceph-users mailing list -- ceph-users@ceph.io>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io>>> _______________________________________________>>> ceph-users mailing
On 2025-03-28 17:14, Peter Linder wrote: the production the available space shown was >>>>> 80TB now is 16TB and I didn't do anything, while I'm having 12 >>>>> OSD (SSD of 14TB):>>>>>>>>>> sudo ceph osd tree>>>>> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF>>>>> -1 167.64825 root default>>>>> -3 167.64825 host sto-core-hpc01>>>>> 0 ssd 13.97069 osd.0 up 1.00000 1.00000>>>>> 1 ssd 13.97069 osd.1 up 1.00000 1.00000>>>>> 2 ssd 13.97069 osd.2 up 1.00000 1.00000>>>>> 3 ssd 13.97069 osd.3 up 1.00000 1.00000>>>>> 4 ssd 13.97069 osd.4 up 1.00000 1.00000>>>>> 5 ssd 13.97069 osd.5 up 1.00000 1.00000>>>>> 6 ssd 13.97069 osd.6 up 1.00000 1.00000>>>>> 7 ssd 13.97069 osd.7 up 1.00000 1.00000>>>>> 8 ssd 13.97069 osd.8 up 1.00000 1.00000>>>>> 9 ssd 13.97069 osd.9 up 1.00000 1.00000>>>>> 10 ssd 13.97069 osd.10 up 1.00000 1.00000>>>>> 11 ssd 13.97069 osd.11 up 1.00000 1.00000>>>>>>>>>> sudo ceph df detail>>>>> --- RAW STORAGE --->>>>> CLASS SIZE AVAIL USED RAW USED %RAW USED>>>>> ssd 168 TiB 156 TiB 12 TiB 12 TiB 7.12>>>>> TOTAL 168 TiB 156 TiB 12 TiB 12 TiB 7.12>>>>>>>>>> --- POOLS --->>>>> POOL ID PGS STORED (DATA) (OMAP) OBJECTS >>>>> USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA >>>>> BYTES DIRTY USED COMPR UNDER COMPR>>>>> .mgr 1 1 705 KiB 705 KiB 0 B 2 >>>>> 1.4 MiB 1.4 MiB 0 B 0 8.1 TiB N/A >>>>> N/A N/A 0 B 0 B>>>>> cephfs.cephfs.meta 2 16 270 MiB 270 MiB 0 B 85.96k >>>>> 270 MiB 270 MiB 0 B 0 16 TiB N/A >>>>> N/A N/A 0 B 0 B>>>>> cephfs.cephfs.data 3 129 12 TiB 12 TiB 0 B 3.73M >>>>> 12 TiB 12 TiB 0 B 42.49 16 TiB N/A list -- ceph-users@ceph.io>>> To unsubscribe send an email to ceph-users-leave@ceph.io>>>>>> _______________________________________________>> ceph-users mailing list -- ceph-users@ceph.io>> To unsubscribe send an email to ceph-users-leave@ceph.io_______________________________________________ceph-users mailing list -- ceph-users@ceph.ioTo unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Mar 28, 2025, at 4:38 PM, Tim Holloway <timh@mousetech.com> wrote:
We're glad to have been of help.
There is no One Size Fits All solution. For you, it seems that speed is more important than high availability. For me, it's HA+redundancy.
Ceph has 3 ways to deliver data to remote clients:
1. As a direct ceph mount on the client. From experience, this is a pain when the clients hibernate.
2. As an internal ganesha NFS server running under ceph
3. As an independent ganesha NFS server using ceph as a backend.
Some deployments also do KRBD mounts onto a VM or BM system that re-exports them via KNFS CephFS and RGW (with caveats) can also be exported with SMB. RBD mounts, either through KRBD or libvirt are popular as well.
Good point, although if I read that properly, all of those present the data as a raw block device and therefore have to have a client-side filesystem mounted on them. Probably more sophisticated than what our intrepid adventurer was looking for RBD as a base image for a VM is great for cloud server VMs, though! On 3/28/25 16:53, Anthony D'Atri wrote:
On Mar 28, 2025, at 4:38 PM, Tim Holloway <timh@mousetech.com> wrote:
We're glad to have been of help.
There is no One Size Fits All solution. For you, it seems that speed is more important than high availability. For me, it's HA+redundancy.
Ceph has 3 ways to deliver data to remote clients:
1. As a direct ceph mount on the client. From experience, this is a pain when the clients hibernate.
2. As an internal ganesha NFS server running under ceph
3. As an independent ganesha NFS server using ceph as a backend. Some deployments also do KRBD mounts onto a VM or BM system that re-exports them via KNFS
CephFS and RGW (with caveats) can also be exported with SMB.
RBD mounts, either through KRBD or libvirt are popular as well.
I was noting that there are other access modalities. CephFS and RGW obviate a client-side filesystem. I would think ZFS suits the OP, allowing sophisticated data protection, compression, and straightforward NFS export.
On Mar 28, 2025, at 7:48 PM, Tim Holloway <timh@mousetech.com> wrote:
Good point, although if I read that properly, all of those present the data as a raw block device and therefore have to have a client-side filesystem mounted on them. Probably more sophisticated than what our intrepid adventurer was looking for
RBD as a base image for a VM is great for cloud server VMs, though!
On 3/28/25 16:53, Anthony D'Atri wrote:
On Mar 28, 2025, at 4:38 PM, Tim Holloway <timh@mousetech.com> wrote:
We're glad to have been of help.
There is no One Size Fits All solution. For you, it seems that speed is more important than high availability. For me, it's HA+redundancy.
Ceph has 3 ways to deliver data to remote clients:
1. As a direct ceph mount on the client. From experience, this is a pain when the clients hibernate.
2. As an internal ganesha NFS server running under ceph
3. As an independent ganesha NFS server using ceph as a backend. Some deployments also do KRBD mounts onto a VM or BM system that re-exports them via KNFS
CephFS and RGW (with caveats) can also be exported with SMB.
RBD mounts, either through KRBD or libvirt are popular as well.
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yikes, something is off here. Is your entire cluster on one host? If so, beware that this is a very risky proposition. ceph config set global osd_crush_chooseleaf_type 0 https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-pg/#on... I suspect that will at least distribute your data better. If not, please send `ceph osd crush dump`
On Mar 28, 2025, at 6:59 AM, Mihai Ciubancan <mihai.ciubancan@eli-np.ro> wrote:
Hi Anthony, Thanks for the answer:
The output of 'ceph osd df' is:
ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB 27 GiB 156 TiB 7.13 1.00 - root default -3 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB 27 GiB 156 TiB 7.13 1.00 - host sto-core-hpc01 0 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.0 1 ssd 13.97069 1.00000 14 TiB 12 TiB 12 TiB 6 KiB 26 GiB 2.0 TiB 85.53 12.00 129 up osd.1 2 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.2 3 ssd 13.97069 1.00000 14 TiB 1.7 GiB 258 MiB 100 MiB 1.3 GiB 14 TiB 0.01 0.00 16 up osd.3 4 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.4 5 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.5 6 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.6 7 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.7 8 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB 63 MiB 14 TiB 0 0 1 up osd.8 9 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.9 10 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.10 11 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB 63 MiB 14 TiB 0 0 1 up osd.11 TOTAL 168 TiB 12 TiB 12 TiB 100 MiB 27 GiB 156 TiB 7.13
So all the date is on osd.1
But I have checked the balancer and seems active: ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.000368", "last_optimize_started": "Fri Mar 28 10:55:06 2025", "mode": "upmap", "no_optimization_needed": false, "optimize_result": "Some objects (0.500000) are degraded; try again later", "plans": [] }
But the output of the commnad 'ceph config dump|grep balancer' gives me nothing.
Best, Mihai
On 2025-03-27 23:06, Anthony D'Atri wrote:
Look at `ceph osd df`. Is the balancer enabled?
On Mar 27, 2025, at 8:50 AM, Mihai Ciubancan <mihai.ciubancan@eli-np.ro> wrote: Hello, My name is Mihai, and I have started using CEPH this mount for a HPC cluster. When was lunch in the production the available space shown was 80TB now is 16TB and I didn't do anything, while I'm having 12 OSD (SSD of 14TB): sudo ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 167.64825 root default -3 167.64825 host sto-core-hpc01 0 ssd 13.97069 osd.0 up 1.00000 1.00000 1 ssd 13.97069 osd.1 up 1.00000 1.00000 2 ssd 13.97069 osd.2 up 1.00000 1.00000 3 ssd 13.97069 osd.3 up 1.00000 1.00000 4 ssd 13.97069 osd.4 up 1.00000 1.00000 5 ssd 13.97069 osd.5 up 1.00000 1.00000 6 ssd 13.97069 osd.6 up 1.00000 1.00000 7 ssd 13.97069 osd.7 up 1.00000 1.00000 8 ssd 13.97069 osd.8 up 1.00000 1.00000 9 ssd 13.97069 osd.9 up 1.00000 1.00000 10 ssd 13.97069 osd.10 up 1.00000 1.00000 11 ssd 13.97069 osd.11 up 1.00000 1.00000 sudo ceph df detail --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 168 TiB 156 TiB 12 TiB 12 TiB 7.12 TOTAL 168 TiB 156 TiB 12 TiB 12 TiB 7.12 --- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR .mgr 1 1 705 KiB 705 KiB 0 B 2 1.4 MiB 1.4 MiB 0 B 0 8.1 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.meta 2 16 270 MiB 270 MiB 0 B 85.96k 270 MiB 270 MiB 0 B 0 16 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.data 3 129 12 TiB 12 TiB 0 B 3.73M 12 TiB 12 TiB 0 B 42.49 16 TiB N/A N/A N/A 0 B 0 B While on the client side I have this: $ df -h 10.18.31.1:6789:/ 21T 13T 8.1T 61% /data I don't know where it's gone all the space that was at the beginning. Someone has any hint? Best regards, Mihai _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I have run the command sent by you(thanks for the link, I understood why is necessary to run it), and I create a file of 21GB and has been wroted on the same osd(while the space of the partition is the same, much less that it should be). PThe output of the crush dump is: ceph osd crush dump { "devices": [ { "id": 0, "name": "osd.0", "class": "ssd" }, { "id": 1, "name": "osd.1", "class": "ssd" }, { "id": 2, "name": "osd.2", "class": "ssd" }, { "id": 3, "name": "osd.3", "class": "ssd" }, { "id": 4, "name": "osd.4", "class": "ssd" }, { "id": 5, "name": "osd.5", "class": "ssd" }, { "id": 6, "name": "osd.6", "class": "ssd" }, { "id": 7, "name": "osd.7", "class": "ssd" }, { "id": 8, "name": "osd.8", "class": "ssd" }, { "id": 9, "name": "osd.9", "class": "ssd" }, { "id": 10, "name": "osd.10", "class": "ssd" }, { "id": 11, "name": "osd.11", "class": "ssd" } ], "types": [ { "type_id": 0, "name": "osd" }, { "type_id": 1, "name": "host" }, { "type_id": 2, "name": "chassis" }, { "type_id": 3, "name": "rack" }, { "type_id": 4, "name": "row" }, { "type_id": 5, "name": "pdu" }, { "type_id": 6, "name": "pod" }, { "type_id": 7, "name": "room" }, { "type_id": 8, "name": "datacenter" }, { "type_id": 9, "name": "zone" }, { "type_id": 10, "name": "region" }, { "type_id": 11, "name": "root" } ], "buckets": [ { "id": -1, "name": "default", "type_id": 11, "type_name": "root", "weight": 10986996, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -3, "weight": 10986996, "pos": 0 } ] }, { "id": -2, "name": "default~ssd", "type_id": 11, "type_name": "root", "weight": 10986996, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -4, "weight": 10986996, "pos": 0 } ] }, { "id": -3, "name": "sto-core-hpc01", "type_id": 1, "type_name": "host", "weight": 10986996, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 0, "weight": 915583, "pos": 0 }, { "id": 1, "weight": 915583, "pos": 1 }, { "id": 10, "weight": 915583, "pos": 2 }, { "id": 11, "weight": 915583, "pos": 3 }, { "id": 2, "weight": 915583, "pos": 4 }, { "id": 3, "weight": 915583, "pos": 5 }, { "id": 4, "weight": 915583, "pos": 6 }, { "id": 5, "weight": 915583, "pos": 7 }, { "id": 6, "weight": 915583, "pos": 8 }, { "id": 7, "weight": 915583, "pos": 9 }, { "id": 8, "weight": 915583, "pos": 10 }, { "id": 9, "weight": 915583, "pos": 11 } ] }, { "id": -4, "name": "sto-core-hpc01~ssd", "type_id": 1, "type_name": "host", "weight": 10986996, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 0, "weight": 915583, "pos": 0 }, { "id": 1, "weight": 915583, "pos": 1 }, { "id": 10, "weight": 915583, "pos": 2 }, { "id": 11, "weight": 915583, "pos": 3 }, { "id": 2, "weight": 915583, "pos": 4 }, { "id": 3, "weight": 915583, "pos": 5 }, { "id": 4, "weight": 915583, "pos": 6 }, { "id": 5, "weight": 915583, "pos": 7 }, { "id": 6, "weight": 915583, "pos": 8 }, { "id": 7, "weight": 915583, "pos": 9 }, { "id": 8, "weight": 915583, "pos": 10 }, { "id": 9, "weight": 915583, "pos": 11 } ] } ], "rules": [ { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ], "tunables": { "choose_local_tries": 0, "choose_local_fallback_tries": 0, "choose_total_tries": 50, "chooseleaf_descend_once": 1, "chooseleaf_vary_r": 1, "chooseleaf_stable": 1, "msr_descents": 100, "msr_collision_tries": 100, "straw_calc_version": 1, "allowed_bucket_algs": 54, "profile": "jewel", "optimal_tunables": 1, "legacy_tunables": 0, "minimum_required_version": "jewel", "require_feature_tunables": 1, "require_feature_tunables2": 1, "has_v2_rules": 0, "require_feature_tunables3": 1, "has_v3_rules": 0, "has_v4_buckets": 1, "require_feature_tunables5": 1, "has_v5_rules": 0, "has_msr_rules": 0 }, "choose_args": { "-1": [ { "bucket_id": -1, "weight_set": [ [ 167.64825439453125 ] ] }, { "bucket_id": -2, "weight_set": [ [ 167.64825439453125 ] ] }, { "bucket_id": -3, "weight_set": [ [ 13.970687866210938, 13.970687866210938, 13.970687866210938, 13.970687866210938, 13.970687866210938, 13.970687866210938, 13.970687866210938, 13.970687866210938, 13.970687866210938, 13.970687866210938, 13.970687866210938, 13.970687866210938 ] ] }, { "bucket_id": -4, "weight_set": [ [ 13.970687866210938, 13.970687866210938, 13.970687866210938, 13.970687866210938, 13.970687866210938, 13.970687866210938, 13.970687866210938, 13.970687866210938, 13.970687866210938, 13.970687866210938, 13.970687866210938, 13.970687866210938 ] ] } ] } } Best, Mihai On 2025-03-28 14:27, Anthony D'Atri wrote:
Yikes, something is off here.
Is your entire cluster on one host? If so, beware that this is a very risky proposition.
ceph config set global osd_crush_chooseleaf_type 0
https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-pg/#on...
I suspect that will at least distribute your data better.
If not, please send `ceph osd crush dump`
On Mar 28, 2025, at 6:59 AM, Mihai Ciubancan <mihai.ciubancan@eli-np.ro> wrote:
Hi Anthony, Thanks for the answer:
The output of 'ceph osd df' is:
ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB 27 GiB 156 TiB 7.13 1.00 - root default -3 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB 27 GiB 156 TiB 7.13 1.00 - host sto-core-hpc01 0 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.0 1 ssd 13.97069 1.00000 14 TiB 12 TiB 12 TiB 6 KiB 26 GiB 2.0 TiB 85.53 12.00 129 up osd.1 2 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.2 3 ssd 13.97069 1.00000 14 TiB 1.7 GiB 258 MiB 100 MiB 1.3 GiB 14 TiB 0.01 0.00 16 up osd.3 4 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.4 5 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.5 6 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.6 7 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.7 8 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB 63 MiB 14 TiB 0 0 1 up osd.8 9 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.9 10 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.10 11 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB 63 MiB 14 TiB 0 0 1 up osd.11 TOTAL 168 TiB 12 TiB 12 TiB 100 MiB 27 GiB 156 TiB 7.13
So all the date is on osd.1
But I have checked the balancer and seems active: ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.000368", "last_optimize_started": "Fri Mar 28 10:55:06 2025", "mode": "upmap", "no_optimization_needed": false, "optimize_result": "Some objects (0.500000) are degraded; try again later", "plans": [] }
But the output of the commnad 'ceph config dump|grep balancer' gives me nothing.
Best, Mihai
On 2025-03-27 23:06, Anthony D'Atri wrote: Look at `ceph osd df`. Is the balancer enabled? On Mar 27, 2025, at 8:50 AM, Mihai Ciubancan <mihai.ciubancan@eli-np.ro> wrote: Hello, My name is Mihai, and I have started using CEPH this mount for a HPC cluster. When was lunch in the production the available space shown was 80TB now is 16TB and I didn't do anything, while I'm having 12 OSD (SSD of 14TB): sudo ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 167.64825 root default -3 167.64825 host sto-core-hpc01 0 ssd 13.97069 osd.0 up 1.00000 1.00000 1 ssd 13.97069 osd.1 up 1.00000 1.00000 2 ssd 13.97069 osd.2 up 1.00000 1.00000 3 ssd 13.97069 osd.3 up 1.00000 1.00000 4 ssd 13.97069 osd.4 up 1.00000 1.00000 5 ssd 13.97069 osd.5 up 1.00000 1.00000 6 ssd 13.97069 osd.6 up 1.00000 1.00000 7 ssd 13.97069 osd.7 up 1.00000 1.00000 8 ssd 13.97069 osd.8 up 1.00000 1.00000 9 ssd 13.97069 osd.9 up 1.00000 1.00000 10 ssd 13.97069 osd.10 up 1.00000 1.00000 11 ssd 13.97069 osd.11 up 1.00000 1.00000 sudo ceph df detail --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 168 TiB 156 TiB 12 TiB 12 TiB 7.12 TOTAL 168 TiB 156 TiB 12 TiB 12 TiB 7.12 --- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR .mgr 1 1 705 KiB 705 KiB 0 B 2 1.4 MiB 1.4 MiB 0 B 0 8.1 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.meta 2 16 270 MiB 270 MiB 0 B 85.96k 270 MiB 270 MiB 0 B 0 16 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.data 3 129 12 TiB 12 TiB 0 B 3.73M 12 TiB 12 TiB 0 B 42.49 16 TiB N/A N/A N/A 0 B 0 B While on the client side I have this: $ df -h 10.18.31.1:6789:/ 21T 13T 8.1T 61% /data I don't know where it's gone all the space that was at the beginning. Someone has any hint? Best regards, Mihai _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'm going to chime in and vote with the majority. You do not sound like an ideal user for ceph. Ceph provides high ability by being able to lose both drives /and servers/ in a transparent manner. With only one server, you have a single point of failure, and that's not considered "high availability". Granted, I'm also a poor fit for ceph, but nonetheless I have 4 servers running at the moment. My experience over the years is that at least once a year, I'll have a server go down. Either because of too much dust in the CPU fan (no clean-room here!) or a dead power supply. Just replaced one, in fact. I'll also lose 1-2 disks per year. Ceph can handle either of these situations quite well. I'm running with 2 OSDs out right now, pending replacement. If I was to go with a single server solution, I'd either put a RAID in the server or use a NAS. They're a lot less maintenance than ceph and the net result would be about the same. Of course you could simply buy a couple of cheap boxes and add extra ceph nodes. Which is what I do. Since I don't require extreme data performance I can get by with refurbished systems for around $250 for a 32GB unit. Good luck! Tim On 3/28/25 09:30, Mihai Ciubancan wrote:
I have run the command sent by you(thanks for the link, I understood why is necessary to run it), and I create a file of 21GB and has been wroted on the same osd(while the space of the partition is the same, much less that it should be). PThe output of the crush dump is:
ceph osd crush dump
{
"devices": [
{
"id": 0,
"name": "osd.0",
"class": "ssd"
},
{
"id": 1,
"name": "osd.1",
"class": "ssd"
},
{
"id": 2,
"name": "osd.2",
"class": "ssd"
},
{
"id": 3,
"name": "osd.3",
"class": "ssd"
},
{
"id": 4,
"name": "osd.4",
"class": "ssd"
},
{
"id": 5,
"name": "osd.5",
"class": "ssd"
},
{
"id": 6,
"name": "osd.6",
"class": "ssd"
},
{
"id": 7,
"name": "osd.7",
"class": "ssd"
},
{
"id": 8,
"name": "osd.8",
"class": "ssd"
},
{
"id": 9,
"name": "osd.9",
"class": "ssd"
},
{
"id": 10,
"name": "osd.10",
"class": "ssd"
},
{
"id": 11,
"name": "osd.11",
"class": "ssd"
}
],
"types": [
{
"type_id": 0,
"name": "osd"
},
{
"type_id": 1,
"name": "host"
},
{
"type_id": 2,
"name": "chassis"
},
{
"type_id": 3,
"name": "rack"
},
{
"type_id": 4,
"name": "row"
},
{
"type_id": 5,
"name": "pdu"
},
{
"type_id": 6,
"name": "pod"
},
{
"type_id": 7,
"name": "room"
},
{
"type_id": 8,
"name": "datacenter"
},
{
"type_id": 9,
"name": "zone"
},
{
"type_id": 10,
"name": "region"
},
{
"type_id": 11,
"name": "root"
}
],
"buckets": [
{
"id": -1,
"name": "default",
"type_id": 11,
"type_name": "root",
"weight": 10986996,
"alg": "straw2",
"hash": "rjenkins1",
"items": [
{
"id": -3,
"weight": 10986996,
"pos": 0
}
]
},
{
"id": -2,
"name": "default~ssd",
"type_id": 11,
"type_name": "root",
"weight": 10986996,
"alg": "straw2",
"hash": "rjenkins1",
"items": [
{
"id": -4,
"weight": 10986996,
"pos": 0
}
]
},
{
"id": -3,
"name": "sto-core-hpc01",
"type_id": 1,
"type_name": "host",
"weight": 10986996,
"alg": "straw2",
"hash": "rjenkins1",
"items": [
{
"id": 0,
"weight": 915583,
"pos": 0
},
{
"id": 1,
"weight": 915583,
"pos": 1
},
{
"id": 10,
"weight": 915583,
"pos": 2
},
{
"id": 11,
"weight": 915583,
"pos": 3
},
{
"id": 2,
"weight": 915583,
"pos": 4
},
{
"id": 3,
"weight": 915583,
"pos": 5
},
{
"id": 4,
"weight": 915583,
"pos": 6
},
{
"id": 5,
"weight": 915583,
"pos": 7
},
{
"id": 6,
"weight": 915583,
"pos": 8
},
{
"id": 7,
"weight": 915583,
"pos": 9
},
{
"id": 8,
"weight": 915583,
"pos": 10
},
{
"id": 9,
"weight": 915583,
"pos": 11
}
]
},
{
"id": -4,
"name": "sto-core-hpc01~ssd",
"type_id": 1,
"type_name": "host",
"weight": 10986996,
"alg": "straw2",
"hash": "rjenkins1",
"items": [
{
"id": 0,
"weight": 915583,
"pos": 0
},
{
"id": 1,
"weight": 915583,
"pos": 1
},
{
"id": 10,
"weight": 915583,
"pos": 2
},
{
"id": 11,
"weight": 915583,
"pos": 3
},
{
"id": 2,
"weight": 915583,
"pos": 4
},
{
"id": 3,
"weight": 915583,
"pos": 5
},
{
"id": 4,
"weight": 915583,
"pos": 6
},
{
"id": 5,
"weight": 915583,
"pos": 7
},
{
"id": 6,
"weight": 915583,
"pos": 8
},
{
"id": 7,
"weight": 915583,
"pos": 9
},
{
"id": 8,
"weight": 915583,
"pos": 10
},
{
"id": 9,
"weight": 915583,
"pos": 11
}
]
}
],
"rules": [
{
"rule_id": 0,
"rule_name": "replicated_rule",
"type": 1,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
],
"tunables": {
"choose_local_tries": 0,
"choose_local_fallback_tries": 0,
"choose_total_tries": 50,
"chooseleaf_descend_once": 1,
"chooseleaf_vary_r": 1,
"chooseleaf_stable": 1,
"msr_descents": 100,
"msr_collision_tries": 100,
"straw_calc_version": 1,
"allowed_bucket_algs": 54,
"profile": "jewel",
"optimal_tunables": 1,
"legacy_tunables": 0,
"minimum_required_version": "jewel",
"require_feature_tunables": 1,
"require_feature_tunables2": 1,
"has_v2_rules": 0,
"require_feature_tunables3": 1,
"has_v3_rules": 0,
"has_v4_buckets": 1,
"require_feature_tunables5": 1,
"has_v5_rules": 0,
"has_msr_rules": 0
},
"choose_args": {
"-1": [
{
"bucket_id": -1,
"weight_set": [
[
167.64825439453125
]
]
},
{
"bucket_id": -2,
"weight_set": [
[
167.64825439453125
]
]
},
{
"bucket_id": -3,
"weight_set": [
[
13.970687866210938,
13.970687866210938,
13.970687866210938,
13.970687866210938,
13.970687866210938,
13.970687866210938,
13.970687866210938,
13.970687866210938,
13.970687866210938,
13.970687866210938,
13.970687866210938,
13.970687866210938
]
]
},
{
"bucket_id": -4,
"weight_set": [
[
13.970687866210938,
13.970687866210938,
13.970687866210938,
13.970687866210938,
13.970687866210938,
13.970687866210938,
13.970687866210938,
13.970687866210938,
13.970687866210938,
13.970687866210938,
13.970687866210938,
13.970687866210938
]
]
}
]
}
}
Best,
Mihai
On 2025-03-28 14:27, Anthony D'Atri wrote:
Yikes, something is off here.
Is your entire cluster on one host? If so, beware that this is a very risky proposition.
ceph config set global osd_crush_chooseleaf_type 0
https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-pg/#on...
I suspect that will at least distribute your data better.
If not, please send `ceph osd crush dump`
On Mar 28, 2025, at 6:59 AM, Mihai Ciubancan <mihai.ciubancan@eli-np.ro> wrote:
Hi Anthony, Thanks for the answer:
The output of 'ceph osd df' is:
ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB 27 GiB 156 TiB 7.13 1.00 - root default -3 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB 27 GiB 156 TiB 7.13 1.00 - host sto-core-hpc01 0 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.0 1 ssd 13.97069 1.00000 14 TiB 12 TiB 12 TiB 6 KiB 26 GiB 2.0 TiB 85.53 12.00 129 up osd.1 2 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.2 3 ssd 13.97069 1.00000 14 TiB 1.7 GiB 258 MiB 100 MiB 1.3 GiB 14 TiB 0.01 0.00 16 up osd.3 4 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.4 5 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.5 6 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.6 7 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.7 8 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB 63 MiB 14 TiB 0 0 1 up osd.8 9 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.9 10 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB 28 MiB 14 TiB 0 0 0 up osd.10 11 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB 63 MiB 14 TiB 0 0 1 up osd.11 TOTAL 168 TiB 12 TiB 12 TiB 100 MiB 27 GiB 156 TiB 7.13
So all the date is on osd.1
But I have checked the balancer and seems active: ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.000368", "last_optimize_started": "Fri Mar 28 10:55:06 2025", "mode": "upmap", "no_optimization_needed": false, "optimize_result": "Some objects (0.500000) are degraded; try again later", "plans": [] }
But the output of the commnad 'ceph config dump|grep balancer' gives me nothing.
Best, Mihai
On 2025-03-27 23:06, Anthony D'Atri wrote: Look at `ceph osd df`. Is the balancer enabled? On Mar 27, 2025, at 8:50 AM, Mihai Ciubancan <mihai.ciubancan@eli-np.ro> wrote: Hello, My name is Mihai, and I have started using CEPH this mount for a HPC cluster. When was lunch in the production the available space shown was 80TB now is 16TB and I didn't do anything, while I'm having 12 OSD (SSD of 14TB): sudo ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 167.64825 root default -3 167.64825 host sto-core-hpc01 0 ssd 13.97069 osd.0 up 1.00000 1.00000 1 ssd 13.97069 osd.1 up 1.00000 1.00000 2 ssd 13.97069 osd.2 up 1.00000 1.00000 3 ssd 13.97069 osd.3 up 1.00000 1.00000 4 ssd 13.97069 osd.4 up 1.00000 1.00000 5 ssd 13.97069 osd.5 up 1.00000 1.00000 6 ssd 13.97069 osd.6 up 1.00000 1.00000 7 ssd 13.97069 osd.7 up 1.00000 1.00000 8 ssd 13.97069 osd.8 up 1.00000 1.00000 9 ssd 13.97069 osd.9 up 1.00000 1.00000 10 ssd 13.97069 osd.10 up 1.00000 1.00000 11 ssd 13.97069 osd.11 up 1.00000 1.00000 sudo ceph df detail --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 168 TiB 156 TiB 12 TiB 12 TiB 7.12 TOTAL 168 TiB 156 TiB 12 TiB 12 TiB 7.12 --- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR .mgr 1 1 705 KiB 705 KiB 0 B 2 1.4 MiB 1.4 MiB 0 B 0 8.1 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.meta 2 16 270 MiB 270 MiB 0 B 85.96k 270 MiB 270 MiB 0 B 0 16 TiB N/A N/A N/A 0 B 0 B cephfs.cephfs.data 3 129 12 TiB 12 TiB 0 B 3.73M 12 TiB 12 TiB 0 B 42.49 16 TiB N/A N/A N/A 0 B 0 B While on the client side I have this: $ df -h 10.18.31.1:6789:/ 21T 13T 8.1T 61% /data I don't know where it's gone all the space that was at the beginning. Someone has any hint? Best regards, Mihai _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (6)
-
Anthony D'Atri
-
Eugen Block
-
Mihai Ciubancan
-
mihai.ciubancan
-
Peter Linder
-
Tim Holloway