Can we please have 1 command, that can dump all config
Sounds like an opportunity to enter a tracker issue or a PR.
Done: https://tracker.ceph.com/issues/71783
What model are they that they’re that small? Are they enterprise-quality? OSDs that small can present difficulties. They are ~1 TiB enterprise SSDs: Samsung PM983, 960GB (MZQLB960HAJR-00007) Our cluster currently has no need for SSD storage (only metadata which is only 7 %USED). Most of the SSD's space is used for the DB/WAL for the HDD SSDs.
For a machine with 10x 16TiB HDD + 2x 960GiB SSD: * Each SSD carries DB/WAL for 5 HDDs, so 5 * 60 GiB each. * 170 GB SSD for SSD OSDs. * The rest of the SSD for the OS disk.
How large are those DB+WAL slices? Please share BlueFS stats:
`ceph health detail` example (osd.2 is one such 16 TiB HDD): osd.2 spilled over 5.2 GiB metadata from 'db' device (47 GiB used of 60 GiB) to slow device ceph daemon osd.2 bluefs stats 1 : device size 0xee5afe000 : using 0xbdc800000(47 GiB) 2 : device size 0xe8d7ed00000 : using 0xc81f80fb000(13 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 14 MiB 0 B 0 B 0 B 12 MiB 1 WAL 0 B 915 MiB 0 B 0 B 0 B 502 MiB 35 DB 0 B 9.7 GiB 0 B 0 B 0 B 6.4 GiB 104 SLOW 0 B 37 GiB 5.2 GiB 0 B 0 B 37 GiB 607 TOTAL 0 B 47 GiB 5.2 GiB 0 B 0 B 0 B 747 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 1.8 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 11 GiB 0 B 0 B 0 B 6.7 GiB SLOW 0 B 39 GiB 8.4 GiB 0 B 0 B 42 GiB TOTAL 0 B 50 GiB 8.4 GiB 0 B 0 B 0 B >> SIZE << 0 B 57 GiB 14 TiB
Oddly, not listed> I look forward to your PR. Done: https://github.com/ceph/ceph/pull/64074
I suspect that once backfill completes you’ll see a ratio > 33 Shouldn't that have been the case already before I added the new machines, given that the PG count didn't change?
Apologies, I meant mon_target_pg_per_osd = 250 That makes sense, thanks!
`upmap_max_deviation` Done: https://github.com/ceph/ceph/pull/64075