On 1/30/23 15:15, Ana Aviles wrote:
Hi,
Josh already suggested, but I will one more time. We had similar behaviour upgrading from Nautilus to Pacific. In our case compacting the OSDs did the trick.
Thanks for chimming in! Unfortunately, in my case neither an online compaction (ceph tell osd.ID compact) or an offline repair (ceph-bluestore-tool repair --path /var/lib/ceph/osd/OSD_ID) does help. Compactions seem to compact some amount. I think that OSD log dumps information about the size of rocksdb. It went from this: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.9 3.9 0.0 1.0 0.0 62.2 64.81 61.59 89 0.728 0 0 L1 3/0 132.84 MB 0.5 7.0 3.9 3.1 5.0 2.0 0.0 1.3 63.8 46.1 112.11 108.52 23 4.874 56M 7276K L2 12/0 690.99 MB 0.8 6.5 1.8 4.7 5.6 0.9 0.1 3.2 21.4 18.5 310.78 307.14 28 11.099 165M 3077K L3 54/0 3.37 GB 0.1 0.9 0.3 0.6 0.5 -0.1 0.0 1.6 35.9 20.2 24.84 24.49 4 6.210 24M 15M Sum 69/0 4.17 GB 0.0 14.4 6.0 8.3 15.1 6.7 0.1 3.8 28.7 30.1 512.54 501.74 144 3.559 246M 26M Int 0/0 0.00 KB 0.0 0.8 0.3 0.5 0.6 0.1 0.0 14.1 27.5 20.7 31.13 30.73 4 7.783 18M 4086K To this: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 72.42 MB 0.5 0.0 0.0 0.0 0.1 0.1 0.0 1.0 0.0 63.2 1.14 0.84 2 0.572 0 0 L3 48/0 3.10 GB 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 50/0 3.17 GB 0.0 0.0 0.0 0.0 0.1 0.1 0.0 1.0 0.0 63.2 1.14 0.84 2 0.572 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Still, it feels "too big" compared to some other OSD in other similarily sized clusters, making me think that there's some kind of "garbage" making the trim to go crazy.
For us there was no performance impact running the compaction (ceph osd daemon osd.0 compact) although we run them in batches and not all at once on all OSDs just in case. Also, no need to restart OSDs for this operation.
Yes, compacting had no perceived impact on client performance, just some higher CPU usage for the OSD process. Does anyone knows by any chance the meaning of "num_pgmeta_omap" on ceph daemon osd.ID calc_objectstore_db_histogram output? As I mentioned, the OSDs in this cluster have very different values in that field but all other clusters have much similar values: osd.0: "num_pgmeta_omap": 17526766, osd.1: "num_pgmeta_omap": 2653379, osd.2: "num_pgmeta_omap": 12358703, osd.3: "num_pgmeta_omap": 6404975, osd.6: "num_pgmeta_omap": 19845318, osd.7: "num_pgmeta_omap": 6043083, osd.12: "num_pgmeta_omap": 18666776, osd.13: "num_pgmeta_omap": 615846, osd.14: "num_pgmeta_omap": 13190188, Thanks a lot! --