On 07/03/2024 08:52, Torkil Svensgaard wrote:

Hi

I tried to do offline read optimization[1] this morning but I am now unable to map the RBDs in the pool.

I did this prior to running the pg-upmap-primary commands suggested by the optimizer, as suggested by the latest documentation[2]:

" ceph osd set-require-min-compat-client reef "

The command didn't complain and the documentation stated it should failed if any pre-reef clients were connected so I thought all was well and rad the pg_upmap_primary commands.

Can I simply remove the applied pg_upmap_primary settings somehow to my RBDs back while investigating?

Found the rm-pg-upmap-primary command. Removing all upmaps made it possible to map the RBD again. Still not sure why this didn't work though. " [ceph: root@lazy /]# ceph osd dump|grep require_min_compat_client require_min_compat_client reef " That's seems to have stuck but we most certainly have pacific clients so according to the documentation the "ceph osd set-require-min-compat-client reef" command should have failed. Then there's this: " [ceph: root@lazy /]# ceph features { "mon": [ { "features": "0x3f01cfbffffdffff", "release": "luminous", "num": 5 } ], "mds": [ { "features": "0x3f01cfbffffdffff", "release": "luminous", "num": 3 } ], "osd": [ { "features": "0x3f01cfbffffdffff", "release": "luminous", "num": 452 } ], "client": [ { "features": "0x2f018fb87aa4aafe", "release": "luminous", "num": 13 }, { "features": "0x3f01cfb8ffecffff", "release": "luminous", "num": 19 }, { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 17 }, { "features": "0x3f01cfbf7ffdffff", "release": "luminous", "num": 14 }, { "features": "0x3f01cfbffffdffff", "release": "luminous", "num": 17 } ], "mgr": [ { "features": "0x3f01cfbffffdffff", "release": "luminous", "num": 3 } ] } " This suggests that min compat client reef might be set but is not in effect? And lastly it's one thing not being able to map pg-upmap-primary RBDs from a pre reef client but it didn't work with a reef client either. Mvh. Torkil

Mvh.

Torkil

[1] https://docs.ceph.com/en/reef/rados/operations/read-balancer/ [2] https://docs.ceph.com/en/latest/rados/operations/read-balancer/

-- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil@drcmr.dk