[ceph-users] Re: Unable to map RBDs after running pg-upmap-primary on the pool

7 Mar 2024

      On 07/03/2024 08:52, Torkil Svensgaard wrote:
...
Hi
I tried to do offline read optimization[1] this morning but I am now 
unable to map the RBDs in the pool.
I did this prior to running the pg-upmap-primary commands suggested by 
the optimizer, as suggested by the latest documentation[2]:
"
ceph osd set-require-min-compat-client reef
"
The command didn't complain and the documentation stated it should 
failed if any pre-reef clients were connected so I thought all was well 
and rad the pg_upmap_primary commands.
Can I simply remove the applied pg_upmap_primary settings somehow to my 
RBDs back while investigating?
Found the rm-pg-upmap-primary command. Removing all upmaps made it 
possible to map the RBD again.

Still not sure why this didn't work though.

"
[ceph: root@lazy /]# ceph osd dump|grep require_min_compat_client
require_min_compat_client reef
"

That's seems to have stuck but we most certainly have pacific clients so 
according to the documentation the "ceph osd 
set-require-min-compat-client reef" command should have failed.

Then there's this:

"
[ceph: root@lazy /]# ceph features
{
     "mon": [
         {
             "features": "0x3f01cfbffffdffff",
             "release": "luminous",
             "num": 5
         }
     ],
     "mds": [
         {
             "features": "0x3f01cfbffffdffff",
             "release": "luminous",
             "num": 3
         }
     ],
     "osd": [
         {
             "features": "0x3f01cfbffffdffff",
             "release": "luminous",
             "num": 452
         }
     ],
     "client": [
         {
             "features": "0x2f018fb87aa4aafe",
             "release": "luminous",
             "num": 13
         },
         {
             "features": "0x3f01cfb8ffecffff",
             "release": "luminous",
             "num": 19
         },
         {
             "features": "0x3f01cfb9fffdffff",
             "release": "luminous",
             "num": 17
         },
         {
             "features": "0x3f01cfbf7ffdffff",
             "release": "luminous",
             "num": 14
         },
         {
             "features": "0x3f01cfbffffdffff",
             "release": "luminous",
             "num": 17
         }
     ],
     "mgr": [
         {
             "features": "0x3f01cfbffffdffff",
             "release": "luminous",
             "num": 3
         }
     ]
}
"

This suggests that min compat client reef might be set but is not in effect?

And lastly it's one thing not being able to map pg-upmap-primary RBDs 
from a pre reef client but it didn't work with a reef client either.

Mvh.

Torkil
...
Mvh.
Torkil
[1] https://docs.ceph.com/en/reef/rados/operations/read-balancer/
[2] https://docs.ceph.com/en/latest/rados/operations/read-balancer/
-- 
Torkil Svensgaard
Sysadmin
MR-Forskningssektionen, afs. 714
DRCMR, Danish Research Centre for Magnetic Resonance
Hvidovre Hospital
Kettegård Allé 30
DK-2650 Hvidovre
Denmark
Tel: +45 386 22828
E-mail: torkil@drcmr.dk