On Tue, Nov 3, 2020 at 12:07 PM Wido den Hollander <wido@42on.com> wrote:
On 03/11/2020 10:02, Dan van der Ster wrote:
Hi all,
We still have legacy caps on our nautilus rbd cluster. I just wanted to check if this is totally safe (and to post here ftr because I don't think this has ever been documented)
Here are the current caps:
[client.images] key = xxx caps mgr = "allow r" caps mon = "allow r, allow command \"osd blacklist\"" caps osd = "allow class-read object_prefix rbd_children, allow rwx pool=images"
[client.volumes] key = xxx caps mgr = "allow r" caps mon = "allow r, allow command \"osd blacklist\"" caps osd = "allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rx pool=images, allow rwx pool=cinder-critical"
Now that we upgraded to nautilus we would do:
# ceph auth caps client.images mon 'profile rbd' osd 'profile rbd pool=images' mgr 'profile rbd pool=images' # ceph auth caps client.volumes mon 'profile rbd' osd 'profile rbd pool=volumes, profile rbd-read-only pool=images, profile rbd pool=cinder-critical' mgr 'profile rbd pool=volumes, profile rbd pool=cinder-critical'
Does that look correct? Does this apply without impacting any client IOs ?
Yes, it looks correct, but what I usually do:
$ ceph auth get client.images -o client.images $ cp client.images client.images.org $ edit the client.images file $ diff -u client.images client.images.org $ ceph auth import -i client.images
This way I also have a way of reverting quickly if things do go wrong.
What I also did is import the key with a name (eg client.images2) and test if I could manually perform RBD operations with the 'rbd' cli tool.
Warning: IF you make a mistake (and I have seen this happen!) ceph will start returning 'Operation Not Permitted' to librados which they causes I/O errors inside librbd. Your VMs will go into Read-Only as filesystems break and probably need an fsck to get back.
So triple-check your work before doing this. But if done properly it can be done online.
Thanks Wido, and good idea with the testing steps beforehand. We created images2, volumes2, tested that they are all good, then updated the real users with `ceph auth caps`. All done! Cheers, Dan