On Tue, Nov 3, 2020 at 12:07 PM Wido den Hollander <wido(a)42on.com> wrote:
On 03/11/2020 10:02, Dan van der Ster wrote:
Hi all,
We still have legacy caps on our nautilus rbd cluster. I just wanted
to check if this is totally safe (and to post here ftr because I don't
think this has ever been documented)
Here are the current caps:
[client.images]
key = xxx
caps mgr = "allow r"
caps mon = "allow r, allow command \"osd blacklist\""
caps osd = "allow class-read object_prefix rbd_children, allow rwx
pool=images"
[client.volumes]
key = xxx
caps mgr = "allow r"
caps mon = "allow r, allow command \"osd blacklist\""
caps osd = "allow class-read object_prefix rbd_children, allow rwx
pool=volumes, allow rx pool=images, allow rwx pool=cinder-critical"
Now that we upgraded to nautilus we would do:
# ceph auth caps client.images mon 'profile rbd' osd 'profile rbd
pool=images' mgr 'profile rbd pool=images'
# ceph auth caps client.volumes mon 'profile rbd' osd 'profile rbd
pool=volumes, profile rbd-read-only pool=images, profile rbd
pool=cinder-critical' mgr 'profile rbd pool=volumes, profile rbd
pool=cinder-critical'
Does that look correct? Does this apply without impacting any client IOs ?
Yes, it looks correct, but what I usually do:
$ ceph auth get client.images -o client.images
$ cp client.images
client.images.org
$ edit the client.images file
$ diff -u client.images
client.images.org
$ ceph auth import -i client.images
This way I also have a way of reverting quickly if things do go wrong.
What I also did is import the key with a name (eg client.images2) and
test if I could manually perform RBD operations with the 'rbd' cli tool.
Warning: IF you make a mistake (and I have seen this happen!) ceph will
start returning 'Operation Not Permitted' to librados which they causes
I/O errors inside librbd. Your VMs will go into Read-Only as filesystems
break and probably need an fsck to get back.
So triple-check your work before doing this. But if done properly it can
be done online.
Thanks Wido, and good idea with the testing steps beforehand.
We created images2, volumes2, tested that they are all good, then
updated the real users with `ceph auth caps`.
All done!
Cheers, Dan