Unable to execute radosgw command using cephx users on client side
Hello, In our Ceph cluster we encountered issues while attempting to execute "radosgw-admin" command on client side using cephx user having read only permission. Whenever we are executing "radosgw-admin user list" command it is throwing an error. "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)" We have performed below steps in our environment Case-1 : First we created cephx user with below privileges client.rgw.username key: <-------key-------> caps: [mgr] allow r caps: [mon] allow r caps: [osd] allow r tag rgw *=* on client side we copied keyring and ceph.conf file What we noticed on client machine all general command like "ceph -s", "ceph health detail" "ceph df" running fine, even "radosgw-admin zonegroup list --id=rgw.username," command returned the expected output, but when attempting commands like "radosgw-admin user list," "radosgw-admin bucket list," or "radosgw-admin user info," errors were encountered. Below are the outputs that is throwing root@control:~# radosgw-admin user list --id=rgw.username 2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed reading data (obj=default.rgw.log:bucket.sync-source-hints.), r=-1 2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed to update sources index for bucket=:[]) r=-1 2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed to initialize bucket sync policy handler: get_bucket_sync_hints() on bucket=-- returned r=-1 2024-01-03T17:34:06.498+0000 7f915ece1fc0 -1 ERROR: could not initialize zone policy handler for zone=default 2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed to start notify service ((1) Operation not permitted 2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed to init services (ret=(1) Operation not permitted) couldn't init storage provider Case- 2 : In this case we granted read permissions to the rgw data pool and index pool for the user, client.rgw.username key: <----key----> caps: [mgr] allow r caps: [mon] allow r caps: [osd] allow r pool=default.rgw.log Despite this, while general commands worked perfectly fine on the client side, but "radosgw-admin" commands still failed to execute. Here is the output root@control:~# radosgw-admin user list --id=rgw.username 2024-01-03T17:43:38.071+0000 7f8b5a8bffc0 0 failed reading realm info: ret -1 (1) Operation not permitted 2024-01-03T17:43:38.071+0000 7f8b5a8bffc0 0 ERROR: failed to start notify service ((1) Operation not permitted 2024-01-03T17:43:38.071+0000 7f8b5a8bffc0 0 ERROR: failed to init services (ret=(1) Operation not permitted) couldn't init storage provider Have I overlooked anything in the process? Any guidance or insight would be greatly appreciated. Thanks, Mohammad Saif Ceph Enthusiast In the first step, we created a CephX user named client.rgw.saif with read permissions for the manager (mgr), monitor (mon), and object storage daemon (osd) components, along with specific RGW capabilities. On the client side, we successfully copied the keyring and ceph.conf, and certain commands, such as radosgw-admin zonegroup list --id=rgw.username,
Hi - I have a drive that is starting to show errors, and was wondering what the best way to replace it is. I am on Ceph 18.2.1, and using cephadm/containers I have 3 hosts, and each host has 4 4Tb drives with a 2 tb NVME device splt amongst them for WAL/DB, and 10 GB Networking. Option 1: Stop the OSD, use dd to copy from old to new, remove old, reboot so LVM recognized new as the volume that old was. Option 2: LVM and mirror the old drive to the new, then remove the old, once the mirroring is complete. In this way, I don't have to remove and reprovision the OSD, and the OSD doesn't need to be down during any Option 3: Remove the OSD, let everything settle down, swap the drive, fight the orchestrator to get the OSD provisioned with the OSD and db partition on the proper partition of the NVME, then let everything sync up again. I am leaning towards Option 2, because it should have the least impact/overhead on the rest of the drives, but am open to the other options as well. Thanks, Rob
Hi, in such a setup I also prefer option 2, we've done this since lvm came into play with OSDs, just not with cephadm yet. But we have a similar configuration and one OSD starts to fail as well. I'm just waiting for the replacement drive to arrive. ;-) Regards, Eugen Zitat von "Robert W. Eckert" <rob@rob.eckert.name>:
Hi - I have a drive that is starting to show errors, and was wondering what the best way to replace it is.
I am on Ceph 18.2.1, and using cephadm/containers I have 3 hosts, and each host has 4 4Tb drives with a 2 tb NVME device splt amongst them for WAL/DB, and 10 GB Networking.
Option 1: Stop the OSD, use dd to copy from old to new, remove old, reboot so LVM recognized new as the volume that old was. Option 2: LVM and mirror the old drive to the new, then remove the old, once the mirroring is complete. In this way, I don't have to remove and reprovision the OSD, and the OSD doesn't need to be down during any Option 3: Remove the OSD, let everything settle down, swap the drive, fight the orchestrator to get the OSD provisioned with the OSD and db partition on the proper partition of the NVME, then let everything sync up again.
I am leaning towards Option 2, because it should have the least impact/overhead on the rest of the drives, but am open to the other options as well.
Thanks, Rob _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
To close this thread, and give people a reference point, this is what I did (and it worked) I wound up not using mirror, I just used vgextend/pvmove/vgreduce to move the data. 1. Verify the devices in question: # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 3.6T 0 disk └─ceph--128bd83e--4bb5--466d--8f1c--1f00d93838f2-osd--block--fe7602d3--8e32--46e3--8dd6--4261e1b2feb6 253:2 0 3.6T 0 lvm sdb 8:16 0 3.6T 0 disk └─ceph--fef757df--55fc--41cb--aaa8--6f7675871c1f-osd--block--358ec8f2--e51b--419c--88bd--f088823c26a9 253:5 0 3.6T 0 lvm sdc 8:32 0 3.6T 0 disk └─ceph--92735252--d160--446b--b757--a2b2bf8e7ba7-osd--block--35c13cd7--bdd9--4c3b--a46d--6caaf7fc400a 253:3 0 3.6T 0 lvm sdd 8:48 0 465.8G 0 disk ├─sdd1 8:49 0 600M 0 part /boot/efi ├─sdd2 8:50 0 1G 0 part /boot └─sdd3 8:51 0 464.2G 0 part ├─rhel-root 253:0 0 288.4G 0 lvm /var/lib/containers/storage/overlay │ / ├─rhel-swap 253:1 0 11.8G 0 lvm └─rhel-home 253:6 0 350G 0 lvm /home sde 8:64 0 3.6T 0 disk └─ceph--fd2da222--2dc7--4be0--89a6--27e485ab4700-osd--block--278cfa46--6c69--4d87--b0c3--27e0b97ce379 253:4 0 3.6T 0 lvm sdf 8:80 0 1.8T 0 disk ├─sdf1 8:81 0 512G 0 part ├─sdf2 8:82 0 512G 0 part ├─sdf3 8:83 0 512G 0 part └─sdf4 8:84 0 327G 0 part nvme0n1 259:0 0 931.5G 0 disk └─nvme0n1p1 259:1 0 931.5G 0 part └─rhel-root 253:0 0 288.4G 0 lvm /var/lib/containers/storage/overlay / SDC was the drive failing, so I knew the lvm to look at 2. create PV on new device pvcreate /dev/sdg 3. Extend the volume group onto the new drive # vgextend ceph-fd2da222-2dc7-4be0-89a6-27e485ab4700 /dev/sdg Volume group "ceph-fd2da222-2dc7-4be0-89a6-27e485ab4700" successfully extended4. move the data 4. Move the data ## this will take some time, go have a nice dinner, get some sleep, binge watch a tv show,... # pvmove /dev/sdc /dev/sdc: Moved: 0.01% /dev/sdc: Moved: 0.06% ....... /dev/sdc: Moved: 99.98% /dev/sdc: Moved: 100.00% 5. remove the dying drive # vgreduce ceph-fd2da222-2dc7-4be0-89a6-27e485ab4700 /dev/sdc Removed "/dev/sdc" from volume group "ceph-fd2da222-2dc7-4be0-89a6-27e485ab4700" 6. (Very important) verify the serial number of the drive to remove: # udevadm info --query=all --name=/dev/sdc | grep ID_SERIAL E: ID_SERIAL=ST4000DX001-1CE168_Z303TB0R E: ID_SERIAL_SHORT=Z303TB0R 7. put host into maintenance, then halt the system, and swap the physical drives. 8. Start the host, take it out of maintenance, and let any sync happen - watch ceph status until it is completely recovered - a few minutes. Doing it this way, I did not notice any issues with ceph during the move - I had things interacting with cephfs, and rbd the whole time without a glitch. There are a few pgs marked as backfilling - I may have missed a delay with the host maintenance. But its only about 20, which is much less than if I took the entire OSD out and rebuilt it. Thanks, Rob -----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Wednesday, January 3, 2024 2:37 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: Best way to replace Data drive of OSD Hi, in such a setup I also prefer option 2, we've done this since lvm came into play with OSDs, just not with cephadm yet. But we have a similar configuration and one OSD starts to fail as well. I'm just waiting for the replacement drive to arrive. ;-) Regards, Eugen Zitat von "Robert W. Eckert" <rob@rob.eckert.name>:
Hi - I have a drive that is starting to show errors, and was wondering what the best way to replace it is.
I am on Ceph 18.2.1, and using cephadm/containers I have 3 hosts, and each host has 4 4Tb drives with a 2 tb NVME device splt amongst them for WAL/DB, and 10 GB Networking.
Option 1: Stop the OSD, use dd to copy from old to new, remove old, reboot so LVM recognized new as the volume that old was. Option 2: LVM and mirror the old drive to the new, then remove the old, once the mirroring is complete. In this way, I don't have to remove and reprovision the OSD, and the OSD doesn't need to be down during any Option 3: Remove the OSD, let everything settle down, swap the drive, fight the orchestrator to get the OSD provisioned with the OSD and db partition on the proper partition of the NVME, then let everything sync up again.
I am leaning towards Option 2, because it should have the least impact/overhead on the rest of the drives, but am open to the other options as well.
Thanks, Rob _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi All, I just wanted to quick follow-up on my previous mail about "Unable to execute radosgw command using cephx users on client side". I wanted to highlight that I'm still facing the issue and urgently need your guidance to resolve it. I appreciate your attention to this matter. Thanks, Saif
Hi, I don't really have any solution, but it appears to require rwx permissions at least for the rgw tag: caps osd = "allow rwx tag rgw *=* This was the only way I got the radosgw-admin commands to work in my limited test attempts. Maybe someone else has more insights. My interpretation of these error messages ("failed to update source index") is that it actually requires to update something:
2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed reading data (obj=default.rgw.log:bucket.sync-source-hints.), r=-1 2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed to update sources index for bucket=:[]) r=-1
But as I said, I might misinterpret things, so I hope someone else can chime in here. Regards, Eugen Zitat von Alam Mohammad <samdto987@gmail.com>:
Hello,
In our Ceph cluster we encountered issues while attempting to execute "radosgw-admin" command on client side using cephx user having read only permission. Whenever we are executing "radosgw-admin user list" command it is throwing an error.
"ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)"
We have performed below steps in our environment Case-1 : First we created cephx user with below privileges
client.rgw.username key: <-------key-------> caps: [mgr] allow r caps: [mon] allow r caps: [osd] allow r tag rgw *=*
on client side we copied keyring and ceph.conf file What we noticed on client machine all general command like "ceph -s", "ceph health detail" "ceph df" running fine, even "radosgw-admin zonegroup list --id=rgw.username," command returned the expected output, but when attempting commands like "radosgw-admin user list," "radosgw-admin bucket list," or "radosgw-admin user info," errors were encountered. Below are the outputs that is throwing
root@control:~# radosgw-admin user list --id=rgw.username 2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed reading data (obj=default.rgw.log:bucket.sync-source-hints.), r=-1 2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed to update sources index for bucket=:[]) r=-1 2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed to initialize bucket sync policy handler: get_bucket_sync_hints() on bucket=-- returned r=-1 2024-01-03T17:34:06.498+0000 7f915ece1fc0 -1 ERROR: could not initialize zone policy handler for zone=default 2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed to start notify service ((1) Operation not permitted 2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed to init services (ret=(1) Operation not permitted) couldn't init storage provider
Case- 2 : In this case we granted read permissions to the rgw data pool and index pool for the user, client.rgw.username key: <----key----> caps: [mgr] allow r caps: [mon] allow r caps: [osd] allow r pool=default.rgw.log Despite this, while general commands worked perfectly fine on the client side, but "radosgw-admin" commands still failed to execute.
Here is the output root@control:~# radosgw-admin user list --id=rgw.username 2024-01-03T17:43:38.071+0000 7f8b5a8bffc0 0 failed reading realm info: ret -1 (1) Operation not permitted 2024-01-03T17:43:38.071+0000 7f8b5a8bffc0 0 ERROR: failed to start notify service ((1) Operation not permitted 2024-01-03T17:43:38.071+0000 7f8b5a8bffc0 0 ERROR: failed to init services (ret=(1) Operation not permitted) couldn't init storage provider
Have I overlooked anything in the process? Any guidance or insight would be greatly appreciated.
Thanks, Mohammad Saif Ceph Enthusiast
In the first step, we created a CephX user named client.rgw.saif with read permissions for the manager (mgr), monitor (mon), and object storage daemon (osd) components, along with specific RGW capabilities. On the client side, we successfully copied the keyring and ceph.conf, and certain commands, such as radosgw-admin zonegroup list --id=rgw.username, _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Alam Mohammad
-
Eugen Block
-
Robert W. Eckert