Re: FileStore OSD, journal direct symlinked, permission troubles.
Riprendo quanto scritto nel suo messaggio del 29/08/2019...
Another possibilty is to convert the MBR to GPT (sgdisk --mbrtogpt) and give the partition its UID (also sgdisk). Then it could be linked by its uuid. and, in another email: And I forgot that you can also re-create the journal by itself. I can't recall the command ATM though.
Ahem, i stated the jornal disk are also the OS disks, and i'm using old server, so i think that converting to GPT will lead to an unbootable node... But, the 'code' that identify (and change permission) for journal dev are PVE specific? or Ceph generic? I suppose the latter... Also, i've done: adduser ceph disk and partition devices are '660 root:disk': why still i get 'permission denied'?
Or if you are not in need of filestore OSDs, re-create them as bluestore ones. AFAICS, Ceph has laid more focus on bluestore and it might be better to do a conversion sooner than later. (my opinion)
Not for now; bluestore migration need a bit more time/study/knowledge... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)
On Thu, Aug 29, 2019 at 05:02:11PM +0200, Marco Gaiarin wrote:
Riprendo quanto scritto nel suo messaggio del 29/08/2019...
Another possibilty is to convert the MBR to GPT (sgdisk --mbrtogpt) and give the partition its UID (also sgdisk). Then it could be linked by its uuid. and, in another email: And I forgot that you can also re-create the journal by itself. I can't recall the command ATM though.
Ahem, i stated the jornal disk are also the OS disks, and i'm using old server, so i think that converting to GPT will lead to an unbootable node...
But, the 'code' that identify (and change permission) for journal dev are PVE specific? or Ceph generic? I suppose the latter... IIRC, in Jewel, Ceph went from running services as root to its own user 'ceph'. The permissions had to be changed once, during upgrade. And then udev takes care of it by checking the GUID [0].
:~# ls /lib/udev/rules.d/60-ceph-by-parttypeuuid.rules :~# ls /lib/udev/rules.d/95-ceph-osd.rules Udev takes care of the correct device node owner. Changing these to work with the msdos partition scheme will not be very practical. Alone that you have to adapt the rule with each Ceph update.
Also, i've done: adduser ceph disk and partition devices are '660 root:disk': why still i get 'permission denied'?
User and group is ceph on my PVE 5.x and later installs. The user & group should have been created during upgrade. And as said above udev should take care of it for the device node. As it is possible to re-create the journal for an OSD, you may consider to add an extra SSD to create the journals there. Then you can free up the OS disks of that burden and have a GPT partition table with the proper GUID. Less risky, then my approach below. I tested an conversion from MBR to GPT on the boot disk in a VM. To make it work, I had to run the following. ## Caution: If it fails the system will be not booting # converts to GPT sgdisk -g /dev/sdX # the conversion left 1MB space # create a new partition with type ef02 on the boot disk uses the 1MB gdisk /dev/sda # install grub, if the disk doesn't contain a bios boot partition # (ef02) it will not install grub-install /dev/sda After the conversion it should be no problem to got through the permission fix on our upgrade guide [0] to get the journals recognised by Ceph. Well, I hope this helps. -- Cheers, Alwin [0] https://pve.proxmox.com/wiki/Ceph_Hammer_to_Jewel#Set_permission
But, the 'code' that identify (and change permission) for journal dev are PVE specific? or Ceph generic? I suppose the latter...
OK, trying to identify how OSDs get initialized. If i understood well: 0) systemd unit for every OSD get created following a template: /lib/systemd/system/ceph-osd@.service 1) every unit call a 'prestart' script: ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i 2) The prestart script, run udev: udevadm settle --timeout=5 that simply force the processing of udev queue, only to be sure there's some 'unhandled' device in the queue. 3) udev (rules in /lib/udev/rules.d/95-ceph-osd.rules), looking for GPT ID_PART_ENTRY_TYPE do two things: a) ceph-disk --log-stdout -v trigger /dev/$name (that AFAIK trigger a disk mount, for filestore) b) chown ceph:ceph /dev/$name; chmod 660 /dev/$name So, seems to me that a decent method to solve/circumvent my trouble is to: i) write a 'static' udev rule that chown ceph:ceph the partition. Very dirty. ii) modify the systemd unit and add an ExecStartPost= script that chown the partition. Dirty but probably effective. iii) modify /usr/lib/ceph/ceph-osd-prestart.sh to add the condition, something like (untested): if [ -L "$journal" -a -e "$journal" ]; then dev_journal=`readlink -f $journal` owner=`stat -c %U $dev_journal` if [ $owner != 'ceph' ]; then echo "ceph-osd(${cluster:-ceph}-$id): journal probably manually symlinked, fixing permission." 1>&2 chown ceph: $dev_journal fi fi I'm not a ceph expert, but solution iii) seems decent for me, with a little overhead (a readlinkk and a stat for every osd start). But still i don't understood why, if i have: root@capitanmarvel:~# LANG=C id ceph uid=64045(ceph) gid=64045(ceph) groups=64045(ceph),6(disk) and: brw-rw---- 1 root disk 8, 6 ago 28 14:38 /dev/sda6 (so, journal partition group-owned by 'disk' and 'ceph' user in group 'disk'), still i have permission access. The ceph-osd process reset group ownership on runtime? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)
On Fri, Aug 30, 2019 at 04:39:39PM +0200, Marco Gaiarin wrote:
But, the 'code' that identify (and change permission) for journal dev are PVE specific? or Ceph generic? I suppose the latter...
OK, trying to identify how OSDs get initialized. If i understood well:
0) systemd unit for every OSD get created following a template: /lib/systemd/system/ceph-osd@.service
1) every unit call a 'prestart' script: ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i
2) The prestart script, run udev:
udevadm settle --timeout=5
that simply force the processing of udev queue, only to be sure there's some 'unhandled' device in the queue.
3) udev (rules in /lib/udev/rules.d/95-ceph-osd.rules), looking for GPT ID_PART_ENTRY_TYPE do two things:
a) ceph-disk --log-stdout -v trigger /dev/$name (that AFAIK trigger a disk mount, for filestore)
b) chown ceph:ceph /dev/$name; chmod 660 /dev/$name
So, seems to me that a decent method to solve/circumvent my trouble is to:
i) write a 'static' udev rule that chown ceph:ceph the partition. Very dirty.
ii) modify the systemd unit and add an ExecStartPost= script that chown the partition. Dirty but probably effective.
iii) modify /usr/lib/ceph/ceph-osd-prestart.sh to add the condition, something like (untested):
if [ -L "$journal" -a -e "$journal" ]; then dev_journal=`readlink -f $journal` owner=`stat -c %U $dev_journal` if [ $owner != 'ceph' ]; then echo "ceph-osd(${cluster:-ceph}-$id): journal probably manually symlinked, fixing permission." 1>&2 chown ceph: $dev_journal fi fi
I'm not a ceph expert, but solution iii) seems decent for me, with a little overhead (a readlinkk and a stat for every osd start).
However you like it. But to note that in Ceph Nautilus the udev rules aren't shipped anymore.
But still i don't understood why, if i have:
root@capitanmarvel:~# LANG=C id ceph uid=64045(ceph) gid=64045(ceph) groups=64045(ceph),6(disk)
and: brw-rw---- 1 root disk 8, 6 ago 28 14:38 /dev/sda6
(so, journal partition group-owned by 'disk' and 'ceph' user in group 'disk'), still i have permission access.
The ceph-osd process reset group ownership on runtime?
In Luminous udev is handling all of that, see 95-ceph-osd.rules. -- Cheers, Alwin
Mandi! Alwin Antreich In chel di` si favelave...
I'm not a ceph expert, but solution iii) seems decent for me, with a little overhead (a readlinkk and a stat for every osd start). However you like it. But to note that in Ceph Nautilus the udev rules aren't shipped anymore.
Ok. I make a note.
But still i don't understood why, if i have: and: (so, journal partition group-owned by 'disk' and 'ceph' user in group 'disk'), still i have permission access. The ceph-osd process reset group ownership on runtime? In Luminous udev is handling all of that, see 95-ceph-osd.rules.
No, sorry, evidently i'm not explaining myself correctly. I've added the 'ceph' user to group 'disk':
root@capitanmarvel:~# LANG=C id ceph uid=64045(ceph) gid=64045(ceph) groups=64045(ceph),6(disk)
and journal devices are group-owned by 'disk' and have read and write permission for the group (660):
brw-rw---- 1 root disk 8, 6 ago 28 14:38 /dev/sda6
So, because user 'ceph' are in group 'disk', and group 'disk' have read and write permission to the device, i can ACTUALLY read and write to the device. But is not the case. So, seems to me that 'ceph-osd' process ''reset'' group membership and ignore the 'disk' group. Note, that if i 'su' to ceph, i can read the disks: ceph@capitanmarvel:~$ ls -la /dev/sda6 brw-rw---- 1 root disk 8, 6 ago 28 14:38 /dev/sda6 ceph@capitanmarvel:~$ dd if=/dev/sda6 count=1 | strings 1+0 record dentro 1+0 record fuori 512 bytes copied, 0,000479581 s, 1,1 MB/s bm?hF lB@4 l+<t -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)
Mandi! Alwin Antreich In chel di` si favelave...
I'm not a ceph expert, but solution iii) seems decent for me, with a little overhead (a readlinkk and a stat for every osd start). However you like it. But to note that in Ceph Nautilus the udev rules aren't shipped anymore.
Ok. I make a note.
But still i don't understood why, if i have: and: (so, journal partition group-owned by 'disk' and 'ceph' user in group 'disk'), still i have permission access. The ceph-osd process reset group ownership on runtime? In Luminous udev is handling all of that, see 95-ceph-osd.rules.
No, sorry, evidently i'm not explaining myself correctly.
I've added the 'ceph' user to group 'disk':
root@capitanmarvel:~# LANG=C id ceph uid=64045(ceph) gid=64045(ceph) groups=64045(ceph),6(disk)
and journal devices are group-owned by 'disk' and have read and write permission for the group (660):
brw-rw---- 1 root disk 8, 6 ago 28 14:38 /dev/sda6
So, because user 'ceph' are in group 'disk', and group 'disk' have read and write permission to the device, i can ACTUALLY read and write to the device. But is not the case.
So, seems to me that 'ceph-osd' process ''reset'' group membership and ignore the 'disk' group. Note, that if i 'su' to ceph, i can read the disks: The ceph-osd process runs with user and group ceph and this is why it wants
On Tue, Sep 03, 2019 at 05:42:22PM +0200, Marco Gaiarin wrote: the group on the disk to be ceph as well. ps xa -o user,group,command | grep ceph-osd You would need to change the service template and alter the setgroup to 'disk' or 'ceph' to make it work. -- 8< -- CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@1.service └─7643 /usr/bin/ceph-osd -f --cluster ceph --id 1 --setuser ceph --setgroup ceph -- 8< -- I still recommend to go with an extra SSD and spare the hassle. -- Cheers, Alwin
I'm not a ceph expert, but solution iii) seems decent for me, with a little overhead (a readlinkk and a stat for every osd start).
I've tested my patch and work as expected, i've created: https://tracker.ceph.com/issues/41777 Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)
participants (2)
-
Alwin Antreich
-
Marco Gaiarin