On Fri, Aug 30, 2019 at 04:39:39PM +0200, Marco Gaiarin wrote:
But, the 'code' that identify (and change
permission) for journal dev
are PVE specific? or Ceph generic? I suppose the latter...
OK, trying to identify how OSDs get initialized. If i understood well:
0) systemd unit for every OSD get created following a template:
/lib/systemd/system/ceph-osd@.service
1) every unit call a 'prestart' script:
ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i
2) The prestart script, run udev:
udevadm settle --timeout=5
that simply force the processing of udev queue, only to be sure
there's some 'unhandled' device in the queue.
3) udev (rules in /lib/udev/rules.d/95-ceph-osd.rules), looking for
GPT ID_PART_ENTRY_TYPE do two things:
a)
ceph-disk --log-stdout -v trigger /dev/$name
(that AFAIK trigger a disk mount, for filestore)
b)
chown ceph:ceph /dev/$name; chmod 660 /dev/$name
So, seems to me that a decent method to solve/circumvent my trouble is
to:
i) write a 'static' udev rule that chown ceph:ceph the partition. Very
dirty.
ii) modify the systemd unit and add an ExecStartPost= script that chown
the partition. Dirty but probably effective.
iii) modify /usr/lib/ceph/ceph-osd-prestart.sh to add the condition,
something like (untested):
if [ -L "$journal" -a -e "$journal" ]; then
dev_journal=`readlink -f $journal`
owner=`stat -c %U $dev_journal`
if [ $owner != 'ceph' ]; then
echo "ceph-osd(${cluster:-ceph}-$id): journal probably manually symlinked, fixing
permission." 1>&2
chown ceph: $dev_journal
fi
fi
I'm not a ceph expert, but solution iii) seems decent for me, with a
little overhead (a readlinkk and a stat for every osd start).
However you like it.
But to note that in Ceph Nautilus the udev rules
aren't shipped anymore.
But still i don't understood why, if i have:
root@capitanmarvel:~# LANG=C id ceph
uid=64045(ceph) gid=64045(ceph) groups=64045(ceph),6(disk)
and:
brw-rw---- 1 root disk 8, 6 ago 28 14:38 /dev/sda6
(so, journal partition group-owned by 'disk' and 'ceph' user in group
'disk'), still i have permission access.
The ceph-osd process reset group ownership on runtime?
In Luminous udev is handling
all of that, see 95-ceph-osd.rules.
--
Cheers,
Alwin