29 Aug
2019
29 Aug
'19
3:02 p.m.
Riprendo quanto scritto nel suo messaggio del 29/08/2019...
Another possibilty is to convert the MBR to GPT
(sgdisk --mbrtogpt) and
give the partition its UID (also sgdisk). Then it could be linked by
its uuid.
and, in another email:
And I forgot that you can also re-create the journal
by itself. I can't
recall the command ATM though.
Ahem, i stated the jornal disk are also the OS disks, and i'm using old
server, so i think that converting to GPT will lead to an unbootable
node...
But, the 'code' that identify (and change permission) for journal dev
are PVE specific? or Ceph generic? I suppose the latter...
Also, i've done:
adduser ceph disk
and partition devices are '660 root:disk': why still i get 'permission
denied'?
Or if you are not in need of filestore OSDs, re-create
them as bluestore
ones. AFAICS, Ceph has laid more focus on bluestore and it might be
better to do a conversion sooner than later. (my opinion)
Not for now; bluestore migration need a bit more
time/study/knowledge...
--
dott. Marco Gaiarin GNUPG Key ID: 240A3D66
Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/
Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN)
marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797
Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)
30 Aug
30 Aug
2:08 p.m.
New subject: FileStore OSD, journal direct symlinked, permission troubles.
On Thu, Aug 29, 2019 at 05:02:11PM +0200, Marco Gaiarin wrote:
Riprendo quanto scritto nel suo messaggio del
29/08/2019...
IIRC, in Jewel, Ceph
went from running services as root to its own user
'ceph'. The permissions had to be changed once, during upgrade. And then
udev takes care of it by checking the GUID [0].
:~# ls /lib/udev/rules.d/60-ceph-by-parttypeuuid.rules
:~# ls /lib/udev/rules.d/95-ceph-osd.rules
Udev takes care of the correct device node owner. Changing these to
work with the msdos partition scheme will not be very practical. Alone
that you have to adapt the rule with each Ceph update.
Another possibilty is to convert the MBR to GPT
(sgdisk --mbrtogpt) and
give the partition its UID (also sgdisk). Then it could be linked by
its uuid.
and, in another email:
And I forgot that you can also re-create the
journal by itself. I can't
recall the command ATM though.
Ahem, i stated the jornal disk are also the OS disks, and i'm using old
server, so i think that converting to GPT will lead to an unbootable
node...
But, the 'code' that identify (and change permission) for journal dev
are PVE specific? or Ceph generic? I suppose the latter...
Also, i've done:
adduser ceph disk
and partition devices are '660 root:disk': why still i get 'permission
denied'?
User and group is ceph on my PVE 5.x and later installs. The user
&
group should have been created during upgrade. And as said above udev
should take care of it for the device node.
As it is possible to re-create the journal for an OSD, you may consider
to add an extra SSD to create the journals there. Then you can free up
the OS disks of that burden and have a GPT partition table with the
proper GUID. Less risky, then my approach below.
I tested an conversion from MBR to GPT on the boot disk in a VM. To make
it work, I had to run the following.
## Caution: If it fails the system will be not booting
# converts to GPT
sgdisk -g /dev/sdX
# the conversion left 1MB space
# create a new partition with type ef02 on the boot disk uses the 1MB
gdisk /dev/sda
# install grub, if the disk doesn't contain a bios boot partition
# (ef02) it will not install
grub-install /dev/sda
After the conversion it should be no problem to got through the
permission fix on our upgrade guide [0] to get the journals recognised
by Ceph.
Well, I hope this helps.
--
Cheers,
Alwin
[0] https://pve.proxmox.com/wiki/Ceph_Hammer_to_Jewel#Set_permission
2:39 p.m.
New subject: FileStore OSD, journal direct symlinked, permission troubles.
But, the 'code' that identify (and change
permission) for journal dev
are PVE specific? or Ceph generic? I suppose the latter...
OK, trying to identify how OSDs get initialized. If i understood well:
0) systemd unit for every OSD get created following a template:
/lib/systemd/system/ceph-osd@.service
1) every unit call a 'prestart' script:
ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i
2) The prestart script, run udev:
udevadm settle --timeout=5
that simply force the processing of udev queue, only to be sure
there's some 'unhandled' device in the queue.
3) udev (rules in /lib/udev/rules.d/95-ceph-osd.rules), looking for
GPT ID_PART_ENTRY_TYPE do two things:
a)
ceph-disk --log-stdout -v trigger /dev/$name
(that AFAIK trigger a disk mount, for filestore)
b)
chown ceph:ceph /dev/$name; chmod 660 /dev/$name
So, seems to me that a decent method to solve/circumvent my trouble is
to:
i) write a 'static' udev rule that chown ceph:ceph the partition. Very
dirty.
ii) modify the systemd unit and add an ExecStartPost= script that chown
the partition. Dirty but probably effective.
iii) modify /usr/lib/ceph/ceph-osd-prestart.sh to add the condition,
something like (untested):
if [ -L "$journal" -a -e "$journal" ]; then
dev_journal=`readlink -f $journal`
owner=`stat -c %U $dev_journal`
if [ $owner != 'ceph' ]; then
echo "ceph-osd(${cluster:-ceph}-$id): journal probably manually symlinked, fixing
permission." 1>&2
chown ceph: $dev_journal
fi
fi
I'm not a ceph expert, but solution iii) seems decent for me, with a
little overhead (a readlinkk and a stat for every osd start).
But still i don't understood why, if i have:
root@capitanmarvel:~# LANG=C id ceph
uid=64045(ceph) gid=64045(ceph) groups=64045(ceph),6(disk)
and:
brw-rw---- 1 root disk 8, 6 ago 28 14:38 /dev/sda6
(so, journal partition group-owned by 'disk' and 'ceph' user in group
'disk'), still i have permission access.
The ceph-osd process reset group ownership on runtime?
Thanks.
--
dott. Marco Gaiarin GNUPG Key ID: 240A3D66
Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/
Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN)
marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797
Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)
2 Sep
2 Sep
8:30 a.m.
New subject: FileStore OSD, journal direct symlinked, permission troubles.
On Fri, Aug 30, 2019 at 04:39:39PM +0200, Marco Gaiarin wrote:
However you like it.
But to note that in Ceph Nautilus the udev rules
aren't shipped anymore.
But, the 'code' that identify (and change
permission) for journal dev
are PVE specific? or Ceph generic? I suppose the latter...
OK, trying to identify how OSDs get initialized. If i understood well:
0) systemd unit for every OSD get created following a template:
/lib/systemd/system/ceph-osd@.service
1) every unit call a 'prestart' script:
ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i
2) The prestart script, run udev:
udevadm settle --timeout=5
that simply force the processing of udev queue, only to be sure
there's some 'unhandled' device in the queue.
3) udev (rules in /lib/udev/rules.d/95-ceph-osd.rules), looking for
GPT ID_PART_ENTRY_TYPE do two things:
a)
ceph-disk --log-stdout -v trigger /dev/$name
(that AFAIK trigger a disk mount, for filestore)
b)
chown ceph:ceph /dev/$name; chmod 660 /dev/$name
So, seems to me that a decent method to solve/circumvent my trouble is
to:
i) write a 'static' udev rule that chown ceph:ceph the partition. Very
dirty.
ii) modify the systemd unit and add an ExecStartPost= script that chown
the partition. Dirty but probably effective.
iii) modify /usr/lib/ceph/ceph-osd-prestart.sh to add the condition,
something like (untested):
if [ -L "$journal" -a -e "$journal" ]; then
dev_journal=`readlink -f $journal`
owner=`stat -c %U $dev_journal`
if [ $owner != 'ceph' ]; then
echo "ceph-osd(${cluster:-ceph}-$id): journal probably manually symlinked, fixing
permission." 1>&2
chown ceph: $dev_journal
fi
fi
I'm not a ceph expert, but solution iii) seems decent for me, with a
little overhead (a readlinkk and a stat for every osd start).
But still i don't understood why, if i have:
root@capitanmarvel:~# LANG=C id ceph
uid=64045(ceph) gid=64045(ceph) groups=64045(ceph),6(disk)
and:
brw-rw---- 1 root disk 8, 6 ago 28 14:38 /dev/sda6
(so, journal partition group-owned by 'disk' and 'ceph' user in group
'disk'), still i have permission access.
The ceph-osd process reset group ownership on runtime?
In Luminous udev is handling
all of that, see 95-ceph-osd.rules.
--
Cheers,
Alwin
3 Sep
3 Sep
3:42 p.m.
New subject: FileStore OSD, journal direct symlinked, permission troubles.
Mandi! Alwin Antreich
In chel di` si favelave...
Ok. I make a note.
No, sorry, evidently i'm not explaining myself correctly.
I've added the 'ceph' user to group 'disk':
I'm not a
ceph expert, but solution iii) seems decent for me, with a
little overhead (a readlinkk and a stat for every osd start).
However you like it.
But to note that in Ceph Nautilus the udev rules
aren't shipped anymore. But still i
don't understood why, if i have:
and:
(so, journal partition group-owned by 'disk' and 'ceph' user in group
'disk'), still i have permission access.
The ceph-osd process reset group ownership on runtime?
In Luminous udev is
handling all of that, see 95-ceph-osd.rules. > root@capitanmarvel:~# LANG=C id ceph
> uid=64045(ceph) gid=64045(ceph) groups=64045(ceph),6(disk)
and journal devices are group-owned by 'disk' and have read and write
permission for the group (660):
> brw-rw---- 1 root disk 8, 6 ago 28 14:38
/dev/sda6
So, because user 'ceph' are in group 'disk', and group 'disk' have
read
and write permission to the device, i can ACTUALLY read and write to
the device. But is not the case.
So, seems to me that 'ceph-osd' process ''reset'' group membership
and
ignore the 'disk' group.
Note, that if i 'su' to ceph, i can read the disks:
ceph@capitanmarvel:~$ ls -la /dev/sda6
brw-rw---- 1 root disk 8, 6 ago 28 14:38 /dev/sda6
ceph@capitanmarvel:~$ dd if=/dev/sda6 count=1 | strings
1+0 record dentro
1+0 record fuori
512 bytes copied, 0,000479581 s, 1,1 MB/s
bm?hF
lB@4
l+<t
--
dott. Marco Gaiarin GNUPG Key ID: 240A3D66
Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/
Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN)
marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797
Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)
4 Sep
4 Sep
12:21 p.m.
New subject: FileStore OSD, journal direct symlinked, permission troubles.
On Tue, Sep 03, 2019 at 05:42:22PM +0200, Marco Gaiarin wrote:
Mandi! Alwin Antreich
In chel di` si favelave...
Ok. I make a note.
No, sorry, evidently i'm not explaining myself correctly.
I've added the 'ceph' user to group 'disk':
The ceph-osd process
runs with user and group ceph and this is why it wants
the group on the disk to be ceph as well.
ps xa -o user,group,command | grep ceph-osd
You would need to change the service template and alter the setgroup to
'disk' or 'ceph' to make it work.
-- 8< --
CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd(a)1.service
└─7643 /usr/bin/ceph-osd -f --cluster ceph --id 1 --setuser ceph --setgroup
ceph
-- 8< --
I still recommend to go with an extra SSD and spare the hassle.
--
Cheers,
Alwin
I'm
not a ceph expert, but solution iii) seems decent for me, with a
little overhead (a readlinkk and a stat for every osd start).
However you like it.
But to note that in Ceph Nautilus the udev rules
aren't shipped anymore. But still
i don't understood why, if i have:
and:
(so, journal partition group-owned by 'disk' and 'ceph' user in group
'disk'), still i have permission access.
The ceph-osd process reset group ownership on runtime?
In Luminous udev is
handling all of that, see 95-ceph-osd.rules. > root@capitanmarvel:~# LANG=C id ceph
> uid=64045(ceph) gid=64045(ceph) groups=64045(ceph),6(disk)
and journal devices are group-owned by 'disk' and have read and write
permission for the group (660):
> brw-rw---- 1 root disk 8, 6 ago 28 14:38
/dev/sda6
So, because user 'ceph' are in group 'disk', and group 'disk'
have read
and write permission to the device, i can ACTUALLY read and write to
the device. But is not the case.
So, seems to me that 'ceph-osd' process ''reset'' group
membership and
ignore the 'disk' group.
Note, that if i 'su' to ceph, i can read the disks:
11 Sep
11 Sep
3:01 p.m.
New subject: FileStore OSD, journal direct symlinked, permission troubles.
I'm not a ceph expert, but solution iii) seems
decent for me, with a
little overhead (a readlinkk and a stat for every osd start).
I've tested my patch and work as expected, i've created:
https://tracker.ceph.com/issues/41777
Thanks.
--
dott. Marco Gaiarin GNUPG Key ID: 240A3D66
Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/
Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN)
marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797
Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)
1701
days inactive
1714
days old
6 comments
2 participants
participants (2)
-
Alwin Antreich -
Marco Gaiarin