ceph-volume lvm filestore OSDs fail to start on reboot. Permission denied on journal partition
22 Jan
2020
22 Jan
'20
5 p.m.
After upgrading to Nautilus 14.2.6 from Luminous 12.2.12 we are seeing the
following behavior on OSDs which were created with "ceph-volume lvm create
--filestore --osd-id <osd> --data <device> --journal <journal>"
Upon restart of the server containing these OSDs they fail to start with
the following error in the logs:
2020-01-21 13:36:11.635 7fee633e8a80 -1
filestore(/var/lib/ceph/osd/ceph-199) mount(1928): failed to open
journal /var/lib/ceph/osd/ceph-199/journal: (13) Permission denied
/var/lib/ceph/osd/ceph-199/journal symlinks to /dev/sdc5 in our case and
inspecting the ownership on /dev/sdc5 it is root:root, chowning that to
ceph:ceph causes the osd to start and come back up and in near instantly.
As a note these OSDs we experience this with are OSDs which have previously
failed and been replaced using the above ceph-volume, longer running OSDs
in the same server created with ceph-disk or ceph-volume simple (that have
a corresponding .json in /etc/ceph/osd) start up fine and get ceph:ceph on
their journal partition. Bluestore OSDs also do not have any issue.
My hope is that I can preemptively fix these OSDs before shutting them down
so that reboots happen seamlessly. Thanks for any insight.
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
22 Jan
22 Jan
5:28 p.m.
Den ons 22 jan. 2020 kl 18:01 skrev Wesley Dillingham <wes(a)wesdillingham.com
:
After upgrading to Nautilus 14.2.6 from Luminous
12.2.12 we are seeing the
following behavior on OSDs which were created with "ceph-volume lvm create
--filestore --osd-id <osd> --data <device> --journal <journal>"
Upon restart of the server containing these OSDs they fail to start with
the following error in the logs:
2020-01-21 13:36:11.635 7fee633e8a80 -1 filestore(/var/lib/ceph/osd/ceph-199)
mount(1928): failed to open journal /var/lib/ceph/osd/ceph-199/journal: (13) Permission
denied
/var/lib/ceph/osd/ceph-199/journal symlinks to /dev/sdc5 in our case and
inspecting the ownership on /dev/sdc5 it is root:root, chowning that to
ceph:ceph causes the osd to start and come back up and in near instantly.
As a note these OSDs we experience this with are OSDs which have
previously failed and been replaced using the above ceph-volume, longer
running OSDs in the same server created with ceph-disk or ceph-volume
simple (that have a corresponding .json in /etc/ceph/osd) start up fine and
get ceph:ceph on their journal partition. Bluestore OSDs also do not have
any issue.
My hope is that I can preemptively fix these OSDs before shutting them
down so that reboots happen seamlessly. Thanks for any insight.
Our workaround (not on Nautilus but still) is to add to the pre-run systemd
unit file pointed out like this:
more /usr/lib/systemd/system/ceph-osd\@.service
...
ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh <more args here>
then in that file, after it figures out what your journal should be (even
if it is a symlink), do a chown to ceph:ceph
more /usr/lib/ceph/ceph-osd-prestart.sh
...
journal="$data/journal"
chown --dereference ceph:ceph $journal
so it has the correct perms before the filestore OSD gets started.
--
May the most significant bit of your life be positive.
6:03 p.m.
Mandi! Wesley Dillingham
In chel di` si favelave...
Upon restart of the server containing these OSDs they
fail to start with the
following error in the logs:
I've hit th exactly same trouble. Look at:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/FH4QHTQZJ3R…
https://tracker.ceph.com/issues/41777
still hoping that patch will be integrated upstream...
--
dott. Marco Gaiarin GNUPG Key ID: 240A3D66
Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/
Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN)
marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797
Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)
23 Jan
23 Jan
9:29 a.m.
On Wed, Jan 22, 2020 at 12:00:28PM -0500, Wesley Dillingham wrote:
After upgrading to Nautilus 14.2.6 from Luminous
12.2.12 we are seeing
the following behavior on OSDs which were created with "ceph-volume lvm
create --filestore --osd-id <osd> --data <device> --journal
<journal>"
Upon restart of the server containing these OSDs they fail to start
with the following error in the logs:
2020-01-21 13:36:11.635 7fee633e8a80 -1 filestore(/var/lib/ceph/osd/ceph-199) mo
unt(1928): failed to open journal /var/lib/ceph/osd/ceph-199/journal: (13) Permi
ssion denied
/var/lib/ceph/osd/ceph-199/journal symlinks to /dev/sdc5 in our case
and inspecting the ownership on /dev/sdc5 it is root:root, chowning
that to ceph:ceph causes the osd to start and come back up and in near
instantly.
As a note these OSDs we experience this with are OSDs which have
previously failed and been replaced using the above ceph-volume, longer
running OSDs in the same server created with ceph-disk or ceph-volume
simple (that have a corresponding .json in /etc/ceph/osd) start up fine
and get ceph:ceph on their journal partition. Bluestore OSDs also do
not have any issue.
My hope is that I can preemptively fix these OSDs before shutting them
down so that reboots happen seamlessly. Thanks for any insight.
ceph-volume is
supposed to take care of this via the ceph-volume@ systemd unit.
This is a one shot unit, that should set things up and then start the osd.
The unit name is a bit convoluted: ceph-volume@<osd-id>-<osd-uuid>, there
should
be symbolic link in /etc/systemd/system/multi-user.target.wants/
You can also check cat /var/log/ceph/ceph-volume-systemd.log for any errors.
Feel free to open a tracker ticket on
https://tracker.ceph.com/projects/ceph-volume
Respectfully,
Wes Dillingham
[1]wes(a)wesdillingham.com
[2]LinkedIn
References
1. mailto:wes@wesdillingham.com
2. http://www.linkedin.com/in/wesleydillingham
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Jan Fajerski
Senior Software Engineer Enterprise Storage
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Felix Imendörffer
10:15 p.m.
Thanks all for responses. It seems that both original responders relied on
modification to the osd-prestart script. I can confirm that is working for
me too and am using that as a temporary solution.
Jan, It seems that the log you mentioned shows the unit attempting to do
the right thing here (chown the partition ceph:ceph) however it does not
seem to take. Will continue to look and file a bug/pr if I make any
progress. Here are the logs related to an affected osd (219) and its
journal partition (/dev/sdc5). As a note this log output is from a server
which has the added chown in the osd prestart script.
[2020-01-23 13:03:27,230][systemd][INFO ] raw systemd input received:
lvm-219-529ea347-b129-4b53-81cb-bb5f2d91f8ae
[2020-01-23 13:03:27,231][systemd][INFO ] parsed sub-command: lvm, extra
data: 219-529ea347-b129-4b53-81cb-bb5f2d91f8ae
[2020-01-23 13:03:27,305][ceph_volume.process][INFO ] Running command:
/usr/sbin/ceph-volume lvm trigger 219-529ea347-b129-4b53-81cb-bb5f2d91f8ae
[2020-01-23 13:03:28,235][ceph_volume.process][INFO ] stderr Running
command: /bin/mount -t xfs -o rw,noatime,inode64
/dev/ceph-dd8d5283-fd1a-4114-9b53-6478dab7101c/osd-data-529ea347-b129-4b53-81cb-bb5f2d91f8ae
/var/lib/ceph/osd/ceph-219
[2020-01-23 13:03:28,472][ceph_volume.process][INFO ] stderr Running
command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-219
[2020-01-23 13:27:19,566][ceph_volume.process][INFO ] stderr Running
command: /bin/ln -snf /dev/sdc5 /var/lib/ceph/osd/ceph-219/journal
[2020-01-23 13:27:19,571][ceph_volume.process][INFO ] stderr Running
command: /bin/chown -R ceph:ceph /dev/sdc5
[2020-01-23 13:27:19,576][ceph_volume.process][INFO ] stderr Running
command: /bin/systemctl enable
ceph-volume@lvm-219-529ea347-b129-4b53-81cb-bb5f2d91f8ae
[2020-01-23 13:27:19,672][ceph_volume.process][INFO ] stderr Running
command: /bin/systemctl enable --runtime ceph-osd@219
[2020-01-23 13:27:19,679][ceph_volume.process][INFO ] stderr stderr:
Created symlink from
/run/systemd/system/ceph-osd.target.wants/ceph-osd(a)219.service to
/usr/lib/systemd/system/ceph-osd@.service.
[2020-01-23 13:27:19,765][ceph_volume.process][INFO ] stderr Running
command: /bin/systemctl start ceph-osd@219
[2020-01-23 13:27:19,773][ceph_volume.process][INFO ] stderr -->
ceph-volume lvm activate successful for osd ID: 219
[2020-01-23 13:27:19,784][systemd][INFO ] successfully triggered
activation for: 219-529ea347-b129-4b53-81cb-bb5f2d91f8ae
Respectfully,
*Wes Dillingham*
wes(a)wesdillingham.com
LinkedIn <http://www.linkedin.com/in/wesleydillingham>
On Thu, Jan 23, 2020 at 4:31 AM Jan Fajerski <jfajerski(a)suse.com> wrote:
On Wed, Jan 22, 2020 at 12:00:28PM -0500, Wesley
Dillingham wrote:
After upgrading to Nautilus 14.2.6 from
Luminous 12.2.12 we are seeing
the following behavior on OSDs which were created with "ceph-volume lvm
create --filestore --osd-id <osd> --data <device> --journal
<journal>"
Upon restart of the server containing these OSDs they fail to start
with the following error in the logs:
2020-01-21 13:36:11.635 7fee633e8a80 -1
filestore(/var/lib/ceph/osd/ceph-199) mo
unt(1928): failed to open journal
/var/lib/ceph/osd/ceph-199/journal:
(13) Permi
ssion denied
/var/lib/ceph/osd/ceph-199/journal symlinks to /dev/sdc5 in our case
and inspecting the ownership on /dev/sdc5 it is root:root, chowning
that to ceph:ceph causes the osd to start and come back up and in near
instantly.
As a note these OSDs we experience this with are OSDs which have
previously failed and been replaced using the above ceph-volume, longer
running OSDs in the same server created with ceph-disk or ceph-volume
simple (that have a corresponding .json in /etc/ceph/osd) start up fine
and get ceph:ceph on their journal partition. Bluestore OSDs also do
not have any issue.
My hope is that I can preemptively fix these OSDs before shutting them
down so that reboots happen seamlessly. Thanks for any insight.
ceph-volume is
supposed to take care of this via the ceph-volume@ systemd
unit.
This is a one shot unit, that should set things up and then start the osd.
The unit name is a bit convoluted: ceph-volume@<osd-id>-<osd-uuid>, there
should
be symbolic link in /etc/systemd/system/multi-user.target.wants/
You can also check cat /var/log/ceph/ceph-volume-systemd.log for any
errors.
Feel free to open a tracker ticket on
https://tracker.ceph.com/projects/ceph-volume
Respectfully,
Wes Dillingham
[1]wes(a)wesdillingham.com
[2]LinkedIn
References
1. mailto:wes@wesdillingham.com
2. http://www.linkedin.com/in/wesleydillingham
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Jan Fajerski
Senior Software Engineer Enterprise Storage
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Felix Imendörffer
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1574
days inactive
1575
days old
4 comments
4 participants
participants (4)
-
Jan Fajerski -
Janne Johansson -
Marco Gaiarin -
Wesley Dillingham