4 Apr
2021
4 Apr
'21
10:30 a.m.
Hi all,
As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the manual
upgrade using the precompiled packages, the OSDs was down with the
following messages:
root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all
--> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir
--dev
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
--path /var/lib/ceph/osd/ceph-2 --no-mon-config
Running command: /usr/bin/ln -snf
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
/var/lib/ceph/osd/ceph-2/block
Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
Running command: /usr/bin/systemctl enable
ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352
Running command: /usr/bin/systemctl enable --runtime ceph-osd@2
Running command: /usr/bin/systemctl start ceph-osd@2
--> ceph-volume lvm activate successful for osd ID: 2
Content of /var/log/ceph/ceph-osd.2.log
2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045
(ceph:ceph)
2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0
(0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process
ceph-osd, pid 5484
2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty
--pid-file
2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1*
bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open
/var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted*
2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open OSD
superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or directory*
root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2
â— ceph-osd(a)2.service - Ceph object storage daemon osd.2
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor
preset: enabled)
Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06
+0430; 50s ago
Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh
--cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS)
Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id
2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
Main PID: 5484 (code=exited, status=1/FAILURE)
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Scheduled restart
job, restart counter is at 3.
Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2.
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Start request
repeated too quickly.
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Failed with result
'exit-code'.
Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage
daemon osd.2.
root@osd03:/var/lib/ceph/osd/ceph-2#
root@osd03:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
loop0 7:0 0 55.5M 1 loop
/snap/core18/1988
loop1 7:1 0 69.9M 1 loop
/snap/lxd/19188
loop2 7:2 0 55.5M 1 loop
/snap/core18/1997
loop3 7:3 0 70.4M 1 loop
/snap/lxd/19647
loop4 7:4 0 32.3M 1 loop
/snap/snapd/11402
loop5 7:5 0 32.3M 1 loop
/snap/snapd/11107
sda 8:0 0 80G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 79G 0 part
└─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm /
sdb 8:16 0 16G 0 disk
└─sdb1 8:17 0 16G 0 part
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
b89--bcd4--105b2df26352
253:1 0 16G 0 lvm
root@osd03:~#
root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph
tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime)
root@osd03:/var/lib/ceph/osd/ceph-2#
any help is much appreciated
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
4 Apr
4 Apr
10:51 a.m.
It worth mentioning as I issue the following command, the Ceph OSD starts
and joins the cluster:
/usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti <khoshbakhtib(a)gmail.com>
wrote:
Hi all,
As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the
manual upgrade using the precompiled packages, the OSDs was down with the
following messages:
root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all
--> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir
--dev
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
--path /var/lib/ceph/osd/ceph-2 --no-mon-config
Running command: /usr/bin/ln -snf
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
/var/lib/ceph/osd/ceph-2/block
Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
Running command: /usr/bin/systemctl enable
ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352
Running command: /usr/bin/systemctl enable --runtime ceph-osd@2
Running command: /usr/bin/systemctl start ceph-osd@2
--> ceph-volume lvm activate successful for osd ID: 2
Content of /var/log/ceph/ceph-osd.2.log
2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045
(ceph:ceph)
2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0
(0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process
ceph-osd, pid 5484
2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty
--pid-file
2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1*
bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open
/var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted*
2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open
OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or directory*
root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2
â— ceph-osd(a)2.service - Ceph object storage daemon osd.2
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled;
vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06
+0430; 50s ago
Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh
--cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS)
Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id
2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
Main PID: 5484 (code=exited, status=1/FAILURE)
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Scheduled restart
job, restart counter is at 3.
Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2.
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Start request
repeated too quickly.
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Failed with result
'exit-code'.
Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage
daemon osd.2.
root@osd03:/var/lib/ceph/osd/ceph-2#
root@osd03:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
loop0 7:0 0 55.5M 1 loop
/snap/core18/1988
loop1 7:1 0 69.9M 1 loop
/snap/lxd/19188
loop2 7:2 0 55.5M 1 loop
/snap/core18/1997
loop3 7:3 0 70.4M 1 loop
/snap/lxd/19647
loop4 7:4 0 32.3M 1 loop
/snap/snapd/11402
loop5 7:5 0 32.3M 1 loop
/snap/snapd/11107
sda 8:0 0 80G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 79G 0 part
└─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm /
sdb 8:16 0 16G 0 disk
└─sdb1 8:17 0 16G 0 part
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
b89--bcd4--105b2df26352
253:1 0 16G 0 lvm
root@osd03:~#
root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph
tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime)
root@osd03:/var/lib/ceph/osd/ceph-2#
any help is much appreciated
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
+989128610474
11:28 a.m.
Hello,
+1 Am facing the same problem in ubuntu after upgrade to pacific
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/
ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block:
(1) Operation not permitted
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to
open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or
directoryESC[0m
On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti <khoshbakhtib(a)gmail.com>
wrote:
It worth mentioning as I issue the following command,
the Ceph OSD starts
and joins the cluster:
/usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti <khoshbakhtib(a)gmail.com>
wrote:
Hi all,
As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the
manual upgrade using the precompiled packages, the OSDs was down with the
following messages:
root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all
--> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph
prime-osd-dir
--dev
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
--path /var/lib/ceph/osd/ceph-2 --no-mon-config
Running command: /usr/bin/ln -snf
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
/var/lib/ceph/osd/ceph-2/block
Running command: /usr/bin/chown -h ceph:ceph
/var/lib/ceph/osd/ceph-2/block
Running command: /usr/bin/chown -R ceph:ceph
/dev/dm-1
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
Running command: /usr/bin/systemctl enable
ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352
Running command: /usr/bin/systemctl enable --runtime ceph-osd@2
Running command: /usr/bin/systemctl start ceph-osd@2
--> ceph-volume lvm activate successful for osd ID: 2
Content of /var/log/ceph/ceph-osd.2.log
2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045
(ceph:ceph)
2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0
(0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process
ceph-osd, pid 5484
2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty
--pid-file
2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1*
bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open
/var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted*
2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open
OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
directory*
root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2
â— ceph-osd(a)2.service - Ceph object storage daemon osd.2
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled;
vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06
+0430; 50s ago
Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh
--cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS)
Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
--id
2 --setuser ceph --setgroup ceph (code=exited,
status=1/FAILURE)
Main PID: 5484 (code=exited, status=1/FAILURE)
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Scheduled restart
job, restart counter is at 3.
Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon
osd.2.
Apr 04 14:55:06 osd03 systemd[1]:
ceph-osd(a)2.service: Start request
repeated too quickly.
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Failed with result
'exit-code'.
Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage
daemon osd.2.
root@osd03:/var/lib/ceph/osd/ceph-2#
root@osd03:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
loop0 7:0 0 55.5M 1 loop
/snap/core18/1988
loop1 7:1 0 69.9M 1 loop
/snap/lxd/19188
loop2 7:2 0 55.5M 1 loop
/snap/core18/1997
loop3 7:3 0 70.4M 1 loop
/snap/lxd/19647
loop4 7:4 0 32.3M 1 loop
/snap/snapd/11402
loop5 7:5 0 32.3M 1 loop
/snap/snapd/11107
sda 8:0 0 80G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 79G 0 part
└─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm /
sdb 8:16 0 16G 0 disk
└─sdb1 8:17 0 16G 0 part
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
b89--bcd4--105b2df26352
253:1 0 16G 0 lvm
root@osd03:~#
root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph
tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime)
root@osd03:/var/lib/ceph/osd/ceph-2#
any help is much appreciated
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
+989128610474
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
12:07 p.m.
Are the file permissions correct and UID/guid in passwd both 167?
Sent from my iPhone
On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer(a)gmail.com> wrote:
Hello,
+1 Am facing the same problem in ubuntu after upgrade to pacific
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/
ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block:
(1) Operation not permitted
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to
open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or
directoryESC[0m
On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti <khoshbakhtib(a)gmail.com>
wrote:
It worth mentioning as I issue the following command,
the Ceph OSD starts
and joins the cluster:
/usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti <khoshbakhtib(a)gmail.com>
wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Hi all,
As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the
manual upgrade using the precompiled packages, the OSDs was down with the
following messages:
root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all
--> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph
prime-osd-dir
--dev
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
--path /var/lib/ceph/osd/ceph-2 --no-mon-config
Running command: /usr/bin/ln -snf
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
/var/lib/ceph/osd/ceph-2/block
Running command: /usr/bin/chown -h ceph:ceph
/var/lib/ceph/osd/ceph-2/block
Running command: /usr/bin/chown -R ceph:ceph
/dev/dm-1
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
Running command: /usr/bin/systemctl enable
ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352
Running command: /usr/bin/systemctl enable --runtime ceph-osd@2
Running command: /usr/bin/systemctl start ceph-osd@2
--> ceph-volume lvm activate successful for osd ID: 2
Content of /var/log/ceph/ceph-osd.2.log
2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045
(ceph:ceph)
2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0
(0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process
ceph-osd, pid 5484
2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty
--pid-file
2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1*
bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open
/var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted*
2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open
OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
directory*
root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2
â— ceph-osd(a)2.service - Ceph object storage daemon osd.2
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled;
vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06
+0430; 50s ago
Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh
--cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS)
Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
--id
2 --setuser ceph --setgroup ceph (code=exited,
status=1/FAILURE)
Main PID: 5484 (code=exited, status=1/FAILURE)
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Scheduled restart
job, restart counter is at 3.
Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon
osd.2.
Apr 04 14:55:06 osd03 systemd[1]:
ceph-osd(a)2.service: Start request
repeated too quickly.
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Failed with result
'exit-code'.
Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage
daemon osd.2.
root@osd03:/var/lib/ceph/osd/ceph-2#
root@osd03:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
loop0 7:0 0 55.5M 1 loop
/snap/core18/1988
loop1 7:1 0 69.9M 1 loop
/snap/lxd/19188
loop2 7:2 0 55.5M 1 loop
/snap/core18/1997
loop3 7:3 0 70.4M 1 loop
/snap/lxd/19647
loop4 7:4 0 32.3M 1 loop
/snap/snapd/11402
loop5 7:5 0 32.3M 1 loop
/snap/snapd/11107
sda 8:0 0 80G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 79G 0 part
└─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm /
sdb 8:16 0 16G 0 disk
└─sdb1 8:17 0 16G 0 part
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
b89--bcd4--105b2df26352
253:1 0 16G 0 lvm
root@osd03:~#
root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph
tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime)
root@osd03:/var/lib/ceph/osd/ceph-2#
any help is much appreciated
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
+989128610474
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
12:43 p.m.
Hello,
Permissions are correct. guid/uid is 64045/64045
ls -alh
total 32K
drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 .
drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 ..
lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/...
-rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid
-rw------- 1 ceph ceph 37 Apr 4 14:11 fsid
-rw------- 1 ceph ceph 56 Apr 4 14:11 keyring
-rw------- 1 ceph ceph 6 Apr 4 14:11 ready
-rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release
-rw------- 1 ceph ceph 10 Apr 4 14:11 type
-rw------- 1 ceph ceph 3 Apr 4 14:11 whoami
On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown <
andrew_jbrown(a)hotmail.com> wrote:
Are the file permissions correct and UID/guid in
passwd both 167?
Sent from my iPhone
On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer(a)gmail.com> wrote:
Hello,
+1 Am facing the same problem in ubuntu after upgrade to pacific
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/
ceph-29/block) _read_bdev_label failed to open
/var/lib/ceph/osd/ceph-29/block:
(1) Operation not permitted
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to
open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or
directoryESC[0m
On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti <khoshbakhtib(a)gmail.com>
wrote:
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
open
result
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
It worth mentioning as I issue the following
command, the Ceph OSD starts
and joins the cluster:
/usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti <
khoshbakhtib(a)gmail.com>
wrote:
> Hi all,
>
> As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the
> manual upgrade using the precompiled packages, the OSDs was down with
the
following
messages:
root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all
--> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph
prime-osd-dir
--dev
--path
/var/lib/ceph/osd/ceph-2 --no-mon-config
Running command: /usr/bin/ln -snf
/var/lib/ceph/osd/ceph-2/block
Running command: /usr/bin/chown -h ceph:ceph
/var/lib/ceph/osd/ceph-2/block
> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
> Running command: /usr/bin/systemctl enable
> ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352
> Running command: /usr/bin/systemctl enable --runtime ceph-osd@2
> Running command: /usr/bin/systemctl start ceph-osd@2
> --> ceph-volume lvm activate successful for osd ID: 2
>
> Content of /var/log/ceph/ceph-osd.2.log
> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045
> (ceph:ceph)
> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0
> (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process
> ceph-osd, pid 5484
> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty
> --pid-file
> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1*
> bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to
/var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted*
2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open
OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
directory*
root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2
â— ceph-osd(a)2.service - Ceph object storage daemon osd.2
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled;
vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06
+0430; 50s ago
Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh
--cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS)
Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
--id
2 --setuser ceph --setgroup ceph (code=exited,
status=1/FAILURE)
Main PID: 5484 (code=exited, status=1/FAILURE)
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Scheduled restart
job, restart counter is at 3.
Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon
osd.2.
> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Start request
> repeated too quickly.
> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Failed with > 'exit-code'.
> Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage
> daemon osd.2.
> root@osd03:/var/lib/ceph/osd/ceph-2#
>
> root@osd03:~# lsblk
> NAME MAJ:MIN RM SIZE RO TYPE
MOUNTPOINT
fd0
2:0 1 4K 0 disk
loop0 7:0 0 55.5M 1 loop
/snap/core18/1988
loop1 7:1 0 69.9M 1 loop
/snap/lxd/19188
loop2 7:2 0 55.5M 1 loop
/snap/core18/1997
loop3 7:3 0 70.4M 1 loop
/snap/lxd/19647
loop4 7:4 0 32.3M 1 loop
/snap/snapd/11402
loop5 7:5 0 32.3M 1 loop
/snap/snapd/11107
sda 8:0 0 80G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 79G 0 part
└─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm /
sdb 8:16 0 16G 0 disk
└─sdb1 8:17 0 16G 0 part
b89--bcd4--105b2df26352
253:1 0 16G 0 lvm
root@osd03:~#
root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph
tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime)
root@osd03:/var/lib/ceph/osd/ceph-2#
any help is much appreciated
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
+989128610474
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:17 p.m.
UID and guid should both be 167 I believe.
Make a note of the current values and change them to 167 using usermod and groupmod.
I had just this issue. It’s partly to do with how perms are used within the containers I
think.
I changed the values to 167 in passwd everything worked again. Symptoms for me were OSDs
not starting and permissions/file not found errors.
Sent from my iPhone
On 4 Apr 2021, at 13:43, Lomayani S. Laizer <lomlaizer(a)gmail.com> wrote:
Hello,
Permissions are correct. guid/uid is 64045/64045
ls -alh
total 32K
drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 .
drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 ..
lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/...
-rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid
-rw------- 1 ceph ceph 37 Apr 4 14:11 fsid
-rw------- 1 ceph ceph 56 Apr 4 14:11 keyring
-rw------- 1 ceph ceph 6 Apr 4 14:11 ready
-rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release
-rw------- 1 ceph ceph 10 Apr 4 14:11 type
-rw------- 1 ceph ceph 3 Apr 4 14:11 whoami
On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown
<andrew_jbrown@hotmail.com<mailto:andrew_jbrown@hotmail.com>> wrote:
Are the file permissions correct and UID/guid in passwd both 167?
Sent from my iPhone
On 4 Apr 2021, at 12:29, Lomayani S. Laizer
<lomlaizer@gmail.com<mailto:lomlaizer@gmail.com>> wrote:
Hello,
+1 Am facing the same problem in ubuntu after upgrade to pacific
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/
ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block:
(1) Operation not permitted
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to
open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or
directoryESC[0m
On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti
<khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>>
wrote:
It worth mentioning as I issue the following command,
the Ceph OSD starts
and joins the cluster:
/usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti
<khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>>
wrote:
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
Hi all,
As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the
manual upgrade using the precompiled packages, the OSDs was down with the
following messages:
root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all
--> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph
prime-osd-dir
--dev
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
--path /var/lib/ceph/osd/ceph-2 --no-mon-config
Running command: /usr/bin/ln -snf
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
/var/lib/ceph/osd/ceph-2/block
Running command: /usr/bin/chown -h ceph:ceph
/var/lib/ceph/osd/ceph-2/block
Running command: /usr/bin/chown -R ceph:ceph
/dev/dm-1
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
Running command: /usr/bin/systemctl enable
ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352
Running command: /usr/bin/systemctl enable --runtime ceph-osd@2
Running command: /usr/bin/systemctl start ceph-osd@2
--> ceph-volume lvm activate successful for osd ID: 2
Content of /var/log/ceph/ceph-osd.2.log
2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045
(ceph:ceph)
2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0
(0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process
ceph-osd, pid 5484
2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty
--pid-file
2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1*
bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open
/var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted*
2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open
OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
directory*
root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2
â— ceph-osd(a)2.service - Ceph object storage daemon osd.2
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled;
vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06
+0430; 50s ago
Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh
--cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS)
Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
--id
2 --setuser ceph --setgroup ceph (code=exited,
status=1/FAILURE)
Main PID: 5484 (code=exited, status=1/FAILURE)
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Scheduled restart
job, restart counter is at 3.
Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon
osd.2.
Apr 04 14:55:06 osd03 systemd[1]:
ceph-osd(a)2.service: Start request
repeated too quickly.
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Failed with result
'exit-code'.
Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage
daemon osd.2.
root@osd03:/var/lib/ceph/osd/ceph-2#
root@osd03:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
loop0 7:0 0 55.5M 1 loop
/snap/core18/1988
loop1 7:1 0 69.9M 1 loop
/snap/lxd/19188
loop2 7:2 0 55.5M 1 loop
/snap/core18/1997
loop3 7:3 0 70.4M 1 loop
/snap/lxd/19647
loop4 7:4 0 32.3M 1 loop
/snap/snapd/11402
loop5 7:5 0 32.3M 1 loop
/snap/snapd/11107
sda 8:0 0 80G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 79G 0 part
└─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm /
sdb 8:16 0 16G 0 disk
└─sdb1 8:17 0 16G 0 part
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
b89--bcd4--105b2df26352
253:1 0 16G 0 lvm
root@osd03:~#
root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph
tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime)
root@osd03:/var/lib/ceph/osd/ceph-2#
any help is much appreciated
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
+989128610474
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
2:33 p.m.
I have changed the uid and gid to 167, but still no progress.
cat /etc/group | grep -i ceph
ceph:x:167:
root@osd03:~# cat /etc/passwd | grep -i ceph
ceph:x:167:167:Ceph storage service:/var/lib/ceph:/usr/sbin/nologin
On Sun, Apr 4, 2021 at 6:47 PM Andrew Walker-Brown <
andrew_jbrown(a)hotmail.com> wrote:
UID and guid should both be 167 I believe.
Make a note of the current values and change them to 167 using usermod and
groupmod.
I had just this issue. It’s partly to do with how perms are used within
the containers I think.
I changed the values to 167 in passwd everything worked again. Symptoms
for me were OSDs not starting and permissions/file not found errors.
Sent from my iPhone
On 4 Apr 2021, at 13:43, Lomayani S. Laizer <lomlaizer(a)gmail.com> wrote:
Hello,
Permissions are correct. guid/uid is 64045/64045
ls -alh
total 32K
drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 .
drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 ..
lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/...
-rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid
-rw------- 1 ceph ceph 37 Apr 4 14:11 fsid
-rw------- 1 ceph ceph 56 Apr 4 14:11 keyring
-rw------- 1 ceph ceph 6 Apr 4 14:11 ready
-rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release
-rw------- 1 ceph ceph 10 Apr 4 14:11 type
-rw------- 1 ceph ceph 3 Apr 4 14:11 whoami
On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown <
andrew_jbrown(a)hotmail.com> wrote:
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
+989128610474
Are the file permissions correct and UID/guid in
passwd both 167?
Sent from my iPhone
On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer(a)gmail.com> wrote:
Hello,
+1 Am facing the same problem in ubuntu after upgrade to pacific
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/
ceph-29/block) _read_bdev_label failed to open
/var/lib/ceph/osd/ceph-29/block:
(1) Operation not permitted
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to
open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or
directoryESC[0m
On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti <khoshbakhtib(a)gmail.com
wrote:
> On Sun, Apr 4, 2021 at 3:00 PM
Behzad Khoshbakhti <
khoshbakhtib(a)gmail.com>
> wrote:
>> Hi all,
>>
>> As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the
>> manual upgrade using the precompiled packages, the OSDs was down with
the
>> following messages:
>>
>> root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all
>> --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352
>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
>> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph
> prime-osd-dir
>> --dev
>>
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
>> --path /var/lib/ceph/osd/ceph-2 --no-mon-config
>> Running command: /usr/bin/ln -snf
>>
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
empty
restart
result
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
>> b89--bcd4--105b2df26352
>> 253:1 0 16G 0 lvm
>> root@osd03:~#
>>
>> root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph
>> tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime)
>> root@osd03:/var/lib/ceph/osd/ceph-2#
>>
>> any help is much appreciated
>> --
>>
>> Regards
>> Behzad Khoshbakhti
>> Computer Network Engineer (CCIE #58887)
>>
>>
> --
> Regards
> Behzad Khoshbakhti
> Computer Network Engineer (CCIE #58887)
> +989128610474
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
It worth mentioning as I issue the following
command, the Ceph OSD
starts
and joins the cluster:
/usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup
ceph
/var/lib/ceph/osd/ceph-2/block
Running command: /usr/bin/chown -h ceph:ceph
/var/lib/ceph/osd/ceph-2/block
> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
> Running command: /usr/bin/systemctl enable
> ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352
> Running command: /usr/bin/systemctl enable --runtime ceph-osd@2
> Running command: /usr/bin/systemctl start ceph-osd@2
> --> ceph-volume lvm activate successful for osd ID: 2
>
> Content of /var/log/ceph/ceph-osd.2.log
> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045
> (ceph:ceph)
> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0
> (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process
> ceph-osd, pid 5484
> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore > --pid-file
> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1*
> bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to
open
/var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted*
2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open
OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
directory*
root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2
â— ceph-osd(a)2.service - Ceph object storage daemon osd.2
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled;
vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06
+0430; 50s ago
Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh
--cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS)
Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
--id
> 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
> Main PID: 5484 (code=exited, status=1/FAILURE)
>
> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Scheduled job,
restart counter is at 3.
Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon
osd.2.
> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Start request
> repeated too quickly.
> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Failed with > 'exit-code'.
> Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage
> daemon osd.2.
> root@osd03:/var/lib/ceph/osd/ceph-2#
>
> root@osd03:~# lsblk
> NAME MAJ:MIN RM SIZE RO TYPE
MOUNTPOINT
>> fd0 2:0 1 4K 0 disk
>> loop0 7:0 0 55.5M 1 loop
>> /snap/core18/1988
>> loop1 7:1 0 69.9M 1 loop
>> /snap/lxd/19188
>> loop2 7:2 0 55.5M 1 loop
>> /snap/core18/1997
>> loop3 7:3 0 70.4M 1 loop
>> /snap/lxd/19647
>> loop4 7:4 0 32.3M 1 loop
>> /snap/snapd/11402
>> loop5 7:5 0 32.3M 1 loop
>> /snap/snapd/11107
>> sda 8:0 0 80G 0 disk
>> ├─sda1 8:1 0 1M 0 part
>> ├─sda2 8:2 0 1G 0 part /boot
>> └─sda3 8:3 0 79G 0 part
>> └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm /
>> sdb 8:16 0 16G 0 disk
>> └─sdb1 8:17 0 16G 0 part
>>
>>
2:36 p.m.
And after a reboot what errors are you getting?
Sent from my iPhone
On 4 Apr 2021, at 15:33, Behzad Khoshbakhti <khoshbakhtib(a)gmail.com> wrote:
I have changed the uid and gid to 167, but still no progress.
cat /etc/group | grep -i ceph
ceph:x:167:
root@osd03:~# cat /etc/passwd | grep -i ceph
ceph:x:167:167:Ceph storage service:/var/lib/ceph:/usr/sbin/nologin
On Sun, Apr 4, 2021 at 6:47 PM Andrew Walker-Brown
<andrew_jbrown@hotmail.com<mailto:andrew_jbrown@hotmail.com>> wrote:
UID and guid should both be 167 I believe.
Make a note of the current values and change them to 167 using usermod and groupmod.
I had just this issue. It’s partly to do with how perms are used within the containers I
think.
I changed the values to 167 in passwd everything worked again. Symptoms for me were OSDs
not starting and permissions/file not found errors.
Sent from my iPhone
On 4 Apr 2021, at 13:43, Lomayani S. Laizer
<lomlaizer@gmail.com<mailto:lomlaizer@gmail.com>> wrote:
Hello,
Permissions are correct. guid/uid is 64045/64045
ls -alh
total 32K
drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 .
drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 ..
lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/...
-rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid
-rw------- 1 ceph ceph 37 Apr 4 14:11 fsid
-rw------- 1 ceph ceph 56 Apr 4 14:11 keyring
-rw------- 1 ceph ceph 6 Apr 4 14:11 ready
-rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release
-rw------- 1 ceph ceph 10 Apr 4 14:11 type
-rw------- 1 ceph ceph 3 Apr 4 14:11 whoami
On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown
<andrew_jbrown@hotmail.com<mailto:andrew_jbrown@hotmail.com>> wrote:
Are the file permissions correct and UID/guid in passwd both 167?
Sent from my iPhone
On 4 Apr 2021, at 12:29, Lomayani S. Laizer
<lomlaizer@gmail.com<mailto:lomlaizer@gmail.com>> wrote:
Hello,
+1 Am facing the same problem in ubuntu after upgrade to pacific
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/
ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block:
(1) Operation not permitted
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to
open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or
directoryESC[0m
On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti
<khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>>
wrote:
It worth mentioning as I issue the following command,
the Ceph OSD starts
and joins the cluster:
/usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti
<khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>>
wrote:
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
+989128610474
Hi all,
As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the
manual upgrade using the precompiled packages, the OSDs was down with the
following messages:
root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all
--> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph
prime-osd-dir
--dev
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
--path /var/lib/ceph/osd/ceph-2 --no-mon-config
Running command: /usr/bin/ln -snf
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
/var/lib/ceph/osd/ceph-2/block
Running command: /usr/bin/chown -h ceph:ceph
/var/lib/ceph/osd/ceph-2/block
Running command: /usr/bin/chown -R ceph:ceph
/dev/dm-1
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
Running command: /usr/bin/systemctl enable
ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352
Running command: /usr/bin/systemctl enable --runtime ceph-osd@2
Running command: /usr/bin/systemctl start ceph-osd@2
--> ceph-volume lvm activate successful for osd ID: 2
Content of /var/log/ceph/ceph-osd.2.log
2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045
(ceph:ceph)
2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0
(0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process
ceph-osd, pid 5484
2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty
--pid-file
2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1*
bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open
/var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted*
2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open
OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
directory*
root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2
â— ceph-osd(a)2.service - Ceph object storage daemon osd.2
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled;
vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06
+0430; 50s ago
Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh
--cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS)
Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
--id
2 --setuser ceph --setgroup ceph (code=exited,
status=1/FAILURE)
Main PID: 5484 (code=exited, status=1/FAILURE)
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Scheduled restart
job, restart counter is at 3.
Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon
osd.2.
Apr 04 14:55:06 osd03 systemd[1]:
ceph-osd(a)2.service: Start request
repeated too quickly.
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Failed with result
'exit-code'.
Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage
daemon osd.2.
root@osd03:/var/lib/ceph/osd/ceph-2#
root@osd03:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
loop0 7:0 0 55.5M 1 loop
/snap/core18/1988
loop1 7:1 0 69.9M 1 loop
/snap/lxd/19188
loop2 7:2 0 55.5M 1 loop
/snap/core18/1997
loop3 7:3 0 70.4M 1 loop
/snap/lxd/19647
loop4 7:4 0 32.3M 1 loop
/snap/snapd/11402
loop5 7:5 0 32.3M 1 loop
/snap/snapd/11107
sda 8:0 0 80G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 79G 0 part
└─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm /
sdb 8:16 0 16G 0 disk
└─sdb1 8:17 0 16G 0 part
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
b89--bcd4--105b2df26352
253:1 0 16G 0 lvm
root@osd03:~#
root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph
tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime)
root@osd03:/var/lib/ceph/osd/ceph-2#
any help is much appreciated
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
+989128610474
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
2:41 p.m.
Following is the error:
Apr 4 19:09:39 osd03 systemd[1]: ceph-osd(a)2.service: Scheduled restart
job, restart counter is at 3.
Apr 4 19:09:39 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2.
Apr 4 19:09:39 osd03 systemd[1]: Starting Ceph object storage daemon
osd.2...
Apr 4 19:09:39 osd03 systemd[1]: Started Ceph object storage daemon osd.2.
Apr 4 19:09:39 osd03 ceph-osd[3272]: 2021-04-04T19:09:39.668+0430
7fb5d7819f00 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label
failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted
Apr 4 19:09:39 osd03 ceph-osd[3272]: 2021-04-04T19:09:39.668+0430
7fb5d7819f00 -1 #033[0;31m ** ERROR: unable to open OSD superblock on
/var/lib/ceph/osd/ceph-2: (2) No such file or directory#033[0m
Apr 4 19:09:39 osd03 systemd[1]: ceph-osd(a)2.service: Main process exited,
code=exited, status=1/FAILURE
Apr 4 19:09:39 osd03 systemd[1]: ceph-osd(a)2.service: Failed with result
'exit-code'.
And this the files that shows the file exists:
root@osd03:/var/lib/ceph/osd/ceph-2# ls -l
total 24
lrwxrwxrwx 1 ceph ceph 93 Apr 4 19:09 block ->
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
-rw------- 1 ceph ceph 37 Apr 4 19:09 ceph_fsid
-rw------- 1 ceph ceph 37 Apr 4 19:09 fsid
-rw------- 1 ceph ceph 55 Apr 4 19:09 keyring
-rw------- 1 ceph ceph 6 Apr 4 19:09 ready
-rw------- 1 ceph ceph 10 Apr 4 19:09 type
-rw------- 1 ceph ceph 2 Apr 4 19:09 whoami
root@osd03:/var/lib/ceph/osd/ceph-2#
root@osd03:/var/lib/ceph/osd/ceph-2#
On Sun, Apr 4, 2021 at 7:06 PM Andrew Walker-Brown <
andrew_jbrown(a)hotmail.com> wrote:
And after a reboot what errors are you getting?
Sent from my iPhone
On 4 Apr 2021, at 15:33, Behzad Khoshbakhti <khoshbakhtib(a)gmail.com>
wrote:
I have changed the uid and gid to 167, but still no progress.
cat /etc/group | grep -i ceph
ceph:x:167:
root@osd03:~# cat /etc/passwd | grep -i ceph
ceph:x:167:167:Ceph storage service:/var/lib/ceph:/usr/sbin/nologin
On Sun, Apr 4, 2021 at 6:47 PM Andrew Walker-Brown <
andrew_jbrown(a)hotmail.com> wrote:
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
+989128610474
UID and guid should both be 167 I believe.
Make a note of the current values and change them to 167 using usermod
and groupmod.
I had just this issue. It’s partly to do with how perms are used within
the containers I think.
I changed the values to 167 in passwd everything worked again. Symptoms
for me were OSDs not starting and permissions/file not found errors.
Sent from my iPhone
On 4 Apr 2021, at 13:43, Lomayani S. Laizer <lomlaizer(a)gmail.com> wrote:
Hello,
Permissions are correct. guid/uid is 64045/64045
ls -alh
total 32K
drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 .
drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 ..
lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/...
-rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid
-rw------- 1 ceph ceph 37 Apr 4 14:11 fsid
-rw------- 1 ceph ceph 56 Apr 4 14:11 keyring
-rw------- 1 ceph ceph 6 Apr 4 14:11 ready
-rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release
-rw------- 1 ceph ceph 10 Apr 4 14:11 type
-rw------- 1 ceph ceph 3 Apr 4 14:11 whoami
On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown <
andrew_jbrown(a)hotmail.com> wrote:
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
+989128610474
Are the file permissions correct and UID/guid in
passwd both 167?
Sent from my iPhone
On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer(a)gmail.com> wrote:
Hello,
+1 Am facing the same problem in ubuntu after upgrade to pacific
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/
ceph-29/block) _read_bdev_label failed to open
/var/lib/ceph/osd/ceph-29/block:
(1) Operation not permitted
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable
to
open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or
directoryESC[0m
On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti <
khoshbakhtib(a)gmail.com>
wrote:
It worth mentioning as I issue the following
command, the Ceph OSD
starts
and joins the cluster:
/usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup
ceph
On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti <
khoshbakhtib(a)gmail.com>
wrote:
> Hi all,
>
> As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the
> manual upgrade using the precompiled packages, the OSDs was down with
the
> following messages:
>
> root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all
> --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352
> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph
prime-osd-dir
> --dev
>
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
> --path /var/lib/ceph/osd/ceph-2
--no-mon-config
> Running command: /usr/bin/ln -snf
>
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
> /var/lib/ceph/osd/ceph-2/block
> Running command: /usr/bin/chown -h ceph:ceph
/var/lib/ceph/osd/ceph-2/block
> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
> Running command: /usr/bin/systemctl enable
> ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352
> Running command: /usr/bin/systemctl enable --runtime ceph-osd@2
> Running command: /usr/bin/systemctl start ceph-osd@2
> --> ceph-volume lvm activate successful for osd ID: 2
>
> Content of /var/log/ceph/ceph-osd.2.log
> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to
64045:64045
> (ceph:ceph)
> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0
> (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process
> ceph-osd, pid 5484
> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore
empty
> --pid-file
> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1*
> bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to
open
> /var/lib/ceph/osd/ceph-2/block: (1) Operation
not permitted*
> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to
open
> OSD superblock on /var/lib/ceph/osd/ceph-2:
(2) No such file or
directory*
>
>
> root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2
> â— ceph-osd(a)2.service - Ceph object storage daemon osd.2
> Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled;
> vendor preset: enabled)
> Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06
> +0430; 50s ago
> Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh
> --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS)
> Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
--id
> 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
> Main PID: 5484 (code=exited, status=1/FAILURE)
>
> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Scheduled
restart
> job, restart counter is at 3.
> Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon
osd.2.
> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Start request
> repeated too quickly.
> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Failed with
result
> 'exit-code'.
> Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage
> daemon osd.2.
> root@osd03:/var/lib/ceph/osd/ceph-2#
>
> root@osd03:~# lsblk
> NAME MAJ:MIN RM SIZE RO TYPE
MOUNTPOINT
> fd0 2:0
1 4K 0 disk
> loop0 7:0 0 55.5M 1 loop
> /snap/core18/1988
> loop1 7:1 0 69.9M 1 loop
> /snap/lxd/19188
> loop2 7:2 0 55.5M 1 loop
> /snap/core18/1997
> loop3 7:3 0 70.4M 1 loop
> /snap/lxd/19647
> loop4 7:4 0 32.3M 1 loop
> /snap/snapd/11402
> loop5 7:5 0 32.3M 1 loop
> /snap/snapd/11107
> sda 8:0 0 80G 0 disk
> ├─sda1 8:1 0 1M 0 part
> ├─sda2 8:2 0 1G 0 part /boot
> └─sda3 8:3 0 79G 0 part
> └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm /
> sdb 8:16 0 16G 0 disk
> └─sdb1 8:17 0 16G 0 part
>
>
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
> b89--bcd4--105b2df26352
> 253:1 0 16G 0 lvm
> root@osd03:~#
>
> root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph
> tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime)
> root@osd03:/var/lib/ceph/osd/ceph-2#
>
> any help is much appreciated
> --
>
> Regards
> Behzad Khoshbakhti
> Computer Network Engineer (CCIE #58887)
>
>
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
+989128610474
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7 Apr
7 Apr
5:38 p.m.
Could there be more smooth migration? On my Ubuntu I have the same behavior and my ceph
uid/gud are also 64045.
I started with Luminous in 2018 when it was not containerized, and still continue updating
it with apt.
Since when we have got this hardcoded value of 167 ???
Andrew Walker-Brown wrote:
UID and guid should both be 167 I believe.
Make a note of the current values and change them to 167 using usermod and groupmod.
I had just this issue. It’s partly to do with how perms are used within the containers I
think.
I changed the values to 167 in passwd everything worked again. Symptoms for me were OSDs
not starting and permissions/file not found errors.
Sent from my iPhone
On 4 Apr 2021, at 13:43, Lomayani S. Laizer <lomlaizer(a)gmail.com> wrote:
Hello,
Permissions are correct. guid/uid is 64045/64045
ls -alh
total 32K
drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 .
drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 ..
lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/...
-rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid
-rw------- 1 ceph ceph 37 Apr 4 14:11 fsid
-rw------- 1 ceph ceph 56 Apr 4 14:11 keyring
-rw------- 1 ceph ceph 6 Apr 4 14:11 ready
-rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release
-rw------- 1 ceph ceph 10 Apr 4 14:11 type
-rw------- 1 ceph ceph 3 Apr 4 14:11 whoami
On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown
<andrew_jbrown@hotmail.com<mailto:andrew_jbrown@hotmail.com>> wrote:
Are the file permissions correct and UID/guid in passwd both 167?
Sent from my iPhone
On 4 Apr 2021, at 12:29, Lomayani S. Laizer
<lomlaizer@gmail.com<mailto:lomlaizer@gmail.com>> wrote:
Hello,
+1 Am facing the same problem in ubuntu after upgrade to pacific
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/
ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block:
(1) Operation not permitted
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to
open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or
directoryESC[0m
On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti
<khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>>
wrote:
It worth mentioning as I issue the following
command, the Ceph OSD starts
and joins the cluster:
/usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti
<khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>>
wrote:
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Hi all,
As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the
manual upgrade using the precompiled packages, the OSDs was down with the
following messages:
root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all
--> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph
prime-osd-dir
--dev
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
--path /var/lib/ceph/osd/ceph-2 --no-mon-config
Running command: /usr/bin/ln -snf
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
/var/lib/ceph/osd/ceph-2/block
Running command: /usr/bin/chown -h ceph:ceph
/var/lib/ceph/osd/ceph-2/block
Running command: /usr/bin/chown -R ceph:ceph
/dev/dm-1
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
Running command: /usr/bin/systemctl enable
ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352
Running command: /usr/bin/systemctl enable --runtime ceph-osd@2
Running command: /usr/bin/systemctl start ceph-osd@2
--> ceph-volume lvm activate successful for osd ID: 2
Content of /var/log/ceph/ceph-osd.2.log
2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045
(ceph:ceph)
2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0
(0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process
ceph-osd, pid 5484
2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty
--pid-file
2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1*
bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open
/var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted*
2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open
OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
directory*
root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2
â— ceph-osd(a)2.service - Ceph object storage daemon osd.2
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled;
vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06
+0430; 50s ago
Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh
--cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS)
Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
--id
2 --setuser ceph --setgroup ceph (code=exited,
status=1/FAILURE)
Main PID: 5484 (code=exited, status=1/FAILURE)
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Scheduled restart
job, restart counter is at 3.
Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon
osd.2.
Apr 04 14:55:06 osd03 systemd[1]:
ceph-osd(a)2.service: Start request
repeated too quickly.
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Failed with result
'exit-code'.
Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage
daemon osd.2.
root@osd03:/var/lib/ceph/osd/ceph-2#
root@osd03:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
loop0 7:0 0 55.5M 1 loop
/snap/core18/1988
loop1 7:1 0 69.9M 1 loop
/snap/lxd/19188
loop2 7:2 0 55.5M 1 loop
/snap/core18/1997
loop3 7:3 0 70.4M 1 loop
/snap/lxd/19647
loop4 7:4 0 32.3M 1 loop
/snap/snapd/11402
loop5 7:5 0 32.3M 1 loop
/snap/snapd/11107
sda 8:0 0 80G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 79G 0 part
└─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm /
sdb 8:16 0 16G 0 disk
└─sdb1 8:17 0 16G 0 part
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
b89--bcd4--105b2df26352
253:1 0 16G 0 lvm
root@osd03:~#
root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph
tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime)
root@osd03:/var/lib/ceph/osd/ceph-2#
any help is much appreciated
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
+989128610474
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
8 Apr
8 Apr
6:02 a.m.
I agree. But why does the process start manual without systemd which
obviously has nothing to do with uid/gid 167 ? It is also not really a fix
to let all users change uid/gids...
On Wed, Apr 7, 2021 at 7:39 PM Wladimir Mutel <mwg(a)mwg.dp.ua> wrote:
Could there be more smooth migration? On my Ubuntu I
have the same
behavior and my ceph uid/gud are also 64045.
I started with Luminous in 2018 when it was not containerized, and still
continue updating it with apt.
Since when we have got this hardcoded value of 167 ???
Andrew Walker-Brown wrote:
> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailtolto:
ceph-users-leave(a)ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io<mailtolto: ceph-users-leave(a)ceph.io>
--
UG (haftungsbeschränkt)
* Enrico Kern, Geschäftsführer*
Südstraße 26
01877 Demitz-Thumitz
https://www.stackxperts.com
enrico.kern(a)stackxperts.com
+49 (0) 152 26814501
skype: flyersa
Geschäftsführer Enrico Kern
Amtsgericht Dresden | HRB 38717
UID and guid should both be 167 I believe.
Make a note of the current values and change them to 167 using usermod
and
groupmod.
I had just this issue. It’s partly to do with how perms are used within
the
containers I think.
I changed the values to 167 in passwd everything worked again. Symptoms
for me
were OSDs not starting and permissions/file not found errors.
Sent from my iPhone
On 4 Apr 2021, at 13:43, Lomayani S. Laizer <lomlaizer(a)gmail.com> wrote:
Hello,
Permissions are correct. guid/uid is 64045/64045
ls -alh
total 32K
drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 .
drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 ..
lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/...
-rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid
-rw------- 1 ceph ceph 37 Apr 4 14:11 fsid
-rw------- 1 ceph ceph 56 Apr 4 14:11 keyring
-rw------- 1 ceph ceph 6 Apr 4 14:11 ready
-rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release
-rw------- 1 ceph ceph 10 Apr 4 14:11 type
-rw------- 1 ceph ceph 3 Apr 4 14:11 whoami
On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown <
andrew_jbrown@hotmail.com<mailto:andrew_jbrown@hotmail.com>> wrote:
Are the file permissions correct and UID/guid in
passwd both 167?
Sent from my iPhone
On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer(a)gmail.com<mailtomailto:
lomlaizer(a)gmail.com>> wrote:
Hello,
+1 Am facing the same problem in ubuntu after upgrade to pacific
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/
ceph-29/block) _read_bdev_label failed to open
/var/lib/ceph/osd/ceph-29/block:
(1) Operation not permitted
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable
to
open OSD superblock on /var/lib/ceph/osd/ceph-29:
(2) No such file or
directoryESC[0m
On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti <
khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>>
wrote:
> It worth mentioning as I issue the following command, the Ceph OSD
starts
> and joins the cluster:
> /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup
ceph
>
>
>
> On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti <
khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>>
> wrote:
>
>> Hi all,
>>
>> As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the
>> manual upgrade using the precompiled packages, the OSDs was down with
the
>> following messages:
>>
>> root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all
>> --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352
>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
>> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph
> prime-osd-dir
>> --dev
>>
>
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
>> --path /var/lib/ceph/osd/ceph-2
--no-mon-config
>> Running command: /usr/bin/ln -snf
>>
>
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
>> /var/lib/ceph/osd/ceph-2/block
>> Running command: /usr/bin/chown -h ceph:ceph
> /var/lib/ceph/osd/ceph-2/block
>> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
>> Running command: /usr/bin/systemctl enable
>> ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352
>> Running command: /usr/bin/systemctl enable --runtime ceph-osd@2
>> Running command: /usr/bin/systemctl start ceph-osd@2
>> --> ceph-volume lvm activate successful for osd ID: 2
>>
>> Content of /var/log/ceph/ceph-osd.2.log
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045
>> (ceph:ceph)
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0
>> (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process
>> ceph-osd, pid 5484
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore
empty
>> --pid-file
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1*
>> bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to
open
>> /var/lib/ceph/osd/ceph-2/block: (1)
Operation not permitted*
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open
>> OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
> directory*
>>
>>
>> root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2
>> â— ceph-osd(a)2.service - Ceph object storage daemon osd.2
>> Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled;
>> vendor preset: enabled)
>> Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06
>> +0430; 50s ago
>> Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh
>> --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS)
>> Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
> --id
>> 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
>> Main PID: 5484 (code=exited, status=1/FAILURE)
>>
>> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Scheduled
restart
>> job, restart counter is at 3.
>> Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon
> osd.2.
>> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Start request
>> repeated too quickly.
>> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Failed with
result
>> 'exit-code'.
>> Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage
>> daemon osd.2.
>> root@osd03:/var/lib/ceph/osd/ceph-2#
>>
>> root@osd03:~# lsblk
>> NAME MAJ:MIN RM SIZE RO TYPE
MOUNTPOINT
>> fd0
2:0 1 4K 0 disk
>> loop0 7:0 0 55.5M 1 loop
>> /snap/core18/1988
>> loop1 7:1 0 69.9M 1 loop
>> /snap/lxd/19188
>> loop2 7:2 0 55.5M 1 loop
>> /snap/core18/1997
>> loop3 7:3 0 70.4M 1 loop
>> /snap/lxd/19647
>> loop4 7:4 0 32.3M 1 loop
>> /snap/snapd/11402
>> loop5 7:5 0 32.3M 1 loop
>> /snap/snapd/11107
>> sda 8:0 0 80G 0 disk
>> ├─sda1 8:1 0 1M 0 part
>> ├─sda2 8:2 0 1G 0 part /boot
>> └─sda3 8:3 0 79G 0 part
>> └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm /
>> sdb 8:16 0 16G 0 disk
>> └─sdb1 8:17 0 16G 0 part
>>
>>
>
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
b89--bcd4--105b2df26352
253:1 0 16G 0 lvm
root@osd03:~#
root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph
tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime)
root@osd03:/var/lib/ceph/osd/ceph-2#
any help is much appreciated
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
+989128610474
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io _______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:48 a.m.
I believe there is some of problem in the systemd as the ceph starts
successfully by running manually using the ceph-osd command.
On Thu, Apr 8, 2021, 10:32 AM Enrico Kern <enrico.kern(a)stackxperts.com>
wrote:
I agree. But why does the process start manual without
systemd which
obviously has nothing to do with uid/gid 167 ? It is also not really a fix
to let all users change uid/gids...
On Wed, Apr 7, 2021 at 7:39 PM Wladimir Mutel <mwg(a)mwg.dp.ua> wrote:
<mailto:
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
open
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
ceph-users(a)ceph.io>
--
UG (haftungsbeschränkt)
* Enrico Kern, Geschäftsführer*
Südstraße 26
01877 Demitz-Thumitz
https://www.stackxperts.com
enrico.kern(a)stackxperts.com
+49 (0) 152 26814501
skype: flyersa
Geschäftsführer Enrico Kern
Amtsgericht Dresden | HRB 38717
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Could there be more smooth migration? On my
Ubuntu I have the same
behavior and my ceph uid/gud are also 64045.
I started with Luminous in 2018 when it was not containerized, and still
continue updating it with apt.
Since when we have got this hardcoded value of 167 ???
Andrew Walker-Brown wrote:
wrote:
UID and guid should both be 167 I believe.
Make a note of the current values and change them to 167 using usermod
and
groupmod.
I had just this issue. It’s partly to do with how perms are used within
the
containers I think.
I changed the values to 167 in passwd everything worked again. Symptoms
for me
were OSDs not starting and permissions/file not found errors.
>
> Sent from my iPhone
>
> On 4 Apr 2021, at 13:43, Lomayani S. Laizer <lomlaizer(a)gmail.com>
Hello,
Permissions are correct. guid/uid is 64045/64045
ls -alh
total 32K
drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 .
drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 ..
lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/...
-rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid
-rw------- 1 ceph ceph 37 Apr 4 14:11 fsid
-rw------- 1 ceph ceph 56 Apr 4 14:11 keyring
-rw------- 1 ceph ceph 6 Apr 4 14:11 ready
-rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release
-rw------- 1 ceph ceph 10 Apr 4 14:11 type
-rw------- 1 ceph ceph 3 Apr 4 14:11 whoami
On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown <
andrew_jbrown@hotmail.com<mailto:andrew_jbrown@hotmail.com>> wrote:
> Are the file permissions correct and UID/guid in passwd both 167?
>
> Sent from my iPhone
>
> On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer(a)gmail.com lomlaizer(a)gmail.com>> wrote:
>
> Hello,
>
> +1 Am facing the same problem in ubuntu after upgrade to pacific
>
> 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1
bluestore(/var/lib/ceph/osd/
ceph-29/block) _read_bdev_label failed to open
/var/lib/ceph/osd/ceph-29/block:
(1) Operation not permitted
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable
to
open OSD superblock on /var/lib/ceph/osd/ceph-29:
(2) No such file or
directoryESC[0m
On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti <
khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>>
wrote:
> It worth mentioning as I issue the following command, the Ceph OSD
starts
> and joins the cluster:
> /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup
ceph
>
>
>
> On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti <
khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>>
> wrote:
>
>> Hi all,
>>
>> As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the
>> manual upgrade using the precompiled packages, the OSDs was down with
the
>> following messages:
>>
>> root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all
>> --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352
>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
>> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph
> prime-osd-dir
>> --dev
>>
>
>>
--path /var/lib/ceph/osd/ceph-2 --no-mon-config
>> Running command: /usr/bin/ln -snf
>>
>
>>> /var/lib/ceph/osd/ceph-2/block
>>> Running command: /usr/bin/chown -h ceph:ceph
>> /var/lib/ceph/osd/ceph-2/block
>>> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
>>> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
>>> Running command: /usr/bin/systemctl enable
>>> ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352
>>> Running command: /usr/bin/systemctl enable --runtime ceph-osd@2
>>> Running command: /usr/bin/systemctl start ceph-osd@2
>>> --> ceph-volume lvm activate successful for osd ID: 2
>>>
>>> Content of /var/log/ceph/ceph-osd.2.log
>>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to
64045:64045
>>
(ceph:ceph)
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0
>> (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process
>> ceph-osd, pid 5484
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore
empty
>> --pid-file
>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1*
>> bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to
open
>>> /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted*
>>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to >>
OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
> directory*
>>
>>
>> root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2
>> â— ceph-osd(a)2.service - Ceph object storage daemon osd.2
>> Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled;
>> vendor preset: enabled)
>> Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06
>> +0430; 50s ago
>> Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh
>> --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS)
>> Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
> --id
>> 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
>> Main PID: 5484 (code=exited, status=1/FAILURE)
>>
>> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Scheduled
restart
>> job, restart counter is at 3.
>> Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon
> osd.2.
>> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Start request
>> repeated too quickly.
>> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Failed with
result
>> 'exit-code'.
>> Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage
>> daemon osd.2.
>> root@osd03:/var/lib/ceph/osd/ceph-2#
>>
>> root@osd03:~# lsblk
>> NAME MAJ:MIN RM SIZE RO TYPE
MOUNTPOINT
>> fd0
2:0 1 4K 0 disk
>> loop0 7:0 0 55.5M 1 loop
>> /snap/core18/1988
>> loop1 7:1 0 69.9M 1 loop
>> /snap/lxd/19188
>> loop2 7:2 0 55.5M 1 loop
>> /snap/core18/1997
>> loop3 7:3 0 70.4M 1 loop
>> /snap/lxd/19647
>> loop4 7:4 0 32.3M 1 loop
>> /snap/snapd/11402
>> loop5 7:5 0 32.3M 1 loop
>> /snap/snapd/11107
>> sda 8:0 0 80G 0 disk
>> ├─sda1 8:1 0 1M 0 part
>> ├─sda2 8:2 0 1G 0 part /boot
>> └─sda3 8:3 0 79G 0 part
>> └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm /
>> sdb 8:16 0 16G 0 disk
>> └─sdb1 8:17 0 16G 0 part
>>
>>
>
>>> b89--bcd4--105b2df26352
>>> 253:1 0 16G 0 lvm
>>> root@osd03:~#
>>>
>>> root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph
>>> tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime)
>>> root@osd03:/var/lib/ceph/osd/ceph-2#
>>>
>>> any help is much appreciated
>>> --
>>>
>>> Regards
>>> Behzad Khoshbakhti
>>> Computer Network Engineer (CCIE #58887)
>>>
>>>
>>
>> --
>>
>> Regards
>> Behzad Khoshbakhti
>> Computer Network Engineer (CCIE #58887)
>> +989128610474
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io<mailtolto:
ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailtolto:
ceph-users-leave(a)ceph.io>
>>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io<mailtolto: To
unsubscribe send an email to ceph-users-leave(a)ceph.io<mailtolto:
ceph-users-leave(a)ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
18 Apr
18 Apr
4:57 a.m.
Hello,
Uncomment ProtectClock=true in /lib/systemd/system/ceph-osd@.service should
fix the issue
On Thu, Apr 8, 2021 at 9:49 AM Behzad Khoshbakhti <khoshbakhtib(a)gmail.com>
wrote:
I believe there is some of problem in the systemd as
the ceph starts
successfully by running manually using the ceph-osd command.
On Thu, Apr 8, 2021, 10:32 AM Enrico Kern <enrico.kern(a)stackxperts.com>
wrote:
<mailto:
unable
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
process
enabled;
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
ceph-users(a)ceph.io>
--
UG (haftungsbeschränkt)
* Enrico Kern, Geschäftsführer*
Südstraße 26
01877 Demitz-Thumitz
https://www.stackxperts.com
enrico.kern(a)stackxperts.com
+49 (0) 152 26814501
skype: flyersa
Geschäftsführer Enrico Kern
Amtsgericht Dresden | HRB 38717
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
I agree. But why does the process start manual
without systemd which
obviously has nothing to do with uid/gid 167 ? It is also not really a
fix
to let all users change uid/gids...
On Wed, Apr 7, 2021 at 7:39 PM Wladimir Mutel <mwg(a)mwg.dp.ua> wrote:
> Could there be more smooth migration? On my Ubuntu I have the same
> behavior and my ceph uid/gud are also 64045.
> I started with Luminous in 2018 when it was not containerized, and
still
> continue updating it with apt.
> Since when we have got this hardcoded value of 167 ???
>
> Andrew Walker-Brown wrote:
> > UID and guid should both be 167 I believe.
> >
> > Make a note of the current values and change them to 167 using
usermod
> and groupmod.
> >
> > I had just this issue. It’s partly to do with how perms are used
within
> the containers I think.
> >
> > I changed the values to 167 in passwd everything worked again.
Symptoms
for me
were OSDs not starting and permissions/file not found errors.
>
> Sent from my iPhone
>
> On 4 Apr 2021, at 13:43, Lomayani S. Laizer <lomlaizer(a)gmail.com>
wrote:
Hello,
Permissions are correct. guid/uid is 64045/64045
ls -alh
total 32K
drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 .
drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 ..
lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/...
-rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid
-rw------- 1 ceph ceph 37 Apr 4 14:11 fsid
-rw------- 1 ceph ceph 56 Apr 4 14:11 keyring
-rw------- 1 ceph ceph 6 Apr 4 14:11 ready
-rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release
-rw------- 1 ceph ceph 10 Apr 4 14:11 type
-rw------- 1 ceph ceph 3 Apr 4 14:11 whoami
On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown <
andrew_jbrown@hotmail.com<mailto:andrew_jbrown@hotmail.com>> wrote:
> Are the file permissions correct and UID/guid in passwd both 167?
>
> Sent from my iPhone
>
> On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer(a)gmail.com lomlaizer(a)gmail.com>> wrote:
>
> Hello,
>
> +1 Am facing the same problem in ubuntu after upgrade to pacific
>
> 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1
bluestore(/var/lib/ceph/osd/
> > ceph-29/block) _read_bdev_label failed to open
> /var/lib/ceph/osd/ceph-29/block:
> > (1) Operation not permitted
> > 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: > to
> > open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or
> > directoryESC[0m
> >
> > On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti <
> khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>>
> > wrote:
> >
> >> It worth mentioning as I issue the following command, the Ceph OSD
> starts
> >> and joins the cluster:
> >> /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup
> ceph
> >>
> >>
> >>
> >> On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti <
> khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>>
> >> wrote:
> >>
> >>> Hi all,
> >>>
> >>> As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during
the
> >>> manual upgrade using the
precompiled packages, the OSDs was down
with
> the
> >>> following messages:
> >>>
> >>> root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate
--all
> >>> --> Activating OSD ID 2 FSID
2d3ffc61-e430-4b89-bcd4-105b2df26352
> >>> Running command: /usr/bin/chown -R ceph:ceph
/var/lib/ceph/osd/ceph-2
>> Running command: /usr/bin/ceph-bluestore-tool
--cluster=ceph
> prime-osd-dir
>> --dev
>>
>
>> --path /var/lib/ceph/osd/ceph-2
--no-mon-config
>> Running command: /usr/bin/ln -snf
>>
>
> >>> /var/lib/ceph/osd/ceph-2/block
> >>> Running command: /usr/bin/chown -h ceph:ceph
> >> /var/lib/ceph/osd/ceph-2/block
> >>> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
> >>> Running command: /usr/bin/chown -R ceph:ceph
/var/lib/ceph/osd/ceph-2
>>> Running command: /usr/bin/systemctl enable
>>> ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352
>>> Running command: /usr/bin/systemctl enable --runtime ceph-osd@2
>>> Running command: /usr/bin/systemctl start ceph-osd@2
>>> --> ceph-volume lvm activate successful for osd ID: 2
>>>
>>> Content of /var/log/ceph/ceph-osd.2.log
>>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to
64045:64045
> >>> (ceph:ceph)
> >>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0
> >>> (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), > >>> ceph-osd, pid 5484
> >>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore
> empty
> >>> --pid-file
> >>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1*
> >>> bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed
to
open
>>> /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted*
>>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to
open
> >>> OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
> >> directory*
> >>>
> >>>
> >>> root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2
> >>> â— ceph-osd(a)2.service - Ceph object storage daemon osd.2
> >>> Loaded: loaded (/lib/systemd/system/ceph-osd@.service; > >>> vendor preset: enabled)
> >>> Active: failed (Result: exit-code) since Sun 2021-04-04
14:55:06
> >>> +0430; 50s ago
> >>> Process: 5471
ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh
> >>> --cluster ${CLUSTER} --id 2
(code=exited, status=0/SUCCESS)
> >>> Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster
${CLUSTER}
> >> --id
> >>> 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
> >>> Main PID: 5484 (code=exited, status=1/FAILURE)
> >>>
> >>> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Scheduled
> restart
> >>> job, restart counter is at 3.
> >>> Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage
daemon
> >> osd.2.
> >>> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Start
request
> >>> repeated too quickly.
> >>> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Failed with
> result
> >>> 'exit-code'.
> >>> Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object
storage
> >>> daemon osd.2.
> >>> root@osd03:/var/lib/ceph/osd/ceph-2#
> >>>
> >>> root@osd03:~# lsblk
> >>> NAME MAJ:MIN RM SIZE RO TYPE
> MOUNTPOINT
> >>> fd0 2:0 1 4K 0 disk
> >>> loop0 7:0 0 55.5M 1 loop
> >>> /snap/core18/1988
> >>> loop1 7:1 0 69.9M 1 loop
> >>> /snap/lxd/19188
> >>> loop2 7:2 0 55.5M 1 loop
> >>> /snap/core18/1997
> >>> loop3 7:3 0 70.4M 1 loop
> >>> /snap/lxd/19647
> >>> loop4 7:4 0 32.3M 1 loop
> >>> /snap/snapd/11402
> >>> loop5 7:5 0 32.3M 1 loop
> >>> /snap/snapd/11107
> >>> sda 8:0 0 80G 0 disk
> >>> ├─sda1 8:1 0 1M 0 part
> >>> ├─sda2 8:2 0 1G 0 part
/boot
>> └─sda3 8:3
0 79G 0 part
>> └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm /
>> sdb 8:16 0 16G 0 disk
>> └─sdb1 8:17 0 16G 0 part
>>
>>
>
>>> b89--bcd4--105b2df26352
>>> 253:1 0 16G 0 lvm
>>> root@osd03:~#
>>>
>>> root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph
>>> tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime)
>>> root@osd03:/var/lib/ceph/osd/ceph-2#
>>>
>>> any help is much appreciated
>>> --
>>>
>>> Regards
>>> Behzad Khoshbakhti
>>> Computer Network Engineer (CCIE #58887)
>>>
>>>
>>
>> --
>>
>> Regards
>> Behzad Khoshbakhti
>> Computer Network Engineer (CCIE #58887)
>> +989128610474
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io<mailtolto:
ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailtolto:
ceph-users-leave(a)ceph.io>
>>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io<mailtolto: To
unsubscribe send an email to ceph-users-leave(a)ceph.io<mailtolto:
ceph-users-leave(a)ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
19 Apr
19 Apr
3:18 p.m.
thanks by commenting the ProtectClock directive, the issue is resolved.
Thanks for the support.
On Sun, Apr 18, 2021 at 9:28 AM Lomayani S. Laizer <lomlaizer(a)gmail.com>
wrote:
Hello,
Uncomment ProtectClock=true in /lib/systemd/system/ceph-osd@.service
should fix the issue
On Thu, Apr 8, 2021 at 9:49 AM Behzad Khoshbakhti <khoshbakhtib(a)gmail.com>
wrote:
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
+989128610474
I believe there is some of problem in the systemd
as the ceph starts
successfully by running manually using the ceph-osd command.
On Thu, Apr 8, 2021, 10:32 AM Enrico Kern <enrico.kern(a)stackxperts.com>
wrote:
unable
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
process
enabled;
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
I agree. But why does the process start manual
without systemd which
obviously has nothing to do with uid/gid 167 ? It is also not really a
fix
to let all users change uid/gids...
On Wed, Apr 7, 2021 at 7:39 PM Wladimir Mutel <mwg(a)mwg.dp.ua> wrote:
> Could there be more smooth migration? On my Ubuntu I have the same
> behavior and my ceph uid/gud are also 64045.
> I started with Luminous in 2018 when it was not containerized, and
still
> continue updating it with apt.
> Since when we have got this hardcoded value of 167 ???
>
> Andrew Walker-Brown wrote:
> > UID and guid should both be 167 I believe.
> >
> > Make a note of the current values and change them to 167 using
usermod
> and groupmod.
> >
> > I had just this issue. It’s partly to do with how perms are used
within
> the containers I think.
> >
> > I changed the values to 167 in passwd everything worked again.
Symptoms
for me
were OSDs not starting and permissions/file not found errors.
>
> Sent from my iPhone
>
> On 4 Apr 2021, at 13:43, Lomayani S. Laizer <lomlaizer(a)gmail.com>
wrote:
>
>
> Hello,
> Permissions are correct. guid/uid is 64045/64045
>
> ls -alh
> total 32K
> drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 .
> drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 ..
> lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/...
> -rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid
> -rw------- 1 ceph ceph 37 Apr 4 14:11 fsid
> -rw------- 1 ceph ceph 56 Apr 4 14:11 keyring
> -rw------- 1 ceph ceph 6 Apr 4 14:11 ready
> -rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release
> -rw------- 1 ceph ceph 10 Apr 4 14:11 type
> -rw------- 1 ceph ceph 3 Apr 4 14:11 whoami
>
> On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown <
andrew_jbrown@hotmail.com<mailto:andrew_jbrown@hotmail.com>> wrote:
> Are the file permissions correct and UID/guid in passwd both 167?
>
> Sent from my iPhone
>
> On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer(a)gmail.com
<mailto:
lomlaizer(a)gmail.com>> wrote:
>
> Hello,
>
> +1 Am facing the same problem in ubuntu after upgrade to pacific
>
> 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1
bluestore(/var/lib/ceph/osd/
> > ceph-29/block) _read_bdev_label failed to open
> /var/lib/ceph/osd/ceph-29/block:
> > (1) Operation not permitted
> > 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: > to
> > open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file
or
> > directoryESC[0m
> >
> > On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti <
> khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>>
> > wrote:
> >
> >> It worth mentioning as I issue the following command, the Ceph OSD
> starts
> >> and joins the cluster:
> >> /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph
--setgroup
> ceph
> >>
> >>
> >>
> >> On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti <
> khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>>
> >> wrote:
> >>
> >>> Hi all,
> >>>
> >>> As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during
the
> >>> manual upgrade using the
precompiled packages, the OSDs was down
with
> the
> >>> following messages:
> >>>
> >>> root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate
--all
> >>> --> Activating OSD ID 2 FSID
2d3ffc61-e430-4b89-bcd4-105b2df26352
> >>> Running command: /usr/bin/chown -R ceph:ceph
/var/lib/ceph/osd/ceph-2
>>> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph
>> prime-osd-dir
>>> --dev
>>>
>>
>>> --path /var/lib/ceph/osd/ceph-2 --no-mon-config
>>> Running command: /usr/bin/ln -snf
>>>
>>
> >>> /var/lib/ceph/osd/ceph-2/block
> >>> Running command: /usr/bin/chown -h ceph:ceph
> >> /var/lib/ceph/osd/ceph-2/block
> >>> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
> >>> Running command: /usr/bin/chown -R ceph:ceph
/var/lib/ceph/osd/ceph-2
>>> Running command: /usr/bin/systemctl enable
>>> ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352
>>> Running command: /usr/bin/systemctl enable --runtime ceph-osd@2
>>> Running command: /usr/bin/systemctl start ceph-osd@2
>>> --> ceph-volume lvm activate successful for osd ID: 2
>>>
>>> Content of /var/log/ceph/ceph-osd.2.log
>>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to
64045:64045
> >>> (ceph:ceph)
> >>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0
> >>> (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), > >>> ceph-osd, pid 5484
> >>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore
> empty
> >>> --pid-file
> >>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1*
> >>> bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed
to
open
>>> /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted*
>>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to
open
> >>> OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
> >> directory*
> >>>
> >>>
> >>> root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2
> >>> â— ceph-osd(a)2.service - Ceph object storage daemon osd.2
> >>> Loaded: loaded (/lib/systemd/system/ceph-osd@.service; > >>> vendor preset: enabled)
> >>> Active: failed (Result: exit-code) since Sun 2021-04-04
14:55:06
> >>> +0430; 50s ago
> >>> Process: 5471
ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh
> >>> --cluster ${CLUSTER} --id 2
(code=exited, status=0/SUCCESS)
> >>> Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster
${CLUSTER}
> >> --id
> >>> 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
> >>> Main PID: 5484 (code=exited, status=1/FAILURE)
> >>>
> >>> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Scheduled
> restart
> >>> job, restart counter is at 3.
> >>> Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage
daemon
> >> osd.2.
> >>> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Start
request
> >>> repeated too quickly.
> >>> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Failed with
> result
> >>> 'exit-code'.
> >>> Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object
storage
> >>> daemon osd.2.
> >>> root@osd03:/var/lib/ceph/osd/ceph-2#
> >>>
> >>> root@osd03:~# lsblk
> >>> NAME MAJ:MIN RM SIZE RO TYPE
> MOUNTPOINT
> >>> fd0 2:0 1 4K 0 disk
> >>> loop0 7:0 0 55.5M 1 loop
> >>> /snap/core18/1988
> >>> loop1 7:1 0 69.9M 1 loop
> >>> /snap/lxd/19188
> >>> loop2 7:2 0 55.5M 1 loop
> >>> /snap/core18/1997
> >>> loop3 7:3 0 70.4M 1 loop
> >>> /snap/lxd/19647
> >>> loop4 7:4 0 32.3M 1 loop
> >>> /snap/snapd/11402
> >>> loop5 7:5 0 32.3M 1 loop
> >>> /snap/snapd/11107
> >>> sda 8:0 0 80G 0 disk
> >>> ├─sda1 8:1 0 1M 0 part
> >>> ├─sda2 8:2 0 1G 0 part
/boot
>>> └─sda3 8:3 0 79G 0 part
>>> └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm /
>>> sdb 8:16 0 16G 0 disk
>>> └─sdb1 8:17 0 16G 0 part
>>>
>>>
>>
>>> b89--bcd4--105b2df26352
>>> 253:1 0 16G 0 lvm
>>> root@osd03:~#
>>>
>>> root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph
>>> tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime)
>>> root@osd03:/var/lib/ceph/osd/ceph-2#
>>>
>>> any help is much appreciated
>>> --
>>>
>>> Regards
>>> Behzad Khoshbakhti
>>> Computer Network Engineer (CCIE #58887)
>>>
>>>
>>
>> --
>>
>> Regards
>> Behzad Khoshbakhti
>> Computer Network Engineer (CCIE #58887)
>> +989128610474
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io<mailtolto:
ceph-users(a)ceph.io
>
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailtolto:
ceph-users-leave(a)ceph.io>
>>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io<mailtolto:
ceph-users(a)ceph.io>
> To unsubscribe send an email to
ceph-users-leave(a)ceph.io<mailtolto:
ceph-users-leave(a)ceph.io>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
UG (haftungsbeschränkt)
* Enrico Kern, Geschäftsführer*
Südstraße 26
01877 Demitz-Thumitz
https://www.stackxperts.com
enrico.kern(a)stackxperts.com
+49 (0) 152 26814501
skype: flyersa
Geschäftsführer Enrico Kern
Amtsgericht Dresden | HRB 38717
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7 May
7 May
9:47 p.m.
Thanks for reporting this issue. It has been fixed by
https://github.com/ceph/ceph/pull/40845 and will be released in the
next pacific point release.
Neha
On Mon, Apr 19, 2021 at 8:19 AM Behzad Khoshbakhti
<khoshbakhtib(a)gmail.com> wrote:
thanks by commenting the ProtectClock directive, the issue is resolved.
Thanks for the support.
On Sun, Apr 18, 2021 at 9:28 AM Lomayani S. Laizer <lomlaizer(a)gmail.com>
wrote:
Hello,
Uncomment ProtectClock=true in /lib/systemd/system/ceph-osd@.service
should fix the issue
On Thu, Apr 8, 2021 at 9:49 AM Behzad Khoshbakhti <khoshbakhtib(a)gmail.com>
wrote:
--
Regards
Behzad Khoshbakhti
Computer Network Engineer (CCIE #58887)
+989128610474
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io I believe there is some of problem in the systemd
as the ceph starts
successfully by running manually using the ceph-osd command.
On Thu, Apr 8, 2021, 10:32 AM Enrico Kern <enrico.kern(a)stackxperts.com>
wrote:
I agree. But why does the process start manual
without systemd which
obviously has nothing to do with uid/gid 167 ? It is also not really a
fix
to let all users change uid/gids...
On Wed, Apr 7, 2021 at 7:39 PM Wladimir Mutel <mwg(a)mwg.dp.ua> wrote:
> Could there be more smooth migration? On my Ubuntu I have the same
> behavior and my ceph uid/gud are also 64045.
> I started with Luminous in 2018 when it was not containerized, and
still
> continue updating it with apt.
> Since when we have got this hardcoded value of 167 ???
>
> Andrew Walker-Brown wrote:
> > UID and guid should both be 167 I believe.
> >
> > Make a note of the current values and change them to 167 using
usermod
> and groupmod.
> >
> > I had just this issue. It’s partly to do with how perms are used
within
> the containers I think.
> >
> > I changed the values to 167 in passwd everything worked again.
Symptoms
> for me were OSDs not starting and
permissions/file not found errors.
> >
> > Sent from my iPhone
> >
> > On 4 Apr 2021, at 13:43, Lomayani S. Laizer <lomlaizer(a)gmail.com>
wrote:
> >
> >
> > Hello,
> > Permissions are correct. guid/uid is 64045/64045
> >
> > ls -alh
> > total 32K
> > drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 .
> > drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 ..
> > lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/...
> > -rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid
> > -rw------- 1 ceph ceph 37 Apr 4 14:11 fsid
> > -rw------- 1 ceph ceph 56 Apr 4 14:11 keyring
> > -rw------- 1 ceph ceph 6 Apr 4 14:11 ready
> > -rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release
> > -rw------- 1 ceph ceph 10 Apr 4 14:11 type
> > -rw------- 1 ceph ceph 3 Apr 4 14:11 whoami
> >
> > On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown <
> andrew_jbrown@hotmail.com<mailto:andrew_jbrown@hotmail.com>> wrote:
> > Are the file permissions correct and UID/guid in passwd both 167?
> >
> > Sent from my iPhone
> >
> > On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer(a)gmail.com
<mailto:
> lomlaizer(a)gmail.com>> wrote:
> >
> > Hello,
> >
> > +1 Am facing the same problem in ubuntu after upgrade to pacific
> >
> > 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1
bluestore(/var/lib/ceph/osd/
> > ceph-29/block) _read_bdev_label failed to open
> /var/lib/ceph/osd/ceph-29/block:
> > (1) Operation not permitted
> > 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR:
unable
> to
> > open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file
or
> > directoryESC[0m
> >
> > On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti <
> khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>>
> > wrote:
> >
> >> It worth mentioning as I issue the following command, the Ceph OSD
> starts
> >> and joins the cluster:
> >> /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph
--setgroup
> ceph
> >>
> >>
> >>
> >> On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti <
> khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>>
> >> wrote:
> >>
> >>> Hi all,
> >>>
> >>> As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during
the
> >>> manual upgrade using the
precompiled packages, the OSDs was down
with
> the
> >>> following messages:
> >>>
> >>> root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate
--all
> >>> --> Activating OSD ID 2 FSID
2d3ffc61-e430-4b89-bcd4-105b2df26352
> >>> Running command: /usr/bin/chown -R ceph:ceph
/var/lib/ceph/osd/ceph-2
> >>> Running command:
/usr/bin/ceph-bluestore-tool --cluster=ceph
> >> prime-osd-dir
> >>> --dev
> >>>
> >>
>
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
> >>> --path /var/lib/ceph/osd/ceph-2
--no-mon-config
> >>> Running command: /usr/bin/ln -snf
> >>>
> >>
>
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
> >>> /var/lib/ceph/osd/ceph-2/block
> >>> Running command: /usr/bin/chown -h ceph:ceph
> >> /var/lib/ceph/osd/ceph-2/block
> >>> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1
> >>> Running command: /usr/bin/chown -R ceph:ceph
/var/lib/ceph/osd/ceph-2
> >>> Running command:
/usr/bin/systemctl enable
> >>> ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352
> >>> Running command: /usr/bin/systemctl enable --runtime ceph-osd@2
> >>> Running command: /usr/bin/systemctl start ceph-osd@2
> >>> --> ceph-volume lvm activate successful for osd ID: 2
> >>>
> >>> Content of /var/log/ceph/ceph-osd.2.log
> >>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to
64045:64045
> >>> (ceph:ceph)
> >>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0
> >>> (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable),
process
> >>> ceph-osd, pid 5484
> >>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore
> empty
> >>> --pid-file
> >>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1*
> >>> bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed
to
> open
> >>> /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted*
> >>> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to
open
> >>> OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
> >> directory*
> >>>
> >>>
> >>> root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2
> >>> â— ceph-osd(a)2.service - Ceph object storage daemon osd.2
> >>> Loaded: loaded (/lib/systemd/system/ceph-osd@.service;
enabled;
> >>> vendor preset: enabled)
> >>> Active: failed (Result: exit-code) since Sun 2021-04-04
14:55:06
> >>> +0430; 50s ago
> >>> Process: 5471
ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh
> >>> --cluster ${CLUSTER} --id 2
(code=exited, status=0/SUCCESS)
> >>> Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster
${CLUSTER}
> >> --id
> >>> 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
> >>> Main PID: 5484 (code=exited, status=1/FAILURE)
> >>>
> >>> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Scheduled
> restart
> >>> job, restart counter is at 3.
> >>> Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage
daemon
> >> osd.2.
> >>> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Start
request
> >>> repeated too quickly.
> >>> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Failed with
> result
> >>> 'exit-code'.
> >>> Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object
storage
> >>> daemon osd.2.
> >>> root@osd03:/var/lib/ceph/osd/ceph-2#
> >>>
> >>> root@osd03:~# lsblk
> >>> NAME MAJ:MIN RM SIZE RO TYPE
> MOUNTPOINT
> >>> fd0 2:0 1 4K 0 disk
> >>> loop0 7:0 0 55.5M 1 loop
> >>> /snap/core18/1988
> >>> loop1 7:1 0 69.9M 1 loop
> >>> /snap/lxd/19188
> >>> loop2 7:2 0 55.5M 1 loop
> >>> /snap/core18/1997
> >>> loop3 7:3 0 70.4M 1 loop
> >>> /snap/lxd/19647
> >>> loop4 7:4 0 32.3M 1 loop
> >>> /snap/snapd/11402
> >>> loop5 7:5 0 32.3M 1 loop
> >>> /snap/snapd/11107
> >>> sda 8:0 0 80G 0 disk
> >>> ├─sda1 8:1 0 1M 0 part
> >>> ├─sda2 8:2 0 1G 0 part
/boot
> >>> └─sda3
8:3 0 79G 0 part
> >>> └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm /
> >>> sdb 8:16 0 16G 0 disk
> >>> └─sdb1 8:17 0 16G 0 part
> >>>
> >>>
> >>
>
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
> >>> b89--bcd4--105b2df26352
> >>> 253:1 0 16G 0 lvm
> >>> root@osd03:~#
> >>>
> >>> root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph
> >>> tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime)
> >>> root@osd03:/var/lib/ceph/osd/ceph-2#
> >>>
> >>> any help is much appreciated
> >>> --
> >>>
> >>> Regards
> >>> Behzad Khoshbakhti
> >>> Computer Network Engineer (CCIE #58887)
> >>>
> >>>
> >>
> >> --
> >>
> >> Regards
> >> Behzad Khoshbakhti
> >> Computer Network Engineer (CCIE #58887)
> >> +989128610474
> >> _______________________________________________
> >> ceph-users mailing list -- ceph-users(a)ceph.io<mailtolto:
ceph-users(a)ceph.io
> >
> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailtolto:
> ceph-users-leave(a)ceph.io>
> >>
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io<mailtolto:
ceph-users(a)ceph.io>
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailtolto:
> ceph-users-leave(a)ceph.io>
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
--
UG (haftungsbeschränkt)
* Enrico Kern, Geschäftsführer*
Südstraße 26
01877 Demitz-Thumitz
https://www.stackxperts.com
enrico.kern(a)stackxperts.com
+49 (0) 152 26814501
skype: flyersa
Geschäftsführer Enrico Kern
Amtsgericht Dresden | HRB 38717
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1084
days inactive
1117
days old
14 comments
6 participants
participants (6)
-
Andrew Walker-Brown -
Behzad Khoshbakhti -
Enrico Kern -
Lomayani S. Laizer -
Neha Ojha -
Wladimir Mutel