Upgrade and lost osds Operation not permitted
Hi all, As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the manual upgrade using the precompiled packages, the OSDs was down with the following messages: root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 --path /var/lib/ceph/osd/ceph-2 --no-mon-config Running command: /usr/bin/ln -snf /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/systemctl enable ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 Running command: /usr/bin/systemctl start ceph-osd@2 --> ceph-volume lvm activate successful for osd ID: 2 Content of /var/log/ceph/ceph-osd.2.log 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045 (ceph:ceph) 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-osd, pid 5484 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty --pid-file 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1* bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted* 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or directory* root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2 â— ceph-osd@2.service - Ceph object storage daemon osd.2 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06 +0430; 50s ago Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS) Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) Main PID: 5484 (code=exited, status=1/FAILURE) Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Scheduled restart job, restart counter is at 3. Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Start request repeated too quickly. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Failed with result 'exit-code'. Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage daemon osd.2. root@osd03:/var/lib/ceph/osd/ceph-2# root@osd03:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk loop0 7:0 0 55.5M 1 loop /snap/core18/1988 loop1 7:1 0 69.9M 1 loop /snap/lxd/19188 loop2 7:2 0 55.5M 1 loop /snap/core18/1997 loop3 7:3 0 70.4M 1 loop /snap/lxd/19647 loop4 7:4 0 32.3M 1 loop /snap/snapd/11402 loop5 7:5 0 32.3M 1 loop /snap/snapd/11107 sda 8:0 0 80G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 79G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm / sdb 8:16 0 16G 0 disk └─sdb1 8:17 0 16G 0 part └─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4 b89--bcd4--105b2df26352 253:1 0 16G 0 lvm root@osd03:~# root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime) root@osd03:/var/lib/ceph/osd/ceph-2# any help is much appreciated -- Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887)
It worth mentioning as I issue the following command, the Ceph OSD starts and joins the cluster: /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti <khoshbakhtib@gmail.com> wrote:
Hi all,
As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the manual upgrade using the precompiled packages, the OSDs was down with the following messages:
root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 --path /var/lib/ceph/osd/ceph-2 --no-mon-config Running command: /usr/bin/ln -snf /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/systemctl enable ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 Running command: /usr/bin/systemctl start ceph-osd@2 --> ceph-volume lvm activate successful for osd ID: 2
Content of /var/log/ceph/ceph-osd.2.log 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045 (ceph:ceph) 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-osd, pid 5484 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty --pid-file 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1* bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted* 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or directory*
root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2 â— ceph-osd@2.service - Ceph object storage daemon osd.2 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06 +0430; 50s ago Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS) Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) Main PID: 5484 (code=exited, status=1/FAILURE)
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Scheduled restart job, restart counter is at 3. Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Start request repeated too quickly. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Failed with result 'exit-code'. Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage daemon osd.2. root@osd03:/var/lib/ceph/osd/ceph-2#
root@osd03:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk loop0 7:0 0 55.5M 1 loop /snap/core18/1988 loop1 7:1 0 69.9M 1 loop /snap/lxd/19188 loop2 7:2 0 55.5M 1 loop /snap/core18/1997 loop3 7:3 0 70.4M 1 loop /snap/lxd/19647 loop4 7:4 0 32.3M 1 loop /snap/snapd/11402 loop5 7:5 0 32.3M 1 loop /snap/snapd/11107 sda 8:0 0 80G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 79G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm / sdb 8:16 0 16G 0 disk └─sdb1 8:17 0 16G 0 part
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4 b89--bcd4--105b2df26352 253:1 0 16G 0 lvm root@osd03:~#
root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime) root@osd03:/var/lib/ceph/osd/ceph-2#
any help is much appreciated --
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887)
-- Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887) +989128610474
Hello, +1 Am facing the same problem in ubuntu after upgrade to pacific 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/ ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block: (1) Operation not permitted 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or directoryESC[0m On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti <khoshbakhtib@gmail.com> wrote:
It worth mentioning as I issue the following command, the Ceph OSD starts and joins the cluster: /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti <khoshbakhtib@gmail.com> wrote:
Hi all,
As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the manual upgrade using the precompiled packages, the OSDs was down with the following messages:
root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
--path /var/lib/ceph/osd/ceph-2 --no-mon-config Running command: /usr/bin/ln -snf
/var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/systemctl enable ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 Running command: /usr/bin/systemctl start ceph-osd@2 --> ceph-volume lvm activate successful for osd ID: 2
Content of /var/log/ceph/ceph-osd.2.log 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045 (ceph:ceph) 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-osd, pid 5484 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty --pid-file 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1* bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted* 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 directory*
root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2 â— ceph-osd@2.service - Ceph object storage daemon osd.2 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06 +0430; 50s ago Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS) Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
--id
2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) Main PID: 5484 (code=exited, status=1/FAILURE)
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Scheduled restart job, restart counter is at 3. Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Start request repeated too quickly. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Failed with result 'exit-code'. Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage daemon osd.2. root@osd03:/var/lib/ceph/osd/ceph-2#
root@osd03:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk loop0 7:0 0 55.5M 1 loop /snap/core18/1988 loop1 7:1 0 69.9M 1 loop /snap/lxd/19188 loop2 7:2 0 55.5M 1 loop /snap/core18/1997 loop3 7:3 0 70.4M 1 loop /snap/lxd/19647 loop4 7:4 0 32.3M 1 loop /snap/snapd/11402 loop5 7:5 0 32.3M 1 loop /snap/snapd/11107 sda 8:0 0 80G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 79G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm / sdb 8:16 0 16G 0 disk └─sdb1 8:17 0 16G 0 part
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
b89--bcd4--105b2df26352 253:1 0 16G 0 lvm root@osd03:~#
root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime) root@osd03:/var/lib/ceph/osd/ceph-2#
any help is much appreciated --
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887)
--
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887) +989128610474 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Are the file permissions correct and UID/guid in passwd both 167? Sent from my iPhone On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer@gmail.com> wrote: Hello, +1 Am facing the same problem in ubuntu after upgrade to pacific 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/ ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block: (1) Operation not permitted 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or directoryESC[0m On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti <khoshbakhtib@gmail.com> wrote:
It worth mentioning as I issue the following command, the Ceph OSD starts and joins the cluster: /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti <khoshbakhtib@gmail.com> wrote:
Hi all,
As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the manual upgrade using the precompiled packages, the OSDs was down with the following messages:
root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
--path /var/lib/ceph/osd/ceph-2 --no-mon-config Running command: /usr/bin/ln -snf
/var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/systemctl enable ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 Running command: /usr/bin/systemctl start ceph-osd@2 --> ceph-volume lvm activate successful for osd ID: 2
Content of /var/log/ceph/ceph-osd.2.log 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045 (ceph:ceph) 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-osd, pid 5484 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty --pid-file 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1* bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted* 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 directory*
root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2 â— ceph-osd@2.service - Ceph object storage daemon osd.2 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06 +0430; 50s ago Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS) Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
--id
2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) Main PID: 5484 (code=exited, status=1/FAILURE)
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Scheduled restart job, restart counter is at 3. Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Start request repeated too quickly. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Failed with result 'exit-code'. Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage daemon osd.2. root@osd03:/var/lib/ceph/osd/ceph-2#
root@osd03:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk loop0 7:0 0 55.5M 1 loop /snap/core18/1988 loop1 7:1 0 69.9M 1 loop /snap/lxd/19188 loop2 7:2 0 55.5M 1 loop /snap/core18/1997 loop3 7:3 0 70.4M 1 loop /snap/lxd/19647 loop4 7:4 0 32.3M 1 loop /snap/snapd/11402 loop5 7:5 0 32.3M 1 loop /snap/snapd/11107 sda 8:0 0 80G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 79G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm / sdb 8:16 0 16G 0 disk └─sdb1 8:17 0 16G 0 part
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
b89--bcd4--105b2df26352 253:1 0 16G 0 lvm root@osd03:~#
root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime) root@osd03:/var/lib/ceph/osd/ceph-2#
any help is much appreciated --
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887)
--
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887) +989128610474 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, Permissions are correct. guid/uid is 64045/64045 ls -alh total 32K drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 . drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 .. lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/... -rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid -rw------- 1 ceph ceph 37 Apr 4 14:11 fsid -rw------- 1 ceph ceph 56 Apr 4 14:11 keyring -rw------- 1 ceph ceph 6 Apr 4 14:11 ready -rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release -rw------- 1 ceph ceph 10 Apr 4 14:11 type -rw------- 1 ceph ceph 3 Apr 4 14:11 whoami On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown < andrew_jbrown@hotmail.com> wrote:
Are the file permissions correct and UID/guid in passwd both 167?
Sent from my iPhone
On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer@gmail.com> wrote:
Hello,
+1 Am facing the same problem in ubuntu after upgrade to pacific
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/ ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block: (1) Operation not permitted 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or directoryESC[0m
On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti <khoshbakhtib@gmail.com> wrote:
It worth mentioning as I issue the following command, the Ceph OSD starts and joins the cluster: /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti < khoshbakhtib@gmail.com> wrote:
Hi all,
As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the manual upgrade using the precompiled packages, the OSDs was down with the following messages:
root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
--path /var/lib/ceph/osd/ceph-2 --no-mon-config Running command: /usr/bin/ln -snf
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
/var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/systemctl enable ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 Running command: /usr/bin/systemctl start ceph-osd@2 --> ceph-volume lvm activate successful for osd ID: 2
Content of /var/log/ceph/ceph-osd.2.log 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045 (ceph:ceph) 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-osd, pid 5484 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty --pid-file 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1* bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted* 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or directory*
root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2 â— ceph-osd@2.service - Ceph object storage daemon osd.2 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06 +0430; 50s ago Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS) Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) Main PID: 5484 (code=exited, status=1/FAILURE)
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Scheduled restart job, restart counter is at 3. Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Start request repeated too quickly. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Failed with result 'exit-code'. Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage daemon osd.2. root@osd03:/var/lib/ceph/osd/ceph-2#
root@osd03:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk loop0 7:0 0 55.5M 1 loop /snap/core18/1988 loop1 7:1 0 69.9M 1 loop /snap/lxd/19188 loop2 7:2 0 55.5M 1 loop /snap/core18/1997 loop3 7:3 0 70.4M 1 loop /snap/lxd/19647 loop4 7:4 0 32.3M 1 loop /snap/snapd/11402 loop5 7:5 0 32.3M 1 loop /snap/snapd/11107 sda 8:0 0 80G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 79G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm / sdb 8:16 0 16G 0 disk └─sdb1 8:17 0 16G 0 part
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
b89--bcd4--105b2df26352 253:1 0 16G 0 lvm root@osd03:~#
root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime) root@osd03:/var/lib/ceph/osd/ceph-2#
any help is much appreciated --
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887)
--
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887) +989128610474 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
UID and guid should both be 167 I believe. Make a note of the current values and change them to 167 using usermod and groupmod. I had just this issue. It’s partly to do with how perms are used within the containers I think. I changed the values to 167 in passwd everything worked again. Symptoms for me were OSDs not starting and permissions/file not found errors. Sent from my iPhone On 4 Apr 2021, at 13:43, Lomayani S. Laizer <lomlaizer@gmail.com> wrote: Hello, Permissions are correct. guid/uid is 64045/64045 ls -alh total 32K drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 . drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 .. lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/... -rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid -rw------- 1 ceph ceph 37 Apr 4 14:11 fsid -rw------- 1 ceph ceph 56 Apr 4 14:11 keyring -rw------- 1 ceph ceph 6 Apr 4 14:11 ready -rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release -rw------- 1 ceph ceph 10 Apr 4 14:11 type -rw------- 1 ceph ceph 3 Apr 4 14:11 whoami On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown <andrew_jbrown@hotmail.com<mailto:andrew_jbrown@hotmail.com>> wrote: Are the file permissions correct and UID/guid in passwd both 167? Sent from my iPhone On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer@gmail.com<mailto:lomlaizer@gmail.com>> wrote: Hello, +1 Am facing the same problem in ubuntu after upgrade to pacific 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/ ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block: (1) Operation not permitted 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or directoryESC[0m On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti <khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>> wrote:
It worth mentioning as I issue the following command, the Ceph OSD starts and joins the cluster: /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti <khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>> wrote:
Hi all,
As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the manual upgrade using the precompiled packages, the OSDs was down with the following messages:
root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
--path /var/lib/ceph/osd/ceph-2 --no-mon-config Running command: /usr/bin/ln -snf
/var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/systemctl enable ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 Running command: /usr/bin/systemctl start ceph-osd@2 --> ceph-volume lvm activate successful for osd ID: 2
Content of /var/log/ceph/ceph-osd.2.log 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045 (ceph:ceph) 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-osd, pid 5484 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty --pid-file 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1* bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted* 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 directory*
root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2 â— ceph-osd@2.service - Ceph object storage daemon osd.2 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06 +0430; 50s ago Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS) Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
--id
2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) Main PID: 5484 (code=exited, status=1/FAILURE)
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Scheduled restart job, restart counter is at 3. Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Start request repeated too quickly. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Failed with result 'exit-code'. Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage daemon osd.2. root@osd03:/var/lib/ceph/osd/ceph-2#
root@osd03:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk loop0 7:0 0 55.5M 1 loop /snap/core18/1988 loop1 7:1 0 69.9M 1 loop /snap/lxd/19188 loop2 7:2 0 55.5M 1 loop /snap/core18/1997 loop3 7:3 0 70.4M 1 loop /snap/lxd/19647 loop4 7:4 0 32.3M 1 loop /snap/snapd/11402 loop5 7:5 0 32.3M 1 loop /snap/snapd/11107 sda 8:0 0 80G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 79G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm / sdb 8:16 0 16G 0 disk └─sdb1 8:17 0 16G 0 part
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
b89--bcd4--105b2df26352 253:1 0 16G 0 lvm root@osd03:~#
root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime) root@osd03:/var/lib/ceph/osd/ceph-2#
any help is much appreciated --
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887)
--
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887) +989128610474 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
I have changed the uid and gid to 167, but still no progress. cat /etc/group | grep -i ceph ceph:x:167: root@osd03:~# cat /etc/passwd | grep -i ceph ceph:x:167:167:Ceph storage service:/var/lib/ceph:/usr/sbin/nologin On Sun, Apr 4, 2021 at 6:47 PM Andrew Walker-Brown < andrew_jbrown@hotmail.com> wrote:
UID and guid should both be 167 I believe.
Make a note of the current values and change them to 167 using usermod and groupmod.
I had just this issue. It’s partly to do with how perms are used within the containers I think.
I changed the values to 167 in passwd everything worked again. Symptoms for me were OSDs not starting and permissions/file not found errors.
Sent from my iPhone
On 4 Apr 2021, at 13:43, Lomayani S. Laizer <lomlaizer@gmail.com> wrote:
Hello, Permissions are correct. guid/uid is 64045/64045
ls -alh total 32K drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 . drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 .. lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/... -rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid -rw------- 1 ceph ceph 37 Apr 4 14:11 fsid -rw------- 1 ceph ceph 56 Apr 4 14:11 keyring -rw------- 1 ceph ceph 6 Apr 4 14:11 ready -rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release -rw------- 1 ceph ceph 10 Apr 4 14:11 type -rw------- 1 ceph ceph 3 Apr 4 14:11 whoami
On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown < andrew_jbrown@hotmail.com> wrote:
Are the file permissions correct and UID/guid in passwd both 167?
Sent from my iPhone
On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer@gmail.com> wrote:
Hello,
+1 Am facing the same problem in ubuntu after upgrade to pacific
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/ ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block: (1) Operation not permitted 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or directoryESC[0m
On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti <khoshbakhtib@gmail.com
wrote:
It worth mentioning as I issue the following command, the Ceph OSD starts and joins the cluster: /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti < khoshbakhtib@gmail.com> wrote:
Hi all,
As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the manual upgrade using the precompiled packages, the OSDs was down with the following messages:
root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
--path /var/lib/ceph/osd/ceph-2 --no-mon-config Running command: /usr/bin/ln -snf
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
/var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/systemctl enable ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 Running command: /usr/bin/systemctl start ceph-osd@2 --> ceph-volume lvm activate successful for osd ID: 2
Content of /var/log/ceph/ceph-osd.2.log 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045 (ceph:ceph) 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-osd, pid 5484 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty --pid-file 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1* bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted* 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or directory*
root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2 â— ceph-osd@2.service - Ceph object storage daemon osd.2 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06 +0430; 50s ago Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS) Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) Main PID: 5484 (code=exited, status=1/FAILURE)
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Scheduled restart job, restart counter is at 3. Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Start request repeated too quickly. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Failed with result 'exit-code'. Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage daemon osd.2. root@osd03:/var/lib/ceph/osd/ceph-2#
root@osd03:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk loop0 7:0 0 55.5M 1 loop /snap/core18/1988 loop1 7:1 0 69.9M 1 loop /snap/lxd/19188 loop2 7:2 0 55.5M 1 loop /snap/core18/1997 loop3 7:3 0 70.4M 1 loop /snap/lxd/19647 loop4 7:4 0 32.3M 1 loop /snap/snapd/11402 loop5 7:5 0 32.3M 1 loop /snap/snapd/11107 sda 8:0 0 80G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 79G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm / sdb 8:16 0 16G 0 disk └─sdb1 8:17 0 16G 0 part
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
b89--bcd4--105b2df26352 253:1 0 16G 0 lvm root@osd03:~#
root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime) root@osd03:/var/lib/ceph/osd/ceph-2#
any help is much appreciated --
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887)
--
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887) +989128610474 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887) +989128610474
And after a reboot what errors are you getting? Sent from my iPhone On 4 Apr 2021, at 15:33, Behzad Khoshbakhti <khoshbakhtib@gmail.com> wrote: I have changed the uid and gid to 167, but still no progress. cat /etc/group | grep -i ceph ceph:x:167: root@osd03:~# cat /etc/passwd | grep -i ceph ceph:x:167:167:Ceph storage service:/var/lib/ceph:/usr/sbin/nologin On Sun, Apr 4, 2021 at 6:47 PM Andrew Walker-Brown <andrew_jbrown@hotmail.com<mailto:andrew_jbrown@hotmail.com>> wrote: UID and guid should both be 167 I believe. Make a note of the current values and change them to 167 using usermod and groupmod. I had just this issue. It’s partly to do with how perms are used within the containers I think. I changed the values to 167 in passwd everything worked again. Symptoms for me were OSDs not starting and permissions/file not found errors. Sent from my iPhone On 4 Apr 2021, at 13:43, Lomayani S. Laizer <lomlaizer@gmail.com<mailto:lomlaizer@gmail.com>> wrote: Hello, Permissions are correct. guid/uid is 64045/64045 ls -alh total 32K drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 . drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 .. lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/... -rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid -rw------- 1 ceph ceph 37 Apr 4 14:11 fsid -rw------- 1 ceph ceph 56 Apr 4 14:11 keyring -rw------- 1 ceph ceph 6 Apr 4 14:11 ready -rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release -rw------- 1 ceph ceph 10 Apr 4 14:11 type -rw------- 1 ceph ceph 3 Apr 4 14:11 whoami On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown <andrew_jbrown@hotmail.com<mailto:andrew_jbrown@hotmail.com>> wrote: Are the file permissions correct and UID/guid in passwd both 167? Sent from my iPhone On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer@gmail.com<mailto:lomlaizer@gmail.com>> wrote: Hello, +1 Am facing the same problem in ubuntu after upgrade to pacific 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/ ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block: (1) Operation not permitted 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or directoryESC[0m On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti <khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>> wrote:
It worth mentioning as I issue the following command, the Ceph OSD starts and joins the cluster: /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti <khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>> wrote:
Hi all,
As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the manual upgrade using the precompiled packages, the OSDs was down with the following messages:
root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
--path /var/lib/ceph/osd/ceph-2 --no-mon-config Running command: /usr/bin/ln -snf
/var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/systemctl enable ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 Running command: /usr/bin/systemctl start ceph-osd@2 --> ceph-volume lvm activate successful for osd ID: 2
Content of /var/log/ceph/ceph-osd.2.log 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045 (ceph:ceph) 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-osd, pid 5484 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty --pid-file 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1* bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted* 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 directory*
root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2 â— ceph-osd@2.service - Ceph object storage daemon osd.2 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06 +0430; 50s ago Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS) Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
--id
2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) Main PID: 5484 (code=exited, status=1/FAILURE)
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Scheduled restart job, restart counter is at 3. Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Start request repeated too quickly. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Failed with result 'exit-code'. Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage daemon osd.2. root@osd03:/var/lib/ceph/osd/ceph-2#
root@osd03:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk loop0 7:0 0 55.5M 1 loop /snap/core18/1988 loop1 7:1 0 69.9M 1 loop /snap/lxd/19188 loop2 7:2 0 55.5M 1 loop /snap/core18/1997 loop3 7:3 0 70.4M 1 loop /snap/lxd/19647 loop4 7:4 0 32.3M 1 loop /snap/snapd/11402 loop5 7:5 0 32.3M 1 loop /snap/snapd/11107 sda 8:0 0 80G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 79G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm / sdb 8:16 0 16G 0 disk └─sdb1 8:17 0 16G 0 part
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
b89--bcd4--105b2df26352 253:1 0 16G 0 lvm root@osd03:~#
root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime) root@osd03:/var/lib/ceph/osd/ceph-2#
any help is much appreciated --
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887)
--
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887) +989128610474 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> -- Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887) +989128610474
Following is the error: Apr 4 19:09:39 osd03 systemd[1]: ceph-osd@2.service: Scheduled restart job, restart counter is at 3. Apr 4 19:09:39 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2. Apr 4 19:09:39 osd03 systemd[1]: Starting Ceph object storage daemon osd.2... Apr 4 19:09:39 osd03 systemd[1]: Started Ceph object storage daemon osd.2. Apr 4 19:09:39 osd03 ceph-osd[3272]: 2021-04-04T19:09:39.668+0430 7fb5d7819f00 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted Apr 4 19:09:39 osd03 ceph-osd[3272]: 2021-04-04T19:09:39.668+0430 7fb5d7819f00 -1 #033[0;31m ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or directory#033[0m Apr 4 19:09:39 osd03 systemd[1]: ceph-osd@2.service: Main process exited, code=exited, status=1/FAILURE Apr 4 19:09:39 osd03 systemd[1]: ceph-osd@2.service: Failed with result 'exit-code'. And this the files that shows the file exists: root@osd03:/var/lib/ceph/osd/ceph-2# ls -l total 24 lrwxrwxrwx 1 ceph ceph 93 Apr 4 19:09 block -> /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 -rw------- 1 ceph ceph 37 Apr 4 19:09 ceph_fsid -rw------- 1 ceph ceph 37 Apr 4 19:09 fsid -rw------- 1 ceph ceph 55 Apr 4 19:09 keyring -rw------- 1 ceph ceph 6 Apr 4 19:09 ready -rw------- 1 ceph ceph 10 Apr 4 19:09 type -rw------- 1 ceph ceph 2 Apr 4 19:09 whoami root@osd03:/var/lib/ceph/osd/ceph-2# root@osd03:/var/lib/ceph/osd/ceph-2# On Sun, Apr 4, 2021 at 7:06 PM Andrew Walker-Brown < andrew_jbrown@hotmail.com> wrote:
And after a reboot what errors are you getting?
Sent from my iPhone
On 4 Apr 2021, at 15:33, Behzad Khoshbakhti <khoshbakhtib@gmail.com> wrote:
I have changed the uid and gid to 167, but still no progress. cat /etc/group | grep -i ceph ceph:x:167: root@osd03:~# cat /etc/passwd | grep -i ceph ceph:x:167:167:Ceph storage service:/var/lib/ceph:/usr/sbin/nologin
On Sun, Apr 4, 2021 at 6:47 PM Andrew Walker-Brown < andrew_jbrown@hotmail.com> wrote:
UID and guid should both be 167 I believe.
Make a note of the current values and change them to 167 using usermod and groupmod.
I had just this issue. It’s partly to do with how perms are used within the containers I think.
I changed the values to 167 in passwd everything worked again. Symptoms for me were OSDs not starting and permissions/file not found errors.
Sent from my iPhone
On 4 Apr 2021, at 13:43, Lomayani S. Laizer <lomlaizer@gmail.com> wrote:
Hello, Permissions are correct. guid/uid is 64045/64045
ls -alh total 32K drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 . drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 .. lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/... -rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid -rw------- 1 ceph ceph 37 Apr 4 14:11 fsid -rw------- 1 ceph ceph 56 Apr 4 14:11 keyring -rw------- 1 ceph ceph 6 Apr 4 14:11 ready -rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release -rw------- 1 ceph ceph 10 Apr 4 14:11 type -rw------- 1 ceph ceph 3 Apr 4 14:11 whoami
On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown < andrew_jbrown@hotmail.com> wrote:
Are the file permissions correct and UID/guid in passwd both 167?
Sent from my iPhone
On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer@gmail.com> wrote:
Hello,
+1 Am facing the same problem in ubuntu after upgrade to pacific
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/ ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block: (1) Operation not permitted 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or directoryESC[0m
On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti < khoshbakhtib@gmail.com> wrote:
It worth mentioning as I issue the following command, the Ceph OSD starts and joins the cluster: /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti < khoshbakhtib@gmail.com> wrote:
Hi all,
As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the manual upgrade using the precompiled packages, the OSDs was down with the following messages:
root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
--path /var/lib/ceph/osd/ceph-2 --no-mon-config Running command: /usr/bin/ln -snf
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
/var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/systemctl enable ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 Running command: /usr/bin/systemctl start ceph-osd@2 --> ceph-volume lvm activate successful for osd ID: 2
Content of /var/log/ceph/ceph-osd.2.log 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045 (ceph:ceph) 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-osd, pid 5484 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty --pid-file 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1* bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted* 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or directory*
root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2 â— ceph-osd@2.service - Ceph object storage daemon osd.2 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06 +0430; 50s ago Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS) Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) Main PID: 5484 (code=exited, status=1/FAILURE)
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Scheduled restart job, restart counter is at 3. Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Start request repeated too quickly. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Failed with result 'exit-code'. Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage daemon osd.2. root@osd03:/var/lib/ceph/osd/ceph-2#
root@osd03:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk loop0 7:0 0 55.5M 1 loop /snap/core18/1988 loop1 7:1 0 69.9M 1 loop /snap/lxd/19188 loop2 7:2 0 55.5M 1 loop /snap/core18/1997 loop3 7:3 0 70.4M 1 loop /snap/lxd/19647 loop4 7:4 0 32.3M 1 loop /snap/snapd/11402 loop5 7:5 0 32.3M 1 loop /snap/snapd/11107 sda 8:0 0 80G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 79G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm / sdb 8:16 0 16G 0 disk └─sdb1 8:17 0 16G 0 part
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
b89--bcd4--105b2df26352 253:1 0 16G 0 lvm root@osd03:~#
root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime) root@osd03:/var/lib/ceph/osd/ceph-2#
any help is much appreciated --
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887)
--
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887) +989128610474 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887) +989128610474
-- Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887) +989128610474
Could there be more smooth migration? On my Ubuntu I have the same behavior and my ceph uid/gud are also 64045. I started with Luminous in 2018 when it was not containerized, and still continue updating it with apt. Since when we have got this hardcoded value of 167 ??? Andrew Walker-Brown wrote:
UID and guid should both be 167 I believe.
Make a note of the current values and change them to 167 using usermod and groupmod.
I had just this issue. It’s partly to do with how perms are used within the containers I think.
I changed the values to 167 in passwd everything worked again. Symptoms for me were OSDs not starting and permissions/file not found errors.
Sent from my iPhone
On 4 Apr 2021, at 13:43, Lomayani S. Laizer <lomlaizer@gmail.com> wrote:
Hello, Permissions are correct. guid/uid is 64045/64045
ls -alh total 32K drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 . drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 .. lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/... -rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid -rw------- 1 ceph ceph 37 Apr 4 14:11 fsid -rw------- 1 ceph ceph 56 Apr 4 14:11 keyring -rw------- 1 ceph ceph 6 Apr 4 14:11 ready -rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release -rw------- 1 ceph ceph 10 Apr 4 14:11 type -rw------- 1 ceph ceph 3 Apr 4 14:11 whoami
On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown <andrew_jbrown@hotmail.com<mailto:andrew_jbrown@hotmail.com>> wrote: Are the file permissions correct and UID/guid in passwd both 167?
Sent from my iPhone
On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer@gmail.com<mailto:lomlaizer@gmail.com>> wrote:
Hello,
+1 Am facing the same problem in ubuntu after upgrade to pacific
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/ ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block: (1) Operation not permitted 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or directoryESC[0m
On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti <khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>> wrote:
It worth mentioning as I issue the following command, the Ceph OSD starts and joins the cluster: /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti <khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>> wrote:
Hi all,
As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the manual upgrade using the precompiled packages, the OSDs was down with the following messages:
root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
--path /var/lib/ceph/osd/ceph-2 --no-mon-config Running command: /usr/bin/ln -snf
/var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/systemctl enable ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 Running command: /usr/bin/systemctl start ceph-osd@2 --> ceph-volume lvm activate successful for osd ID: 2
Content of /var/log/ceph/ceph-osd.2.log 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045 (ceph:ceph) 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-osd, pid 5484 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty --pid-file 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1* bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted* 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 directory*
root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2 â— ceph-osd@2.service - Ceph object storage daemon osd.2 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06 +0430; 50s ago Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS) Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
--id
2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) Main PID: 5484 (code=exited, status=1/FAILURE)
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Scheduled restart job, restart counter is at 3. Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Start request repeated too quickly. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Failed with result 'exit-code'. Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage daemon osd.2. root@osd03:/var/lib/ceph/osd/ceph-2#
root@osd03:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk loop0 7:0 0 55.5M 1 loop /snap/core18/1988 loop1 7:1 0 69.9M 1 loop /snap/lxd/19188 loop2 7:2 0 55.5M 1 loop /snap/core18/1997 loop3 7:3 0 70.4M 1 loop /snap/lxd/19647 loop4 7:4 0 32.3M 1 loop /snap/snapd/11402 loop5 7:5 0 32.3M 1 loop /snap/snapd/11107 sda 8:0 0 80G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 79G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm / sdb 8:16 0 16G 0 disk └─sdb1 8:17 0 16G 0 part
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
b89--bcd4--105b2df26352 253:1 0 16G 0 lvm root@osd03:~#
root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime) root@osd03:/var/lib/ceph/osd/ceph-2#
any help is much appreciated --
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887)
--
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887) +989128610474 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I agree. But why does the process start manual without systemd which obviously has nothing to do with uid/gid 167 ? It is also not really a fix to let all users change uid/gids... On Wed, Apr 7, 2021 at 7:39 PM Wladimir Mutel <mwg@mwg.dp.ua> wrote:
Could there be more smooth migration? On my Ubuntu I have the same behavior and my ceph uid/gud are also 64045. I started with Luminous in 2018 when it was not containerized, and still continue updating it with apt. Since when we have got this hardcoded value of 167 ???
UID and guid should both be 167 I believe.
Make a note of the current values and change them to 167 using usermod and groupmod.
I had just this issue. It’s partly to do with how perms are used within
Andrew Walker-Brown wrote: the containers I think.
I changed the values to 167 in passwd everything worked again. Symptoms
for me were OSDs not starting and permissions/file not found errors.
Sent from my iPhone
On 4 Apr 2021, at 13:43, Lomayani S. Laizer <lomlaizer@gmail.com> wrote:
Hello, Permissions are correct. guid/uid is 64045/64045
ls -alh total 32K drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 . drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 .. lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/... -rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid -rw------- 1 ceph ceph 37 Apr 4 14:11 fsid -rw------- 1 ceph ceph 56 Apr 4 14:11 keyring -rw------- 1 ceph ceph 6 Apr 4 14:11 ready -rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release -rw------- 1 ceph ceph 10 Apr 4 14:11 type -rw------- 1 ceph ceph 3 Apr 4 14:11 whoami
On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown <
Are the file permissions correct and UID/guid in passwd both 167?
Sent from my iPhone
On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer@gmail.com<mailto: lomlaizer@gmail.com>> wrote:
Hello,
+1 Am facing the same problem in ubuntu after upgrade to pacific
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/ ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block: (1) Operation not permitted 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or directoryESC[0m
On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti < khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>> wrote:
It worth mentioning as I issue the following command, the Ceph OSD starts and joins the cluster: /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti < khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>> wrote:
Hi all,
As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the manual upgrade using the precompiled packages, the OSDs was down with
andrew_jbrown@hotmail.com<mailto:andrew_jbrown@hotmail.com>> wrote: the
following messages:
root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
--path /var/lib/ceph/osd/ceph-2 --no-mon-config Running command: /usr/bin/ln -snf
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
/var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/systemctl enable ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 Running command: /usr/bin/systemctl start ceph-osd@2 --> ceph-volume lvm activate successful for osd ID: 2
Content of /var/log/ceph/ceph-osd.2.log 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045 (ceph:ceph) 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-osd, pid 5484 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty --pid-file 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1* bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted* 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or directory*
root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2 â— ceph-osd@2.service - Ceph object storage daemon osd.2 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06 +0430; 50s ago Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS) Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) Main PID: 5484 (code=exited, status=1/FAILURE)
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Scheduled restart job, restart counter is at 3. Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Start request repeated too quickly. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Failed with result 'exit-code'. Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage daemon osd.2. root@osd03:/var/lib/ceph/osd/ceph-2#
root@osd03:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk loop0 7:0 0 55.5M 1 loop /snap/core18/1988 loop1 7:1 0 69.9M 1 loop /snap/lxd/19188 loop2 7:2 0 55.5M 1 loop /snap/core18/1997 loop3 7:3 0 70.4M 1 loop /snap/lxd/19647 loop4 7:4 0 32.3M 1 loop /snap/snapd/11402 loop5 7:5 0 32.3M 1 loop /snap/snapd/11107 sda 8:0 0 80G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 79G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm / sdb 8:16 0 16G 0 disk └─sdb1 8:17 0 16G 0 part
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
b89--bcd4--105b2df26352 253:1 0 16G 0 lvm root@osd03:~#
root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime) root@osd03:/var/lib/ceph/osd/ceph-2#
any help is much appreciated --
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887)
--
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887) +989128610474 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io<mailto: ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto: ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- UG (haftungsbeschränkt) * Enrico Kern, Geschäftsführer* Südstraße 26 01877 Demitz-Thumitz https://www.stackxperts.com enrico.kern@stackxperts.com +49 (0) 152 26814501 skype: flyersa Geschäftsführer Enrico Kern Amtsgericht Dresden | HRB 38717
I believe there is some of problem in the systemd as the ceph starts successfully by running manually using the ceph-osd command. On Thu, Apr 8, 2021, 10:32 AM Enrico Kern <enrico.kern@stackxperts.com> wrote:
I agree. But why does the process start manual without systemd which obviously has nothing to do with uid/gid 167 ? It is also not really a fix to let all users change uid/gids...
On Wed, Apr 7, 2021 at 7:39 PM Wladimir Mutel <mwg@mwg.dp.ua> wrote:
Could there be more smooth migration? On my Ubuntu I have the same behavior and my ceph uid/gud are also 64045. I started with Luminous in 2018 when it was not containerized, and still continue updating it with apt. Since when we have got this hardcoded value of 167 ???
UID and guid should both be 167 I believe.
Make a note of the current values and change them to 167 using usermod and groupmod.
I had just this issue. It’s partly to do with how perms are used within
Andrew Walker-Brown wrote: the containers I think.
I changed the values to 167 in passwd everything worked again. Symptoms
for me were OSDs not starting and permissions/file not found errors.
Sent from my iPhone
On 4 Apr 2021, at 13:43, Lomayani S. Laizer <lomlaizer@gmail.com>
wrote:
Hello, Permissions are correct. guid/uid is 64045/64045
ls -alh total 32K drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 . drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 .. lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/... -rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid -rw------- 1 ceph ceph 37 Apr 4 14:11 fsid -rw------- 1 ceph ceph 56 Apr 4 14:11 keyring -rw------- 1 ceph ceph 6 Apr 4 14:11 ready -rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release -rw------- 1 ceph ceph 10 Apr 4 14:11 type -rw------- 1 ceph ceph 3 Apr 4 14:11 whoami
On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown <
Are the file permissions correct and UID/guid in passwd both 167?
Sent from my iPhone
On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer@gmail.com <mailto: lomlaizer@gmail.com>> wrote:
Hello,
+1 Am facing the same problem in ubuntu after upgrade to pacific
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/ ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block: (1) Operation not permitted 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or directoryESC[0m
On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti < khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>> wrote:
It worth mentioning as I issue the following command, the Ceph OSD starts and joins the cluster: /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti < khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>> wrote:
Hi all,
As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the manual upgrade using the precompiled packages, the OSDs was down with
andrew_jbrown@hotmail.com<mailto:andrew_jbrown@hotmail.com>> wrote: the
following messages:
root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
--path /var/lib/ceph/osd/ceph-2 --no-mon-config Running command: /usr/bin/ln -snf
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
/var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/systemctl enable ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 Running command: /usr/bin/systemctl start ceph-osd@2 --> ceph-volume lvm activate successful for osd ID: 2
Content of /var/log/ceph/ceph-osd.2.log 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045 (ceph:ceph) 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-osd, pid 5484 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty --pid-file 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1* bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted* 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or directory*
root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2 â— ceph-osd@2.service - Ceph object storage daemon osd.2 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06 +0430; 50s ago Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS) Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) Main PID: 5484 (code=exited, status=1/FAILURE)
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Scheduled restart job, restart counter is at 3. Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Start request repeated too quickly. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Failed with result 'exit-code'. Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage daemon osd.2. root@osd03:/var/lib/ceph/osd/ceph-2#
root@osd03:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk loop0 7:0 0 55.5M 1 loop /snap/core18/1988 loop1 7:1 0 69.9M 1 loop /snap/lxd/19188 loop2 7:2 0 55.5M 1 loop /snap/core18/1997 loop3 7:3 0 70.4M 1 loop /snap/lxd/19647 loop4 7:4 0 32.3M 1 loop /snap/snapd/11402 loop5 7:5 0 32.3M 1 loop /snap/snapd/11107 sda 8:0 0 80G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 79G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm / sdb 8:16 0 16G 0 disk └─sdb1 8:17 0 16G 0 part
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
b89--bcd4--105b2df26352 253:1 0 16G 0 lvm root@osd03:~#
root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime) root@osd03:/var/lib/ceph/osd/ceph-2#
any help is much appreciated --
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887)
--
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887) +989128610474 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto: ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io<mailto: ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto: ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto: ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- UG (haftungsbeschränkt) * Enrico Kern, Geschäftsführer* Südstraße 26 01877 Demitz-Thumitz
https://www.stackxperts.com enrico.kern@stackxperts.com +49 (0) 152 26814501 skype: flyersa
Geschäftsführer Enrico Kern Amtsgericht Dresden | HRB 38717 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, Uncomment ProtectClock=true in /lib/systemd/system/ceph-osd@.service should fix the issue On Thu, Apr 8, 2021 at 9:49 AM Behzad Khoshbakhti <khoshbakhtib@gmail.com> wrote:
I believe there is some of problem in the systemd as the ceph starts successfully by running manually using the ceph-osd command.
On Thu, Apr 8, 2021, 10:32 AM Enrico Kern <enrico.kern@stackxperts.com> wrote:
I agree. But why does the process start manual without systemd which obviously has nothing to do with uid/gid 167 ? It is also not really a fix to let all users change uid/gids...
On Wed, Apr 7, 2021 at 7:39 PM Wladimir Mutel <mwg@mwg.dp.ua> wrote:
Could there be more smooth migration? On my Ubuntu I have the same behavior and my ceph uid/gud are also 64045. I started with Luminous in 2018 when it was not containerized, and still continue updating it with apt. Since when we have got this hardcoded value of 167 ???
UID and guid should both be 167 I believe.
Make a note of the current values and change them to 167 using usermod and groupmod.
I had just this issue. It’s partly to do with how perms are used within
Andrew Walker-Brown wrote: the containers I think.
I changed the values to 167 in passwd everything worked again.
Symptoms for me were OSDs not starting and permissions/file not found errors.
Sent from my iPhone
On 4 Apr 2021, at 13:43, Lomayani S. Laizer <lomlaizer@gmail.com>
wrote:
Hello, Permissions are correct. guid/uid is 64045/64045
ls -alh total 32K drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 . drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 .. lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/... -rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid -rw------- 1 ceph ceph 37 Apr 4 14:11 fsid -rw------- 1 ceph ceph 56 Apr 4 14:11 keyring -rw------- 1 ceph ceph 6 Apr 4 14:11 ready -rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release -rw------- 1 ceph ceph 10 Apr 4 14:11 type -rw------- 1 ceph ceph 3 Apr 4 14:11 whoami
On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown <
Are the file permissions correct and UID/guid in passwd both 167?
Sent from my iPhone
On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer@gmail.com <mailto: lomlaizer@gmail.com>> wrote:
Hello,
+1 Am facing the same problem in ubuntu after upgrade to pacific
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/ ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block: (1) Operation not permitted 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or directoryESC[0m
On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti < khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>> wrote:
It worth mentioning as I issue the following command, the Ceph OSD starts and joins the cluster: /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti < khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>> wrote:
Hi all,
As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during
manual upgrade using the precompiled packages, the OSDs was down with
andrew_jbrown@hotmail.com<mailto:andrew_jbrown@hotmail.com>> wrote: the the
following messages:
root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
--path /var/lib/ceph/osd/ceph-2 --no-mon-config Running command: /usr/bin/ln -snf
/var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/systemctl enable ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 Running command: /usr/bin/systemctl start ceph-osd@2 --> ceph-volume lvm activate successful for osd ID: 2
Content of /var/log/ceph/ceph-osd.2.log 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045 (ceph:ceph) 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable),
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 process
ceph-osd, pid 5484 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty --pid-file 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1* bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted* 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or directory*
root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2 â— ceph-osd@2.service - Ceph object storage daemon osd.2 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06 +0430; 50s ago Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS) Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) Main PID: 5484 (code=exited, status=1/FAILURE)
Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Scheduled restart job, restart counter is at 3. Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Start request repeated too quickly. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Failed with result 'exit-code'. Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage daemon osd.2. root@osd03:/var/lib/ceph/osd/ceph-2#
root@osd03:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk loop0 7:0 0 55.5M 1 loop /snap/core18/1988 loop1 7:1 0 69.9M 1 loop /snap/lxd/19188 loop2 7:2 0 55.5M 1 loop /snap/core18/1997 loop3 7:3 0 70.4M 1 loop /snap/lxd/19647 loop4 7:4 0 32.3M 1 loop /snap/snapd/11402 loop5 7:5 0 32.3M 1 loop /snap/snapd/11107 sda 8:0 0 80G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 79G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm / sdb 8:16 0 16G 0 disk └─sdb1 8:17 0 16G 0 part
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
b89--bcd4--105b2df26352 253:1 0 16G 0 lvm root@osd03:~#
root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime) root@osd03:/var/lib/ceph/osd/ceph-2#
any help is much appreciated --
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887)
--
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887) +989128610474 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto: ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io<mailto: ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto: ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto: ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- UG (haftungsbeschränkt) * Enrico Kern, Geschäftsführer* Südstraße 26 01877 Demitz-Thumitz
https://www.stackxperts.com enrico.kern@stackxperts.com +49 (0) 152 26814501 skype: flyersa
Geschäftsführer Enrico Kern Amtsgericht Dresden | HRB 38717 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
thanks by commenting the ProtectClock directive, the issue is resolved. Thanks for the support. On Sun, Apr 18, 2021 at 9:28 AM Lomayani S. Laizer <lomlaizer@gmail.com> wrote:
Hello,
Uncomment ProtectClock=true in /lib/systemd/system/ceph-osd@.service should fix the issue
On Thu, Apr 8, 2021 at 9:49 AM Behzad Khoshbakhti <khoshbakhtib@gmail.com> wrote:
I believe there is some of problem in the systemd as the ceph starts successfully by running manually using the ceph-osd command.
On Thu, Apr 8, 2021, 10:32 AM Enrico Kern <enrico.kern@stackxperts.com> wrote:
I agree. But why does the process start manual without systemd which obviously has nothing to do with uid/gid 167 ? It is also not really a fix to let all users change uid/gids...
On Wed, Apr 7, 2021 at 7:39 PM Wladimir Mutel <mwg@mwg.dp.ua> wrote:
Could there be more smooth migration? On my Ubuntu I have the same behavior and my ceph uid/gud are also 64045. I started with Luminous in 2018 when it was not containerized, and still continue updating it with apt. Since when we have got this hardcoded value of 167 ???
UID and guid should both be 167 I believe.
Make a note of the current values and change them to 167 using usermod and groupmod.
I had just this issue. It’s partly to do with how perms are used within
Andrew Walker-Brown wrote: the containers I think.
I changed the values to 167 in passwd everything worked again.
Symptoms for me were OSDs not starting and permissions/file not found errors.
Sent from my iPhone
On 4 Apr 2021, at 13:43, Lomayani S. Laizer <lomlaizer@gmail.com>
wrote:
Hello, Permissions are correct. guid/uid is 64045/64045
ls -alh total 32K drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 . drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 .. lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/... -rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid -rw------- 1 ceph ceph 37 Apr 4 14:11 fsid -rw------- 1 ceph ceph 56 Apr 4 14:11 keyring -rw------- 1 ceph ceph 6 Apr 4 14:11 ready -rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release -rw------- 1 ceph ceph 10 Apr 4 14:11 type -rw------- 1 ceph ceph 3 Apr 4 14:11 whoami
On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown <
Are the file permissions correct and UID/guid in passwd both 167?
Sent from my iPhone
On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer@gmail.com <mailto: lomlaizer@gmail.com>> wrote:
Hello,
+1 Am facing the same problem in ubuntu after upgrade to pacific
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/ ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block: (1) Operation not permitted 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or directoryESC[0m
On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti < khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>> wrote:
It worth mentioning as I issue the following command, the Ceph OSD starts and joins the cluster: /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti < khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>> wrote:
> Hi all, > > As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during
> manual upgrade using the precompiled packages, the OSDs was down with
andrew_jbrown@hotmail.com<mailto:andrew_jbrown@hotmail.com>> wrote: the the
> following messages: > > root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all > --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352 > Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 > Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir > --dev >
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
> --path /var/lib/ceph/osd/ceph-2 --no-mon-config > Running command: /usr/bin/ln -snf >
> /var/lib/ceph/osd/ceph-2/block > Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block > Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 > Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 > Running command: /usr/bin/systemctl enable > ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352 > Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 > Running command: /usr/bin/systemctl start ceph-osd@2 > --> ceph-volume lvm activate successful for osd ID: 2 > > Content of /var/log/ceph/ceph-osd.2.log > 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045 > (ceph:ceph) > 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0 > (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable),
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 process
> ceph-osd, pid 5484 > 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty > --pid-file > 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1* > bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open > /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted* > 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open > OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or directory* > > > root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2 > â— ceph-osd@2.service - Ceph object storage daemon osd.2 > Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; > vendor preset: enabled) > Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06 > +0430; 50s ago > Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh > --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS) > Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id > 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) > Main PID: 5484 (code=exited, status=1/FAILURE) > > Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Scheduled restart > job, restart counter is at 3. > Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2. > Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Start request > repeated too quickly. > Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Failed with result > 'exit-code'. > Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage > daemon osd.2. > root@osd03:/var/lib/ceph/osd/ceph-2# > > root@osd03:~# lsblk > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > fd0 2:0 1 4K 0 disk > loop0 7:0 0 55.5M 1 loop > /snap/core18/1988 > loop1 7:1 0 69.9M 1 loop > /snap/lxd/19188 > loop2 7:2 0 55.5M 1 loop > /snap/core18/1997 > loop3 7:3 0 70.4M 1 loop > /snap/lxd/19647 > loop4 7:4 0 32.3M 1 loop > /snap/snapd/11402 > loop5 7:5 0 32.3M 1 loop > /snap/snapd/11107 > sda 8:0 0 80G 0 disk > ├─sda1 8:1 0 1M 0 part > ├─sda2 8:2 0 1G 0 part /boot > └─sda3 8:3 0 79G 0 part > └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm / > sdb 8:16 0 16G 0 disk > └─sdb1 8:17 0 16G 0 part > >
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
> b89--bcd4--105b2df26352 > 253:1 0 16G 0 lvm > root@osd03:~# > > root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph > tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime) > root@osd03:/var/lib/ceph/osd/ceph-2# > > any help is much appreciated > -- > > Regards > Behzad Khoshbakhti > Computer Network Engineer (CCIE #58887) > >
--
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887) +989128610474 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto: ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io<mailto: ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto: ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto: ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- UG (haftungsbeschränkt) * Enrico Kern, Geschäftsführer* Südstraße 26 01877 Demitz-Thumitz
https://www.stackxperts.com enrico.kern@stackxperts.com +49 (0) 152 26814501 skype: flyersa
Geschäftsführer Enrico Kern Amtsgericht Dresden | HRB 38717 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887) +989128610474
Thanks for reporting this issue. It has been fixed by https://github.com/ceph/ceph/pull/40845 and will be released in the next pacific point release. Neha On Mon, Apr 19, 2021 at 8:19 AM Behzad Khoshbakhti <khoshbakhtib@gmail.com> wrote:
thanks by commenting the ProtectClock directive, the issue is resolved. Thanks for the support.
On Sun, Apr 18, 2021 at 9:28 AM Lomayani S. Laizer <lomlaizer@gmail.com> wrote:
Hello,
Uncomment ProtectClock=true in /lib/systemd/system/ceph-osd@.service should fix the issue
On Thu, Apr 8, 2021 at 9:49 AM Behzad Khoshbakhti <khoshbakhtib@gmail.com> wrote:
I believe there is some of problem in the systemd as the ceph starts successfully by running manually using the ceph-osd command.
On Thu, Apr 8, 2021, 10:32 AM Enrico Kern <enrico.kern@stackxperts.com> wrote:
I agree. But why does the process start manual without systemd which obviously has nothing to do with uid/gid 167 ? It is also not really a fix to let all users change uid/gids...
On Wed, Apr 7, 2021 at 7:39 PM Wladimir Mutel <mwg@mwg.dp.ua> wrote:
Could there be more smooth migration? On my Ubuntu I have the same behavior and my ceph uid/gud are also 64045. I started with Luminous in 2018 when it was not containerized, and still continue updating it with apt. Since when we have got this hardcoded value of 167 ???
UID and guid should both be 167 I believe.
Make a note of the current values and change them to 167 using usermod and groupmod.
I had just this issue. It’s partly to do with how perms are used within
Andrew Walker-Brown wrote: the containers I think.
I changed the values to 167 in passwd everything worked again.
Symptoms for me were OSDs not starting and permissions/file not found errors.
Sent from my iPhone
On 4 Apr 2021, at 13:43, Lomayani S. Laizer <lomlaizer@gmail.com>
wrote:
Hello, Permissions are correct. guid/uid is 64045/64045
ls -alh total 32K drwxrwxrwt 2 ceph ceph 200 Apr 4 14:11 . drwxr-xr-x 8 ceph ceph 4.0K Sep 18 2018 .. lrwxrwxrwx 1 ceph ceph 93 Apr 4 14:11 block -> /dev/... -rw------- 1 ceph ceph 37 Apr 4 14:11 ceph_fsid -rw------- 1 ceph ceph 37 Apr 4 14:11 fsid -rw------- 1 ceph ceph 56 Apr 4 14:11 keyring -rw------- 1 ceph ceph 6 Apr 4 14:11 ready -rw------- 1 ceph ceph 3 Apr 4 14:11 require_osd_release -rw------- 1 ceph ceph 10 Apr 4 14:11 type -rw------- 1 ceph ceph 3 Apr 4 14:11 whoami
On Sun, Apr 4, 2021 at 3:07 PM Andrew Walker-Brown <
Are the file permissions correct and UID/guid in passwd both 167?
Sent from my iPhone
On 4 Apr 2021, at 12:29, Lomayani S. Laizer <lomlaizer@gmail.com <mailto: lomlaizer@gmail.com>> wrote:
Hello,
+1 Am facing the same problem in ubuntu after upgrade to pacific
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/ ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block: (1) Operation not permitted 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or directoryESC[0m
On Sun, Apr 4, 2021 at 1:52 PM Behzad Khoshbakhti < khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>> wrote:
> It worth mentioning as I issue the following command, the Ceph OSD starts > and joins the cluster: > /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph > > > > On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti < khoshbakhtib@gmail.com<mailto:khoshbakhtib@gmail.com>> > wrote: > >> Hi all, >> >> As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during
>> manual upgrade using the precompiled packages, the OSDs was down with
andrew_jbrown@hotmail.com<mailto:andrew_jbrown@hotmail.com>> wrote: the the
>> following messages: >> >> root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all >> --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352 >> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 >> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph > prime-osd-dir >> --dev >> >
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352
>> --path /var/lib/ceph/osd/ceph-2 --no-mon-config >> Running command: /usr/bin/ln -snf >> >
>> /var/lib/ceph/osd/ceph-2/block >> Running command: /usr/bin/chown -h ceph:ceph > /var/lib/ceph/osd/ceph-2/block >> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 >> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 >> Running command: /usr/bin/systemctl enable >> ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352 >> Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 >> Running command: /usr/bin/systemctl start ceph-osd@2 >> --> ceph-volume lvm activate successful for osd ID: 2 >> >> Content of /var/log/ceph/ceph-osd.2.log >> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045 >> (ceph:ceph) >> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0 >> (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable),
/dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 process
>> ceph-osd, pid 5484 >> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty >> --pid-file >> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1* >> bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open >> /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted* >> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open >> OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or > directory* >> >> >> root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2 >> â— ceph-osd@2.service - Ceph object storage daemon osd.2 >> Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; >> vendor preset: enabled) >> Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06 >> +0430; 50s ago >> Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh >> --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS) >> Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} > --id >> 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) >> Main PID: 5484 (code=exited, status=1/FAILURE) >> >> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Scheduled restart >> job, restart counter is at 3. >> Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon > osd.2. >> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Start request >> repeated too quickly. >> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd@2.service: Failed with result >> 'exit-code'. >> Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage >> daemon osd.2. >> root@osd03:/var/lib/ceph/osd/ceph-2# >> >> root@osd03:~# lsblk >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT >> fd0 2:0 1 4K 0 disk >> loop0 7:0 0 55.5M 1 loop >> /snap/core18/1988 >> loop1 7:1 0 69.9M 1 loop >> /snap/lxd/19188 >> loop2 7:2 0 55.5M 1 loop >> /snap/core18/1997 >> loop3 7:3 0 70.4M 1 loop >> /snap/lxd/19647 >> loop4 7:4 0 32.3M 1 loop >> /snap/snapd/11402 >> loop5 7:5 0 32.3M 1 loop >> /snap/snapd/11107 >> sda 8:0 0 80G 0 disk >> ├─sda1 8:1 0 1M 0 part >> ├─sda2 8:2 0 1G 0 part /boot >> └─sda3 8:3 0 79G 0 part >> └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm / >> sdb 8:16 0 16G 0 disk >> └─sdb1 8:17 0 16G 0 part >> >> >
└─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4
>> b89--bcd4--105b2df26352 >> 253:1 0 16G 0 lvm >> root@osd03:~# >> >> root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph >> tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime) >> root@osd03:/var/lib/ceph/osd/ceph-2# >> >> any help is much appreciated >> -- >> >> Regards >> Behzad Khoshbakhti >> Computer Network Engineer (CCIE #58887) >> >> > > -- > > Regards > Behzad Khoshbakhti > Computer Network Engineer (CCIE #58887) > +989128610474 > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io<mailto: ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto: ceph-users-leave@ceph.io> > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto: ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto: ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- UG (haftungsbeschränkt) * Enrico Kern, Geschäftsführer* Südstraße 26 01877 Demitz-Thumitz
https://www.stackxperts.com enrico.kern@stackxperts.com +49 (0) 152 26814501 skype: flyersa
Geschäftsführer Enrico Kern Amtsgericht Dresden | HRB 38717 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887) +989128610474 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
参加者 (6)
-
Andrew Walker-Brown
-
Behzad Khoshbakhti
-
Enrico Kern
-
Lomayani S. Laizer
-
Neha Ojha
-
Wladimir Mutel