Hello, Some time ago I deployed a ceph cluster. It works great. Today I collect some statistics and found that the BlueFs utility is not working. # ceph-bluestore-tool bluefs-bdev-sizes --path /var/lib/ceph/osd/ceph-8 inferring bluefs devices from bluestore path slot 1 /var/lib/ceph/osd/ceph-8/block -> /dev/dm-5 unable to open /var/lib/ceph/osd/ceph-8/block: (11) Resource temporarily unavailable 2019-11-11 15:03:30.665 7f4b9a427f00 -1 bdev(0x55d5b0310a80 /var/lib/ceph/osd/ceph-8/block) _lock flock failed on /var/lib/ceph/osd/ceph-8/block 2019-11-11 15:03:30.665 7f4b9a427f00 -1 bdev(0x55d5b0310a80 /var/lib/ceph/osd/ceph-8/block) open failed to lock /var/lib/ceph/osd/ceph-8/block: (11) Resource temporarily unavailable As far as I understand, blocks.db and blocks.wal are missing. I don’t know how it turned out. # ls -l /var/lib/ceph/osd/ceph-8 total 28 lrwxrwxrwx 1 ceph ceph 93 Nov 9 15:56 block -> /dev/ceph-55b8a53d-1740-402a-b6f4-09d4befdd564/osd-block-c5488db7-621a-490a-88a0-904c12e8b8ed -rw------- 1 ceph ceph 37 Nov 9 15:56 ceph_fsid -rw------- 1 ceph ceph 37 Nov 9 15:56 fsid -rw------- 1 ceph ceph 55 Nov 9 15:56 keyring -rw------- 1 ceph ceph 6 Nov 9 15:56 ready -rw-r--r-- 1 ceph ceph 3 Nov 9 15:56 require_osd_release -rw------- 1 ceph ceph 10 Nov 9 15:56 type -rw------- 1 ceph ceph 2 Nov 9 15:56 whoami Did as standard: .... ceph-deploy osd create test-host1:/dev/sde ceph-deploy osd create test-host2:/dev/sde .... What now to do and whether it is necessary to repair it at all? Assembled on luminous, yesterday updated to Nautilus.
Hi Andrey, this log output rather looks like some other process is using /var/lib/ceph/osd/ceph-8 Have you stopped OSD.8 daemon? And are you sure you deployed standalone DB/WAL devices for this OSD? Thanks, Igor On 11/11/2019 3:10 PM, Andrey Groshev wrote:
Hello,
Some time ago I deployed a ceph cluster. It works great. Today I collect some statistics and found that the BlueFs utility is not working.
# ceph-bluestore-tool bluefs-bdev-sizes --path /var/lib/ceph/osd/ceph-8 inferring bluefs devices from bluestore path slot 1 /var/lib/ceph/osd/ceph-8/block -> /dev/dm-5 unable to open /var/lib/ceph/osd/ceph-8/block: (11) Resource temporarily unavailable 2019-11-11 15:03:30.665 7f4b9a427f00 -1 bdev(0x55d5b0310a80 /var/lib/ceph/osd/ceph-8/block) _lock flock failed on /var/lib/ceph/osd/ceph-8/block 2019-11-11 15:03:30.665 7f4b9a427f00 -1 bdev(0x55d5b0310a80 /var/lib/ceph/osd/ceph-8/block) open failed to lock /var/lib/ceph/osd/ceph-8/block: (11) Resource temporarily unavailable
As far as I understand, blocks.db and blocks.wal are missing. I don’t know how it turned out.
# ls -l /var/lib/ceph/osd/ceph-8 total 28 lrwxrwxrwx 1 ceph ceph 93 Nov 9 15:56 block -> /dev/ceph-55b8a53d-1740-402a-b6f4-09d4befdd564/osd-block-c5488db7-621a-490a-88a0-904c12e8b8ed -rw------- 1 ceph ceph 37 Nov 9 15:56 ceph_fsid -rw------- 1 ceph ceph 37 Nov 9 15:56 fsid -rw------- 1 ceph ceph 55 Nov 9 15:56 keyring -rw------- 1 ceph ceph 6 Nov 9 15:56 ready -rw-r--r-- 1 ceph ceph 3 Nov 9 15:56 require_osd_release -rw------- 1 ceph ceph 10 Nov 9 15:56 type -rw------- 1 ceph ceph 2 Nov 9 15:56 whoami
Did as standard: .... ceph-deploy osd create test-host1:/dev/sde ceph-deploy osd create test-host2:/dev/sde ....
What now to do and whether it is necessary to repair it at all? Assembled on luminous, yesterday updated to Nautilus.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Igor! Service is UP. I did not make separate devices. blocks.db and blocks.wal are created only if they are on separate devices? # systemctl status ceph-osd@8.service ● ceph-osd@8.service - Ceph object storage daemon osd.8 Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: disabled) Active: active (running) since Sat 2019-11-09 15:56:28 MSK; 1 day 23h ago Process: 2676 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i (code=exited, status=0/SUCCESS) Main PID: 2707 (ceph-osd) CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@8.service └─2707 /usr/bin/ceph-osd -f --cluster ceph --id 8 --setuser ceph --setgroup ceph Nov 09 15:56:28 test-host5 systemd[1]: Starting Ceph object storage daemon osd.8... Nov 09 15:56:28 test-host5 systemd[1]: Started Ceph object storage daemon osd.8. 11.11.2019, 15:18, "Igor Fedotov" <ifedotov@suse.de>:
Hi Andrey,
this log output rather looks like some other process is using /var/lib/ceph/osd/ceph-8
Have you stopped OSD.8 daemon?
And are you sure you deployed standalone DB/WAL devices for this OSD?
Thanks,
Igor
On 11/11/2019 3:10 PM, Andrey Groshev wrote:
Hello,
Some time ago I deployed a ceph cluster. It works great. Today I collect some statistics and found that the BlueFs utility is not working.
# ceph-bluestore-tool bluefs-bdev-sizes --path /var/lib/ceph/osd/ceph-8 inferring bluefs devices from bluestore path slot 1 /var/lib/ceph/osd/ceph-8/block -> /dev/dm-5 unable to open /var/lib/ceph/osd/ceph-8/block: (11) Resource temporarily unavailable 2019-11-11 15:03:30.665 7f4b9a427f00 -1 bdev(0x55d5b0310a80 /var/lib/ceph/osd/ceph-8/block) _lock flock failed on /var/lib/ceph/osd/ceph-8/block 2019-11-11 15:03:30.665 7f4b9a427f00 -1 bdev(0x55d5b0310a80 /var/lib/ceph/osd/ceph-8/block) open failed to lock /var/lib/ceph/osd/ceph-8/block: (11) Resource temporarily unavailable
As far as I understand, blocks.db and blocks.wal are missing. I don’t know how it turned out.
# ls -l /var/lib/ceph/osd/ceph-8 total 28 lrwxrwxrwx 1 ceph ceph 93 Nov 9 15:56 block -> /dev/ceph-55b8a53d-1740-402a-b6f4-09d4befdd564/osd-block-c5488db7-621a-490a-88a0-904c12e8b8ed -rw------- 1 ceph ceph 37 Nov 9 15:56 ceph_fsid -rw------- 1 ceph ceph 37 Nov 9 15:56 fsid -rw------- 1 ceph ceph 55 Nov 9 15:56 keyring -rw------- 1 ceph ceph 6 Nov 9 15:56 ready -rw-r--r-- 1 ceph ceph 3 Nov 9 15:56 require_osd_release -rw------- 1 ceph ceph 10 Nov 9 15:56 type -rw------- 1 ceph ceph 2 Nov 9 15:56 whoami
Did as standard: .... ceph-deploy osd create test-host1:/dev/sde ceph-deploy osd create test-host2:/dev/sde ....
What now to do and whether it is necessary to repair it at all? Assembled on luminous, yesterday updated to Nautilus.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 11/11/2019 3:51 PM, Andrey Groshev wrote:
Hi, Igor! Service is UP.
This prevents ceph-bluestore-tool from starting, you should shutdown it first.
I did not make separate devices. blocks.db and blocks.wal are created only if they are on separate devices? in general - yes. They make sense when using separate devices only...
# systemctl status ceph-osd@8.service ● ceph-osd@8.service - Ceph object storage daemon osd.8 Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: disabled) Active: active (running) since Sat 2019-11-09 15:56:28 MSK; 1 day 23h ago Process: 2676 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i (code=exited, status=0/SUCCESS) Main PID: 2707 (ceph-osd) CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@8.service └─2707 /usr/bin/ceph-osd -f --cluster ceph --id 8 --setuser ceph --setgroup ceph
Nov 09 15:56:28 test-host5 systemd[1]: Starting Ceph object storage daemon osd.8... Nov 09 15:56:28 test-host5 systemd[1]: Started Ceph object storage daemon osd.8.
11.11.2019, 15:18, "Igor Fedotov" <ifedotov@suse.de>:
Hi Andrey,
this log output rather looks like some other process is using /var/lib/ceph/osd/ceph-8
Have you stopped OSD.8 daemon?
And are you sure you deployed standalone DB/WAL devices for this OSD?
Thanks,
Igor
On 11/11/2019 3:10 PM, Andrey Groshev wrote:
Hello,
Some time ago I deployed a ceph cluster. It works great. Today I collect some statistics and found that the BlueFs utility is not working.
# ceph-bluestore-tool bluefs-bdev-sizes --path /var/lib/ceph/osd/ceph-8 inferring bluefs devices from bluestore path slot 1 /var/lib/ceph/osd/ceph-8/block -> /dev/dm-5 unable to open /var/lib/ceph/osd/ceph-8/block: (11) Resource temporarily unavailable 2019-11-11 15:03:30.665 7f4b9a427f00 -1 bdev(0x55d5b0310a80 /var/lib/ceph/osd/ceph-8/block) _lock flock failed on /var/lib/ceph/osd/ceph-8/block 2019-11-11 15:03:30.665 7f4b9a427f00 -1 bdev(0x55d5b0310a80 /var/lib/ceph/osd/ceph-8/block) open failed to lock /var/lib/ceph/osd/ceph-8/block: (11) Resource temporarily unavailable
As far as I understand, blocks.db and blocks.wal are missing. I don’t know how it turned out.
# ls -l /var/lib/ceph/osd/ceph-8 total 28 lrwxrwxrwx 1 ceph ceph 93 Nov 9 15:56 block -> /dev/ceph-55b8a53d-1740-402a-b6f4-09d4befdd564/osd-block-c5488db7-621a-490a-88a0-904c12e8b8ed -rw------- 1 ceph ceph 37 Nov 9 15:56 ceph_fsid -rw------- 1 ceph ceph 37 Nov 9 15:56 fsid -rw------- 1 ceph ceph 55 Nov 9 15:56 keyring -rw------- 1 ceph ceph 6 Nov 9 15:56 ready -rw-r--r-- 1 ceph ceph 3 Nov 9 15:56 require_osd_release -rw------- 1 ceph ceph 10 Nov 9 15:56 type -rw------- 1 ceph ceph 2 Nov 9 15:56 whoami
Did as standard: .... ceph-deploy osd create test-host1:/dev/sde ceph-deploy osd create test-host2:/dev/sde ....
What now to do and whether it is necessary to repair it at all? Assembled on luminous, yesterday updated to Nautilus.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ok. Thanks. This is how it works. I did not think that with a working service it would not work. 11.11.2019, 15:54, "Igor Fedotov" <ifedotov@suse.de>:
On 11/11/2019 3:51 PM, Andrey Groshev wrote:
Hi, Igor! Service is UP.
This prevents ceph-bluestore-tool from starting, you should shutdown it first.
I did not make separate devices. blocks.db and blocks.wal are created only if they are on separate devices?
in general - yes. They make sense when using separate devices only...
# systemctl status ceph-osd@8.service ● ceph-osd@8.service - Ceph object storage daemon osd.8 Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: disabled) Active: active (running) since Sat 2019-11-09 15:56:28 MSK; 1 day 23h ago Process: 2676 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i (code=exited, status=0/SUCCESS) Main PID: 2707 (ceph-osd) CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@8.service └─2707 /usr/bin/ceph-osd -f --cluster ceph --id 8 --setuser ceph --setgroup ceph
Nov 09 15:56:28 test-host5 systemd[1]: Starting Ceph object storage daemon osd.8... Nov 09 15:56:28 test-host5 systemd[1]: Started Ceph object storage daemon osd.8.
11.11.2019, 15:18, "Igor Fedotov" <ifedotov@suse.de>:
Hi Andrey,
this log output rather looks like some other process is using /var/lib/ceph/osd/ceph-8
Have you stopped OSD.8 daemon?
And are you sure you deployed standalone DB/WAL devices for this OSD?
Thanks,
Igor
On 11/11/2019 3:10 PM, Andrey Groshev wrote:
Hello,
Some time ago I deployed a ceph cluster. It works great. Today I collect some statistics and found that the BlueFs utility is not working.
# ceph-bluestore-tool bluefs-bdev-sizes --path /var/lib/ceph/osd/ceph-8 inferring bluefs devices from bluestore path slot 1 /var/lib/ceph/osd/ceph-8/block -> /dev/dm-5 unable to open /var/lib/ceph/osd/ceph-8/block: (11) Resource temporarily unavailable 2019-11-11 15:03:30.665 7f4b9a427f00 -1 bdev(0x55d5b0310a80 /var/lib/ceph/osd/ceph-8/block) _lock flock failed on /var/lib/ceph/osd/ceph-8/block 2019-11-11 15:03:30.665 7f4b9a427f00 -1 bdev(0x55d5b0310a80 /var/lib/ceph/osd/ceph-8/block) open failed to lock /var/lib/ceph/osd/ceph-8/block: (11) Resource temporarily unavailable
As far as I understand, blocks.db and blocks.wal are missing. I don’t know how it turned out.
# ls -l /var/lib/ceph/osd/ceph-8 total 28 lrwxrwxrwx 1 ceph ceph 93 Nov 9 15:56 block -> /dev/ceph-55b8a53d-1740-402a-b6f4-09d4befdd564/osd-block-c5488db7-621a-490a-88a0-904c12e8b8ed -rw------- 1 ceph ceph 37 Nov 9 15:56 ceph_fsid -rw------- 1 ceph ceph 37 Nov 9 15:56 fsid -rw------- 1 ceph ceph 55 Nov 9 15:56 keyring -rw------- 1 ceph ceph 6 Nov 9 15:56 ready -rw-r--r-- 1 ceph ceph 3 Nov 9 15:56 require_osd_release -rw------- 1 ceph ceph 10 Nov 9 15:56 type -rw------- 1 ceph ceph 2 Nov 9 15:56 whoami
Did as standard: .... ceph-deploy osd create test-host1:/dev/sde ceph-deploy osd create test-host2:/dev/sde ....
What now to do and whether it is necessary to repair it at all? Assembled on luminous, yesterday updated to Nautilus.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Andrey Groshev
-
Igor Fedotov