flock is held after ceph-osd daemon being stopped
Hi All, I noticed a locking issue in kernel device. When I stopped the ceph cluster and all daemons, the kernel device _lock somehow is still held and this line below will return r < 0: int KernelDevice::_lock() { int r = ::flock(fd_directs[WRITE_LIFE_NOT_SET], LOCK_EX | LOCK_NB); … } The way I stop the cluster and daemons: sudo ../src/stop.sh sudo bin/init-ceph --verbose forcestop This error happens even after the reboot when I try to use vstart: bdev _lock flock failed on ceph/build/dev/osd0/block bdev open failed to lock /home/yzhan298/ceph/build/dev/osd0/block: (11) Resource temporarily unavailable OSD::mkfs: couldn't mount ObjectStore: error (11) Resource temporarily unavailable ** ERROR: error creating empty object store in ceph/build/dev/osd0: (11) Resource temporarily unavailable Please advice. (On master branch) Thanks, Yiming
Talib was chasing down a similar issue a while back and found that the root cause was systemd-udev, which spawns a process that opens the device after it is closed. You might try removing or disabling that package and see if it goes away? On Wed, 12 Feb 2020, Yiming Zhang wrote:
Hi All,
I noticed a locking issue in kernel device. When I stopped the ceph cluster and all daemons, the kernel device _lock somehow is still held and this line below will return r < 0:
int KernelDevice::_lock() { int r = ::flock(fd_directs[WRITE_LIFE_NOT_SET], LOCK_EX | LOCK_NB); … }
The way I stop the cluster and daemons:
sudo ../src/stop.sh sudo bin/init-ceph --verbose forcestop
This error happens even after the reboot when I try to use vstart:
bdev _lock flock failed on ceph/build/dev/osd0/block bdev open failed to lock /home/yzhan298/ceph/build/dev/osd0/block: (11) Resource temporarily unavailable OSD::mkfs: couldn't mount ObjectStore: error (11) Resource temporarily unavailable ** ERROR: error creating empty object store in ceph/build/dev/osd0: (11) Resource temporarily unavailable
Please advice. (On master branch)
Thanks, Yiming _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
The weird thing is I don’t have systemd-udev installed on my server. Is there any other possible solutions? The error only happens when I redirect osd data to a raw device. Thanks, Yiming
On Feb 12, 2020, at 8:36 AM, Sage Weil <sage@newdream.net> wrote:
Talib was chasing down a similar issue a while back and found that the root cause was systemd-udev, which spawns a process that opens the device after it is closed. You might try removing or disabling that package and see if it goes away?
On Wed, 12 Feb 2020, Yiming Zhang wrote:
Hi All,
I noticed a locking issue in kernel device. When I stopped the ceph cluster and all daemons, the kernel device _lock somehow is still held and this line below will return r < 0:
int KernelDevice::_lock() { int r = ::flock(fd_directs[WRITE_LIFE_NOT_SET], LOCK_EX | LOCK_NB); … }
The way I stop the cluster and daemons:
sudo ../src/stop.sh sudo bin/init-ceph --verbose forcestop
This error happens even after the reboot when I try to use vstart:
bdev _lock flock failed on ceph/build/dev/osd0/block bdev open failed to lock /home/yzhan298/ceph/build/dev/osd0/block: (11) Resource temporarily unavailable OSD::mkfs: couldn't mount ObjectStore: error (11) Resource temporarily unavailable ** ERROR: error creating empty object store in ceph/build/dev/osd0: (11) Resource temporarily unavailable
Please advice. (On master branch)
Thanks, Yiming _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
If the OSD daemon dies, then it will have closed all of its fd's and there should be no more lock. Therefore you almost certainly have some other process running that is holding the lock. You may have to do a bit of digging in /proc/locks. Determine the dev+inode number of the file on which the lock is being set and find it in /proc/locks. Then you can track down the PID that's holding that lock. Cheers, Jeff On Wed, 2020-02-12 at 09:03 -0800, Yiming Zhang wrote:
The weird thing is I don’t have systemd-udev installed on my server. Is there any other possible solutions?
The error only happens when I redirect osd data to a raw device.
Thanks, Yiming
On Feb 12, 2020, at 8:36 AM, Sage Weil <sage@newdream.net> wrote:
Talib was chasing down a similar issue a while back and found that the root cause was systemd-udev, which spawns a process that opens the device after it is closed. You might try removing or disabling that package and see if it goes away?
On Wed, 12 Feb 2020, Yiming Zhang wrote:
Hi All,
I noticed a locking issue in kernel device. When I stopped the ceph cluster and all daemons, the kernel device _lock somehow is still held and this line below will return r < 0:
int KernelDevice::_lock() { int r = ::flock(fd_directs[WRITE_LIFE_NOT_SET], LOCK_EX | LOCK_NB); … }
The way I stop the cluster and daemons:
sudo ../src/stop.sh sudo bin/init-ceph --verbose forcestop
This error happens even after the reboot when I try to use vstart:
bdev _lock flock failed on ceph/build/dev/osd0/block bdev open failed to lock /home/yzhan298/ceph/build/dev/osd0/block: (11) Resource temporarily unavailable OSD::mkfs: couldn't mount ObjectStore: error (11) Resource temporarily unavailable ** ERROR: error creating empty object store in ceph/build/dev/osd0: (11) Resource temporarily unavailable
Please advice. (On master branch)
Thanks, Yiming _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jeff Layton <jlayton@redhat.com>
On Feb 13, 2020, at 3:52 AM, Jeff Layton <jlayton@redhat.com> wrote:
If the OSD daemon dies, then it will have closed all of its fd's and there should be no more lock. Therefore you almost certainly have some other process running that is holding the lock.
You may have to do a bit of digging in /proc/locks. Determine the dev+inode number of the file on which the lock is being set and find it in /proc/locks. Then you can track down the PID that's holding that lock.
I have checked the locks with lslocks, here is the locks when I vstarted ceph (bluestore block = /dev/sdc where sdc is a raw device): COMMAND PID TYPE SIZE MODE M START END PATH ceph-mgr 19852 POSIX WRITE 0 0 0 /... iscsid 1061 POSIX WRITE 0 0 0 /run... ceph-mgr 14889 POSIX WRITE 0 0 0 /... rpcbind 990 FLOCK WRITE 0 0 0 /run... ceph-mon 16430 POSIX WRITE 0 0 0 /... ceph-mon 16430 POSIX WRITE 0 0 0 /... ceph-mon 18107 POSIX WRITE 0 0 0 /... ceph-mon 18107 POSIX WRITE 0 0 0 /... ceph-mon 19711 POSIX WRITE 0 0 0 /... ceph-mon 19711 POSIX WRITE 0 0 0 /... ceph-mon 10495 POSIX WRITE 0 0 0 /... ceph-mon 10495 POSIX WRITE 0 0 0 /... ceph-mon 14748 POSIX WRITE 0 0 0 /... ceph-mon 14748 POSIX WRITE 0 0 0 /... cron 1085 FLOCK WRITE 0 0 0 /run... ceph-mgr 18247 POSIX WRITE 0 0 0 /... atd 1111 POSIX WRITE 0 0 0 /run... lvmetad 807 POSIX WRITE 0 0 0 /run... ceph-mgr 10635 POSIX WRITE 0 0 0 /... ceph-mgr 16571 POSIX WRITE 0 0 0 /… Then I kill all related processes and restart cluster, the error “_lock flock failed on /users/xxx/ceph/build/dev/osd0/block” persists. After the kill, locks are: COMMAND PID TYPE SIZE MODE M START END PATH rpcbind 20267 FLOCK WRITE 0 0 0 /run... lvmetad 20266 POSIX WRITE 0 0 0 /run… The error happens in KernelDevice.cc: int r = ::flock(fd_directs[WRITE_LIFE_NOT_SET], LOCK_EX | LOCK_NB); Where r gives -1, and fd_directs[WRITE_LIFE_NOT_SET] will give 11, and WRITE_LIFE_NOT_SET is 0. Any suggestions how to proceed with the issue? Thanks, -ym
Cheers, Jeff
On Wed, 2020-02-12 at 09:03 -0800, Yiming Zhang wrote:
The weird thing is I don’t have systemd-udev installed on my server. Is there any other possible solutions?
The error only happens when I redirect osd data to a raw device.
Thanks, Yiming
On Feb 12, 2020, at 8:36 AM, Sage Weil <sage@newdream.net> wrote:
Talib was chasing down a similar issue a while back and found that the root cause was systemd-udev, which spawns a process that opens the device after it is closed. You might try removing or disabling that package and see if it goes away?
On Wed, 12 Feb 2020, Yiming Zhang wrote:
Hi All,
I noticed a locking issue in kernel device. When I stopped the ceph cluster and all daemons, the kernel device _lock somehow is still held and this line below will return r < 0:
int KernelDevice::_lock() { int r = ::flock(fd_directs[WRITE_LIFE_NOT_SET], LOCK_EX | LOCK_NB); … }
The way I stop the cluster and daemons:
sudo ../src/stop.sh sudo bin/init-ceph --verbose forcestop
This error happens even after the reboot when I try to use vstart:
bdev _lock flock failed on ceph/build/dev/osd0/block bdev open failed to lock /home/yzhan298/ceph/build/dev/osd0/block: (11) Resource temporarily unavailable OSD::mkfs: couldn't mount ObjectStore: error (11) Resource temporarily unavailable ** ERROR: error creating empty object store in ceph/build/dev/osd0: (11) Resource temporarily unavailable
Please advice. (On master branch)
Thanks, Yiming _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jeff Layton <jlayton@redhat.com>
On Fri, 2020-02-14 at 07:13 -0800, Yiming Zhang wrote:
On Feb 13, 2020, at 3:52 AM, Jeff Layton <jlayton@redhat.com> wrote:
If the OSD daemon dies, then it will have closed all of its fd's and there should be no more lock. Therefore you almost certainly have some other process running that is holding the lock.
You may have to do a bit of digging in /proc/locks. Determine the dev+inode number of the file on which the lock is being set and find it in /proc/locks. Then you can track down the PID that's holding that lock.
I have checked the locks with lslocks, here is the locks when I vstarted ceph (bluestore block = /dev/sdc where sdc is a raw device): COMMAND PID TYPE SIZE MODE M START END PATH ceph-mgr 19852 POSIX WRITE 0 0 0 /... iscsid 1061 POSIX WRITE 0 0 0 /run... ceph-mgr 14889 POSIX WRITE 0 0 0 /... rpcbind 990 FLOCK WRITE 0 0 0 /run... ceph-mon 16430 POSIX WRITE 0 0 0 /... ceph-mon 16430 POSIX WRITE 0 0 0 /... ceph-mon 18107 POSIX WRITE 0 0 0 /... ceph-mon 18107 POSIX WRITE 0 0 0 /... ceph-mon 19711 POSIX WRITE 0 0 0 /... ceph-mon 19711 POSIX WRITE 0 0 0 /... ceph-mon 10495 POSIX WRITE 0 0 0 /... ceph-mon 10495 POSIX WRITE 0 0 0 /... ceph-mon 14748 POSIX WRITE 0 0 0 /... ceph-mon 14748 POSIX WRITE 0 0 0 /... cron 1085 FLOCK WRITE 0 0 0 /run... ceph-mgr 18247 POSIX WRITE 0 0 0 /... atd 1111 POSIX WRITE 0 0 0 /run... lvmetad 807 POSIX WRITE 0 0 0 /run... ceph-mgr 10635 POSIX WRITE 0 0 0 /... ceph-mgr 16571 POSIX WRITE 0 0 0 /…
Then I kill all related processes and restart cluster, the error “_lock flock failed on /users/xxx/ceph/build/dev/osd0/block” persists.
After the kill, locks are: COMMAND PID TYPE SIZE MODE M START END PATH rpcbind 20267 FLOCK WRITE 0 0 0 /run... lvmetad 20266 POSIX WRITE 0 0 0 /run…
The error happens in KernelDevice.cc: int r = ::flock(fd_directs[WRITE_LIFE_NOT_SET], LOCK_EX | LOCK_NB); Where r gives -1, and fd_directs[WRITE_LIFE_NOT_SET] will give 11, and WRITE_LIFE_NOT_SET is 0.
Any suggestions how to proceed with the issue?
Sorry, no. Any lock set on a block device should show up in /proc/locks (as it uses the kernel's generic flock lock mechanism for local filesystems). You may want to play with strace and verify that the error is coming from the kernel and that the program is attempting to set the lock on the file you think it is. What kernel is this running on? -- Jeff Layton <jlayton@redhat.com>
On Feb 18, 2020, at 4:11 AM, Jeff Layton <jlayton@redhat.com> wrote:
On Fri, 2020-02-14 at 07:13 -0800, Yiming Zhang wrote:
On Feb 13, 2020, at 3:52 AM, Jeff Layton <jlayton@redhat.com> wrote:
If the OSD daemon dies, then it will have closed all of its fd's and there should be no more lock. Therefore you almost certainly have some other process running that is holding the lock.
You may have to do a bit of digging in /proc/locks. Determine the dev+inode number of the file on which the lock is being set and find it in /proc/locks. Then you can track down the PID that's holding that lock.
I have checked the locks with lslocks, here is the locks when I vstarted ceph (bluestore block = /dev/sdc where sdc is a raw device): COMMAND PID TYPE SIZE MODE M START END PATH ceph-mgr 19852 POSIX WRITE 0 0 0 /... iscsid 1061 POSIX WRITE 0 0 0 /run... ceph-mgr 14889 POSIX WRITE 0 0 0 /... rpcbind 990 FLOCK WRITE 0 0 0 /run... ceph-mon 16430 POSIX WRITE 0 0 0 /... ceph-mon 16430 POSIX WRITE 0 0 0 /... ceph-mon 18107 POSIX WRITE 0 0 0 /... ceph-mon 18107 POSIX WRITE 0 0 0 /... ceph-mon 19711 POSIX WRITE 0 0 0 /... ceph-mon 19711 POSIX WRITE 0 0 0 /... ceph-mon 10495 POSIX WRITE 0 0 0 /... ceph-mon 10495 POSIX WRITE 0 0 0 /... ceph-mon 14748 POSIX WRITE 0 0 0 /... ceph-mon 14748 POSIX WRITE 0 0 0 /... cron 1085 FLOCK WRITE 0 0 0 /run... ceph-mgr 18247 POSIX WRITE 0 0 0 /... atd 1111 POSIX WRITE 0 0 0 /run... lvmetad 807 POSIX WRITE 0 0 0 /run... ceph-mgr 10635 POSIX WRITE 0 0 0 /... ceph-mgr 16571 POSIX WRITE 0 0 0 /…
Then I kill all related processes and restart cluster, the error “_lock flock failed on /users/xxx/ceph/build/dev/osd0/block” persists.
After the kill, locks are: COMMAND PID TYPE SIZE MODE M START END PATH rpcbind 20267 FLOCK WRITE 0 0 0 /run... lvmetad 20266 POSIX WRITE 0 0 0 /run…
The error happens in KernelDevice.cc: int r = ::flock(fd_directs[WRITE_LIFE_NOT_SET], LOCK_EX | LOCK_NB); Where r gives -1, and fd_directs[WRITE_LIFE_NOT_SET] will give 11, and WRITE_LIFE_NOT_SET is 0.
Any suggestions how to proceed with the issue?
Sorry, no. Any lock set on a block device should show up in /proc/locks (as it uses the kernel's generic flock lock mechanism for local filesystems).
You may want to play with strace and verify that the error is coming from the kernel and that the program is attempting to set the lock on the file you think it is.
What kernel is this running on?
The kernel is 4.15.0-70-generic( I also has the same issue on another kernel 4.15.18-041518-generic). I used the strace to track the issue, and it led to this paticular function _lock in KernelDevice (`r = _lock();` in KernelDevice::open function). If I commented it out, the error goest away. But it’s not a fix. Maybe there is a bug here. I’ll keep digging this. Thanks, -ym
-- Jeff Layton <jlayton@redhat.com>
participants (3)
-
Jeff Layton
-
Sage Weil
-
Yiming Zhang