OK, found it. The second line in the error messages actually gives it away:
stderr: 2021-02-06 13:48:27.477 7f46756b4b80 -1 bdev(0x561db199c700
/var/lib/ceph/osd/ceph-342//block) _aio_start io_setup(2) failed with EAGAIN; try
increasing /proc/sys/fs/aio-max-nr
On my system, the default is rather small:
# sysctl fs.aio-max-nr
fs.aio-max-nr = 65536
Seemingly not a problem for ceph-disk OSDs:
# sysctl fs.aio-nr
fs.aio-nr = 32768
However, LVM OSDs seem to be quite hungry. Increasing the value to
sysctl -w fs.aio-max-nr=1048576
solved it for me.
Should have read it more carefully.
Best regards and a nice weekend.
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans(a)dtu.dk>
Sent: 06 February 2021 14:37:15
To: ceph-users(a)ceph.io
Cc: Dave Hall
Subject: [ceph-users] Re: ceph-volume bluestore _read_fsid unparsable uuid
I just noticed one difference between the two servers:
"Broken" server:
# lvm vgs
Failed to set up async io, using sync io.
VG #PV #LV #SN Attr VSize VFree
[listing follows]
"Good" server:
# lvm vgs
VG #PV #LV #SN Attr VSize VFree
[listing follows]
Could this play a role here?
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans(a)dtu.dk>
Sent: 06 February 2021 14:31:38
To: ceph-users(a)ceph.io
Cc: Dave Hall
Subject: [ceph-users] ceph-volume bluestore _read_fsid unparsable uuid
Hi Dave and everyone else affected,
I'm responding to a thread you opened on an issue with lvm OSD creation:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YYH3VANVV22…
https://tracker.ceph.com/issues/43868
Most important question: is there a workaround?
My observations: I'm running into the exact same issue on mimic 13.2.10. The strange
thing is, that some OSDs get created and others fail. I can't see a pattern here. I
have 1 host where every create worked out and another where half failed. The important
lines in the log are probably:
stderr: 2021-02-06 13:48:27.477 7f46756b4b80 -1
bluestore(/var/lib/ceph/osd/ceph-342/) _read_fsid unparsable uuid
stderr: 2021-02-06 13:48:27.477 7f46756b4b80 -1 bdev(0x561db199c700
/var/lib/ceph/osd/ceph-342//block) _aio_start io_setup(2) failed with EAGAIN; try
increasing /proc/sys/fs/aio-max-nr
stderr: 2021-02-06 13:48:27.477 7f46756b4b80 -1
bluestore(/var/lib/ceph/osd/ceph-342/) mkfs failed, (11) Resource temporarily unavailable
stderr: 2021-02-06 13:48:27.477 7f46756b4b80 -1 OSD::mkfs: ObjectStore::mkfs failed
with error (11) Resource temporarily unavailable
stderr: 2021-02-06 13:48:27.477 7f46756b4b80 -1 [0;31m ** ERROR: error creating empty
object store in /var/lib/ceph/osd/ceph-342/: (11) Resource temporarily unavailable[0m
I really need to get a decent number of disks up very soon. Any help is appreciated. I can
provide more output if that helps.
Best regards and good weekend!
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io