I have a cluster (14.2.11) with bluestore osds that have the block.DB on an SSD partition that is separate from the primary osd device.  On some of my storage servers, the osd processes fail to start up at boot time because the permissions on the block.db device are not being changed from root:root, or are being reset by udev after the ceph-volume-systemd has run successfully.  The problem only occurs on a couple of the storage servers, though all of them are configured the same and are running the same software versions.

I suspect a race condition or conflict with the udev rules, but I have not been successful in identifying where the problem lies and udev is a complete nightmare to debug and diagnose.

One workaround solution is to update /usr/lib/ceph/ceph-osd-prestart.sh so that it checks (and corrects) the permissions on the block.db device so that the osd can start correctly.  This particular script looks like is hasn't been updated to support bluestore, so I added a some lines to address the problem, which works for me.

Has anyone else seen a similar issue and found a different solution? 

Here is the code I added to the ceph-osd-prestart.sh script:

...

if [ -L "$blockdb" -a -e "$blockdb" ]; then

    dev_db=`readlink -f $blockdb`

    owner=`stat -c %U $dev_db`

    if [ $owner != 'ceph' ]; then

        echo "ceph-osd(${cluster:-ceph}-$id): bluestore DB ($dev_db) has incorrect permissions, fixing." 1>&2

        chown ceph:ceph $dev_db

    fi

fi

...

thanks,

   Wyllys Ingersoll