Hello Marco, On Thu, Aug 29, 2019 at 12:55:56PM +0200, Marco Gaiarin wrote:

I've just finished a double upgrade on my ceph (PVE-based) from hammer to jewel and from jewel to luminous.

All went well, apart that... OSD does not restart automatically, because permission troubles on the journal:

Aug 28 14:41:55 capitanmarvel ceph-osd[6645]: starting osd.2 at - osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal Aug 28 14:41:55 capitanmarvel ceph-osd[6645]: 2019-08-28 14:41:55.449886 7fa505a43e00 -1 filestore(/var/lib/ceph/osd/ceph-2) mount(1822): failed to open journal /var/lib/ceph/osd/ceph-2/journal: (13) Permission denied Aug 28 14:41:55 capitanmarvel ceph-osd[6645]: 2019-08-28 14:41:55.453524 7fa505a43e00 -1 osd.2 0 OSD:init: unable to mount object store Aug 28 14:41:55 capitanmarvel ceph-osd[6645]: 2019-08-28 14:41:55.453535 7fa505a43e00 -1 #033[0;31m ** ERROR: osd init failed: (13) Permission denied#033[0m

A little fast rewind: when i've setup the cluster i've used some 'old' servers, using a couple of SSD disks as SO and as journal. Because servers was old, i was forced to partition the boot disk in DOS, not GPT mode.

While creating the OSD, i've received some warnings:

WARNING:ceph-disk:Journal /dev/sdaX was not prepared with ceph-disk. Symlinking directly.

Looking at the cluster now, seems to me that osd init scripts try to idetify journal based on GPT partition label/info, and clearly fail.

Not that if i do, on servers that hold OSD:

for l in $(readlink -f /var/lib/ceph/osd/ceph-*/journal); do chown ceph: $l; done

OSD start flawlessy.

There's something i can do? Thanks.

Did you go through our upgrade guide(s)? See the link [0] below, for the permission changes. They are needed when an upgrade from Hammer to Jewel is done. On the wiki you can also find the upgrade guides for PVE 5.x -> 6.x and Luminous -> Nautilus. -- Cheers, Alwin [0] https://pve.proxmox.com/wiki/Ceph_Hammer_to_Jewel#Set_permission