So, the thing that seems to cause all the problems is not doing: ceph osd require-osd-release nautilus as that is associated with this bug: https://tracker.ceph.com/issues/44184 which can be distressing if you make new pools. I found that ceph-ansible didn't seem to do this for my cluster using the rolling_upgrade playbook, and I'm not sure why... so be careful if automating your upgrade. (If you follow the manual steps, then you'll do that when you get to it anyway, but it's worth mentioning how important it is)