Migrating Luminous → Nautilus "Required devices (data, and journal) not present for filestore" - ceph-users

29 Aug 2020

Hi all,

I'm trying to migrate my 5-node cluster to Nautilus… Debian Stretch (I
intend to update this once I'm onto Nautilus) as the OS, all 5 machines
are combined OSD/MON/MDS nodes.

Nodes were originally deployed with an earlier version of Ceph (can't
recall now) using filestore on btrfs.  Later, after an abortive attempt
to move to Bluestore (it ran… veeeeerrryyy ssslllooowwwlllyyy), I moved
back to filestore on xfs, which whilst no speed demon, worked well enough.

Journal is on the OSD due to space constraints elsewhere in the nodes.

So step 1:
...
  If you are unsure whether or not your Luminous cluster
has completed a full scrub of all PGs, you can check your cluster’s state by running:

 # ceph osd dump | grep ^flags

 In order to be able to proceed to Nautilus, your OSD map must include the
recovery_deletes and purged_snapdirs flags. 
Tick to that, I see both flags.

Step 2.
...
  Make sure your cluster is stable and healthy (no down
or recovering OSDs). (Optional, but recommended.) 
Tick, all healthy.

Step 3.
...
  Set the noout flag for the duration of the upgrade.
(Optional, but recommended.): Done

Step 4.
...
  Upgrade monitors by installing the new packages and
restarting the monitor daemons.  
Okay, so I update /etc/apt/sources.list.d/ceph.list to point to the
`nautilus` repository, `apt-get update`, `apt-get dist-upgrade -y`.

This goes without a hitch, I now have Ceph 12 binaries on my nodes.

...
  systemctl restart ceph-mon.target 
Ran that, on all of them (and yes, they are all in the quorum)…

...
  # ceph mon dump | grep min_mon_release

 should report:

 min_mon_release 14 (nautilus) 
I get this:> root@helium:~# ceph mon dump | grep min_mon_release
...
  dumped monmap epoch 4 
No `min_mon_release` mentioned anywhere.  I tried re-starting the MON
daemons on all 5 nodes, even doing it in parallel using Ansible… no dice.

Nothing says what to do at this point.  It's not reporting an earlier
release, it's just not reporting full stop.

Figuring that, well, I *did* update all the monitors, and re-start them,
I press on.

Step 5.
...
  Upgrade ceph-mgr daemons by installing the new
packages and restarting all manager daemons. 
Done, no issues.

...
  Verify the ceph-mgr daemons are running by checking
ceph -s: 
I get:
...
  root@helium:~# ceph -s
   cluster:
     id:     45b532b7-aa3d-4754-9906-d4a70b57630c
     health: HEALTH_WARN
             noout flag(s) set

   services:
     mon: 5 daemons, quorum hydrogen,helium,carbon,nitrogen,boron
     mgr: hydrogen(active), standbys: carbon, helium
     mds: cephfs-1/1/1 up  {0=helium=up:active}, 2 up:standby
     osd: 5 osds: 5 up, 5 in
          flags noout

   data:
     pools:   4 pools, 324 pgs
     objects: 302.75k objects, 1.13TiB
     usage:   3.52TiB used, 5.53TiB / 9.05TiB avail
     pgs:     323 active+clean
              1   active+clean+scrubbing+deep

   io:
     client:   45.0KiB/s rd, 1.10MiB/s wr, 1op/s rd, 38op/s wr 
So far so good.

Step 6.
...
  Upgrade all OSDs by installing the new packages and
restarting the ceph-osd daemons on all OSD hosts:
 systemctl restart ceph-osd.target 
I actually missed this step at first and went on to step 7, but doubled
back and did it.

Step 7.
...
  If there are any OSDs in the cluster deployed with
ceph-disk (e.g., almost any OSDs that were created before the Mimic release), you need to
tell ceph-volume to adopt responsibility for starting the daemons. 
Okay, seems simple enough.  Now I did this by mistake earlier, so I'll
force it to ensure everything is up to scratch.

...
  root@hydrogen:~# ceph-volume simple scan --force
  stderr: lsblk: /var/lib/ceph/osd/ceph-5: not a block device
  stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected.
 Running command: /sbin/cryptsetup status /dev/sdc1
 --> OSD 5 got scanned and metadata persisted to file:
/etc/ceph/osd/5-2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe.json
 --> To take over managment of this scanned OSD, and disable ceph-disk and udev, run:
 -->     ceph-volume simple activate 5 2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe
 root@hydrogen:~# ceph-volume simple activate --all 
 --> activating OSD specified in
/etc/ceph/osd/5-2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe.json
 --> Required devices (data, and journal) not present for filestore
 --> filestore devices found: [u'data']
 -->  RuntimeError: Unable to activate filestore OSD due to missing devices 
Uhh ohh?  So, my journal is on the OSD itself:
...
  root@hydrogen:~# ls -l
/var/lib/ceph/osd/ceph-5/journal 
 -rw-r--r-- 1 ceph ceph 21474836480 Aug 29 14:16 /var/lib/ceph/osd/ceph-5/journal 
...
  root@hydrogen:~# cat
/etc/ceph/osd/5-2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe.json
 {
     "active": "ok", 
     "ceph_fsid": "45b532b7-aa3d-4754-9906-d4a70b57630c", 
     "cluster_name": "ceph", 
     "data": {
         "path": "/dev/sdc1", 
         "uuid": "2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe"
     }, 
     "fsid": "2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe", 
     "keyring": "…censored…", 
     "magic": "ceph osd volume v026", 
     "ready": "ready", 
     "require_osd_release": "", 
     "systemd": "", 
     "type": "filestore", 
     "whoami": 5
 } 
How do I tell it that the journal is on `/dev/sdc1` as a file within the
XFS file store?
-- 
Stuart Longland (aka Redhatter, VK4MSL)

I haven't lost my mind...
  ...it's backed up on a tape somewhere.