29 Aug
2020
29 Aug
'20
7:20 a.m.
Hi all,
I'm trying to migrate my 5-node cluster to Nautilus… Debian Stretch (I
intend to update this once I'm onto Nautilus) as the OS, all 5 machines
are combined OSD/MON/MDS nodes.
Nodes were originally deployed with an earlier version of Ceph (can't
recall now) using filestore on btrfs. Later, after an abortive attempt
to move to Bluestore (it ran… veeeeerrryyy ssslllooowwwlllyyy), I moved
back to filestore on xfs, which whilst no speed demon, worked well enough.
Journal is on the OSD due to space constraints elsewhere in the nodes.
So step 1:
If you are unsure whether or not your Luminous cluster
has completed a full scrub of all PGs, you can check your cluster’s state by running:
# ceph osd dump | grep ^flags
In order to be able to proceed to Nautilus, your OSD map must include the
recovery_deletes and purged_snapdirs flags.
Tick to that, I see both flags.
Step 2.
Make sure your cluster is stable and healthy (no down
or recovering OSDs). (Optional, but recommended.)
Tick, all healthy.
Step 3.
Set the noout flag for the duration of the upgrade.
(Optional, but recommended.):
Done
Step 4.
Upgrade monitors by installing the new packages and
restarting the monitor daemons.
Okay, so I update /etc/apt/sources.list.d/ceph.list to point to the
`nautilus` repository, `apt-get update`, `apt-get dist-upgrade -y`.
This goes without a hitch, I now have Ceph 12 binaries on my nodes.
systemctl restart ceph-mon.target
Ran that, on all of them (and yes, they are all in the quorum)…
# ceph mon dump | grep min_mon_release
should report:
min_mon_release 14 (nautilus)
I get this:> root@helium:~# ceph mon dump | grep min_mon_release
dumped monmap epoch 4
No `min_mon_release` mentioned anywhere. I tried re-starting the MON
daemons on all 5 nodes, even doing it in parallel using Ansible… no dice.
Nothing says what to do at this point. It's not reporting an earlier
release, it's just not reporting full stop.
Figuring that, well, I *did* update all the monitors, and re-start them,
I press on.
Step 5.
Upgrade ceph-mgr daemons by installing the new
packages and restarting all manager daemons.
Done, no issues.
Verify the ceph-mgr daemons are running by checking
ceph -s:
I get:
root@helium:~# ceph -s
cluster:
id: 45b532b7-aa3d-4754-9906-d4a70b57630c
health: HEALTH_WARN
noout flag(s) set
services:
mon: 5 daemons, quorum hydrogen,helium,carbon,nitrogen,boron
mgr: hydrogen(active), standbys: carbon, helium
mds: cephfs-1/1/1 up {0=helium=up:active}, 2 up:standby
osd: 5 osds: 5 up, 5 in
flags noout
data:
pools: 4 pools, 324 pgs
objects: 302.75k objects, 1.13TiB
usage: 3.52TiB used, 5.53TiB / 9.05TiB avail
pgs: 323 active+clean
1 active+clean+scrubbing+deep
io:
client: 45.0KiB/s rd, 1.10MiB/s wr, 1op/s rd, 38op/s wr
So far so good.
Step 6.
Upgrade all OSDs by installing the new packages and
restarting the ceph-osd daemons on all OSD hosts:
systemctl restart ceph-osd.target
I actually missed this step at first and went on to step 7, but doubled
back and did it.
Step 7.
If there are any OSDs in the cluster deployed with
ceph-disk (e.g., almost any OSDs that were created before the Mimic release), you need to
tell ceph-volume to adopt responsibility for starting the daemons.
Okay, seems simple enough. Now I did this by mistake earlier, so I'll
force it to ensure everything is up to scratch.
root@hydrogen:~# ceph-volume simple scan --force
stderr: lsblk: /var/lib/ceph/osd/ceph-5: not a block device
stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected.
Running command: /sbin/cryptsetup status /dev/sdc1
--> OSD 5 got scanned and metadata persisted to file:
/etc/ceph/osd/5-2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe.json
--> To take over managment of this scanned OSD, and disable ceph-disk and udev, run:
--> ceph-volume simple activate 5 2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe
root@hydrogen:~# ceph-volume simple activate --all
--> activating OSD specified in
/etc/ceph/osd/5-2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe.json
--> Required devices (data, and journal) not present for filestore
--> filestore devices found: [u'data']
--> RuntimeError: Unable to activate filestore OSD due to missing devices
Uhh ohh? So, my journal is on the OSD itself:
root@hydrogen:~# ls -l
/var/lib/ceph/osd/ceph-5/journal
-rw-r--r-- 1 ceph ceph 21474836480 Aug 29 14:16 /var/lib/ceph/osd/ceph-5/journal
root@hydrogen:~# cat
/etc/ceph/osd/5-2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe.json
{
"active": "ok",
"ceph_fsid": "45b532b7-aa3d-4754-9906-d4a70b57630c",
"cluster_name": "ceph",
"data": {
"path": "/dev/sdc1",
"uuid": "2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe"
},
"fsid": "2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe",
"keyring": "…censored…",
"magic": "ceph osd volume v026",
"ready": "ready",
"require_osd_release": "",
"systemd": "",
"type": "filestore",
"whoami": 5
}
How do I tell it that the journal is on `/dev/sdc1` as a file within the
XFS file store?
--
Stuart Longland (aka Redhatter, VK4MSL)
I haven't lost my mind...
...it's backed up on a tape somewhere.