28 Aug
2020
28 Aug
'20
5:08 a.m.
Hi All,
We encountered an issue while upgrading our Ceph cluster from Luminous
12.2.12 to Nautilus 14.2.11. We used
https://docs.ceph.com/docs/master/releases/nautilus/#upgrading-from-mimic-o…
and ceph-ansible to upgrade the cluster. We use HDD for data and NVME for
WAL and DB.
*Cluster Background:*
HP DL360
24 x 3.6T SATA
2x1.6T NVME for Journal
osd_scenario: non-collocated
current version: Luminous 12.2.12 & 12.2.5
type: bluestore
The upgrade went well for MONs (though I had to overcome the systemd
masking issues). While testing OSD upgrade with one OSD node, we
encountered issue with OSD daemon failing quickly after startup. After
comparing and checking the block devices mapping, everything looks fine.
The nodes was up for more than 700+ days and then I decided to do a clean
reboot. After that noticed the mount points are completely missing and
also ceph-disk is no longer part of nautilus. Had to manually mount the
partitions after checking disk partitions and whoami information. After
manually mounting the osd.108, now it's throwing permission error which I'm
still reviewing (bdev(0xd1be000 /var/lib/ceph/osd/ceph-108/block) open open
got: (13) Permission denied). Enclosed the log of the OSD for full review
- https://pastebin.com/7k0xBfDV.
*Questions*:
What could have went wrong here and how can we fix this?
Do we need to migrate the Luminous cluster from ceph-disk to ceph-volume
before attempting upgrade or any other best practice can be followed?
What's the best upgrade method using ceph-ansible to move from Luminous to
Nautilus? Manual upgrade of Ceph-ansible?
Started thinking now Octopus release which uses container, what is the best
transition path for long run? We don't want to destroy and rebuild the
entire cluster but we can do one node at a time but that would be a very
lengthy process for 2500+ systems of 13 clusters. Looking for help and
expert comments on the transition path.
Any help would be greatly appreciated.
2020-08-27 14:41:01.132 7f0e0ebf2c00 0 bdev(0xb7e2a80
/var/lib/ceph/osd/ceph-108/block.wal) ioctl(F_SET_FILE_RW_HINT) on
/var/lib/ceph/osd/ceph-108/block.wal failed: (22) Invalid argument
2020-08-27 14:41:01.132 7f0e0ebf2c00 1 bdev(0xb7e2a80
/var/lib/ceph/osd/ceph-108/block.wal) open size 1073741824 (0x40000000, 1
GiB) block_size 4096 (4 KiB) non-rotational discard supported
2020-08-27 14:41:01.132 7f0e0ebf2c00 1 bluefs add_block_device bdev 0 path
/var/lib/ceph/osd/ceph-108/block.wal size 1 GiB
2020-08-27 14:41:01.132 7f0e0ebf2c00 0 set rocksdb option
compaction_style = kCompactionStyleLevel
2020-08-27 14:41:01.132 7f0e0ebf2c00 -1 rocksdb: Invalid argument: Can't
parse option compaction_threads
2020-08-27 14:41:01.136 7f0e0ebf2c00 -1
/build/ceph-14.2.11/src/os/bluestore/BlueStore.cc: In function 'int
BlueStore::_upgrade_super()' thread 7f0e0ebf2c00 time 2020-08-27
14:41:01.135973
/build/ceph-14.2.11/src/os/bluestore/BlueStore.cc: 10249: FAILED
ceph_assert(ondisk_format > 0)
ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus
(stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x152) [0x846368]
2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*,
char const*, ...)+0) [0x846543]
3: (BlueStore::_upgrade_super()+0x4b6) [0xd62346]
4: (BlueStore::_mount(bool, bool)+0x592) [0xdb0b52]
5: (OSD::init()+0x3f3) [0x8f5483]
6: (main()+0x27e2) [0x84c462]
7: (__libc_start_main()+0xf0) [0x7f0e0bda3830]
8: (_start()+0x29) [0x880389]
Journalctl -xu log
================
Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.309
7fe9410bfc00 -1 bluestore(/var/lib/ceph/osd/ceph-108/block)
_read_bdev_label failed to op
Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.309
7fe9410bfc00 -1 bluestore(/var/lib/ceph/osd/ceph-108/block)
_read_bdev_label failed to op
Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.309
7fe9410bfc00 -1 bluestore(/var/lib/ceph/osd/ceph-108/block)
_read_bdev_label failed to op
Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.309
7fe9410bfc00 -1 bluestore(/var/lib/ceph/osd/ceph-108/block)
_read_bdev_label failed to op
Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.309
7fe9410bfc00 -1 bluestore(/var/lib/ceph/osd/ceph-108/block)
_read_bdev_label failed to op
Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.309
7fe9410bfc00 -1 bdev(0xd1be000 /var/lib/ceph/osd/ceph-108/block) open open
got: (13) Perm
Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.317
7fe9410bfc00 -1 bdev(0xd1be000 /var/lib/ceph/osd/ceph-108/block) open open
got: (13) Perm
Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.317
7fe9410bfc00 -1 bluestore(/var/lib/ceph/osd/ceph-108/block)
_read_bdev_label failed to op
Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.317
7fe9410bfc00 -1 bluestore(/var/lib/ceph/osd/ceph-108/block)
_read_bdev_label failed to op
Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.317
7fe9410bfc00 -1 bdev(0xd1be000 /var/lib/ceph/osd/ceph-108/block) open open
got: (13) Perm
Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.317
7fe9410bfc00 -1 osd.108 0 OSD:init: unable to mount object store
Aug 27 20:18:39 pistore-as-b03 ceph-osd[345903]: 2020-08-27 20:18:39.317
7fe9410bfc00 -1 ** ERROR: osd init failed: (13) Permission denied
Aug 27 20:18:39 pistore-as-b03 systemd[1]: ceph-osd(a)108.service: Main
process exited, code=exited, status=1/FAILURE
Aug 27 20:18:39 pistore-as-b03 systemd[1]: ceph-osd(a)108.service: Unit
entered failed state.
Aug 27 20:18:39 pistore-as-b03 systemd[1]: ceph-osd(a)108.service: Failed
with result 'exit-code'.
Aug 27 20:18:39 pistore-as-b03 systemd[1]: ceph-osd(a)108.service: Service
hold-off time over, scheduling restart.
Aug 27 20:18:39 pistore-as-b03 systemd[1]: Stopped Ceph object storage
daemon osd.108.
Aug 27 20:18:39 pistore-as-b03 systemd[1]: ceph-osd(a)108.service: Start
request repeated too quickly.
Aug 27 20:18:39 pistore-as-b03 systemd[1]: Failed to start Ceph object
storage daemon osd.108.
Aug 27 20:18:40 pistore-as-b03 systemd[1]:
[/lib/systemd/system/ceph-osd@.service:15]
Unknown lvalue 'LockPersonality' in section 'Service'
Aug 27 20:18:40 pistore-as-b03 systemd[1]:
[/lib/systemd/system/ceph-osd@.service:16]
Unknown lvalue 'MemoryDenyWriteExecute' in section 'Service'
Aug 27 20:18:40 pistore-as-b03 systemd[1]:
[/lib/systemd/system/ceph-osd@.service:19]
Unknown lvalue 'ProtectControlGroups' in section 'Service'
Aug 27 20:18:40 pistore-as-b03 systemd[1]:
[/lib/systemd/system/ceph-osd@.service:21]
Unknown lvalue 'ProtectKernelModules' in section 'Service'
Aug 27 20:18:40 pistore-as-b03 systemd[1]:
[/lib/systemd/system/ceph-osd@.service:23]
Unknown lvalue 'ProtectKernelTunables' in section 'Service'
--
Regards,
Suresh
Note: Resent as the size was big and waiting on moderator approval