octopus upgrade stuck: Assertion `map->require_osd_release >= ceph_release_t::mimic' failed.
Hello List, i followed: https://ceph.io/releases/v15-2-0-octopus-released/ I came from a healthy nautilus and i am stuck at: 5.) Upgrade all OSDs by installing the new packages and restarting the ceph-osd daemons on all OSD host When i try to start an osd like this, i get: /usr/bin/ceph-osd -f --cluster ceph --id 32 --setuser ceph --setgroup ceph ... 2020-03-25T20:11:03.292+0100 7f2762874e00 -1 osd.32 57223 log_to_monitors {default=true} ceph-osd: /build/ceph-15.2.0/src/osd/PeeringState.cc:109: void PGPool::update(ceph::common::CephContext*, OSDMapRef): Assertion `map->require_osd_release >= ceph_release_t::mimic' failed. ceph-osd: /build/ceph-15.2.0/src/osd/PeeringState.cc:109: void PGPool::update(ceph::common::CephContext*, OSDMapRef): Assertion `map->require_osd_release >= ceph_release_t::mimic' failed. *** Caught signal (Aborted) ** in thread 7f274854f700 thread_name:tp_osd_tp Aborted My current status: root@ceph03:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 60.70999 root default -2 20.25140 host ceph01 0 hdd 1.71089 osd.0 up 1.00000 1.00000 8 hdd 2.67029 osd.8 up 1.00000 1.00000 11 hdd 1.59999 osd.11 up 1.00000 1.00000 12 hdd 1.59999 osd.12 up 1.00000 1.00000 14 hdd 2.79999 osd.14 up 1.00000 1.00000 18 hdd 1.59999 osd.18 up 1.00000 1.00000 22 hdd 2.79999 osd.22 up 1.00000 1.00000 23 hdd 2.79999 osd.23 up 1.00000 1.00000 26 hdd 2.67029 osd.26 up 1.00000 1.00000 -3 23.05193 host ceph02 2 hdd 2.67029 osd.2 up 1.00000 1.00000 3 hdd 2.00000 osd.3 up 1.00000 1.00000 7 hdd 2.67029 osd.7 up 1.00000 1.00000 9 hdd 2.67029 osd.9 up 1.00000 1.00000 13 hdd 2.00000 osd.13 up 1.00000 1.00000 16 hdd 1.59999 osd.16 up 1.00000 1.00000 19 hdd 2.38409 osd.19 up 1.00000 1.00000 24 hdd 2.67020 osd.24 up 1.00000 1.00000 25 hdd 1.71649 osd.25 up 1.00000 1.00000 28 hdd 2.67029 osd.28 up 1.00000 1.00000 -4 17.40666 host ceph03 5 hdd 1.71660 osd.5 down 1.00000 1.00000 6 hdd 1.71660 osd.6 down 1.00000 1.00000 10 hdd 2.67029 osd.10 down 1.00000 1.00000 15 hdd 2.00000 osd.15 down 1.00000 1.00000 17 hdd 1.20000 osd.17 down 1.00000 1.00000 20 hdd 1.71649 osd.20 down 1.00000 1.00000 21 hdd 2.00000 osd.21 down 1.00000 1.00000 27 hdd 1.71649 osd.27 down 1.00000 1.00000 32 hdd 2.67020 osd.32 down 1.00000 1.00000 root@ceph03:~# ceph osd dump | grep require_osd_release require_osd_release nautilus root@ceph03:~# ceph osd versions { "ceph version 14.2.8 (88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19 } oot@ceph03:~# ceph mon dump | grep min_mon_release dumped monmap epoch 12 min_mon_release 15 (octopus) ceph versions { "mon": { "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 }, "mgr": { "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 }, "osd": { "ceph version 14.2.8 (88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19 }, "mds": { "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 }, "overall": { "ceph version 14.2.8 (88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19, "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 9 } } Why does it complain about map->require_osd_release >= ceph_release_t::mimic ? Cheers, Michael
in the logs it says: 2020-03-25T22:10:00.823+0100 7f0bd5320e00 0 <cls> /build/ceph-15.2.0/src/cls/hello/cls_hello.cc:312: loading cls_hello 2020-03-25T22:10:00.823+0100 7f0bd5320e00 0 osd.32 57223 crush map has features 288232576282525696, adjusting msgr requires for clients 2020-03-25T22:10:00.823+0100 7f0bd5320e00 0 osd.32 57223 crush map has features 288232576282525696 was 8705, adjusting msgr requires for mons 2020-03-25T22:10:00.823+0100 7f0bd5320e00 0 osd.32 57223 crush map has features 1008808516661821440, adjusting msgr requires for osds 2020-03-25T22:10:00.823+0100 7f0bd5320e00 1 osd.32 57223 check_osdmap_features require_osd_release unknown -> luminous 2020-03-25T22:10:04.695+0100 7f0bd5320e00 0 osd.32 57223 load_pgs 2020-03-25T22:10:10.907+0100 7f0bcc01d700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 3] Generated table #59886: 2107241 keys, 72886355 bytes 2020-03-25T22:10:10.907+0100 7f0bcc01d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1585170610911598, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 59886, "file_size": 72886355, "table_properties": {"data_size": 67112666, "index_size": 504659, "filter_size": 5268165, "raw_key_size": 38673953, "raw_average_key_size": 18, "raw_value_size": 35746098, "raw_average_value_size": 16, "num_data_blocks": 16488, "num_entries": 2107241, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} 2020-03-25T22:10:13.047+0100 7f0bd5320e00 0 osd.32 57223 load_pgs opened 230 pgs 2020-03-25T22:10:13.047+0100 7f0bd5320e00 -1 osd.32 57223 log_to_monitors {default=true} 2020-03-25T22:10:13.107+0100 7f0bd5320e00 0 osd.32 57223 done with init, starting boot process 2020-03-25T22:10:13.107+0100 7f0bd5320e00 1 osd.32 57223 start_boot does the line: check_osdmap_features require_osd_release unknown -> luminous mean it thinks the local osd itself is luminous? On Wed, Mar 25, 2020 at 8:12 PM Ml Ml <mliebherr99@googlemail.com> wrote:
Hello List,
i followed: https://ceph.io/releases/v15-2-0-octopus-released/
I came from a healthy nautilus and i am stuck at: 5.) Upgrade all OSDs by installing the new packages and restarting the ceph-osd daemons on all OSD host
When i try to start an osd like this, i get: /usr/bin/ceph-osd -f --cluster ceph --id 32 --setuser ceph --setgroup ceph ... 2020-03-25T20:11:03.292+0100 7f2762874e00 -1 osd.32 57223 log_to_monitors {default=true} ceph-osd: /build/ceph-15.2.0/src/osd/PeeringState.cc:109: void PGPool::update(ceph::common::CephContext*, OSDMapRef): Assertion `map->require_osd_release >= ceph_release_t::mimic' failed. ceph-osd: /build/ceph-15.2.0/src/osd/PeeringState.cc:109: void PGPool::update(ceph::common::CephContext*, OSDMapRef): Assertion `map->require_osd_release >= ceph_release_t::mimic' failed. *** Caught signal (Aborted) ** in thread 7f274854f700 thread_name:tp_osd_tp Aborted
My current status:
root@ceph03:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 60.70999 root default -2 20.25140 host ceph01 0 hdd 1.71089 osd.0 up 1.00000 1.00000 8 hdd 2.67029 osd.8 up 1.00000 1.00000 11 hdd 1.59999 osd.11 up 1.00000 1.00000 12 hdd 1.59999 osd.12 up 1.00000 1.00000 14 hdd 2.79999 osd.14 up 1.00000 1.00000 18 hdd 1.59999 osd.18 up 1.00000 1.00000 22 hdd 2.79999 osd.22 up 1.00000 1.00000 23 hdd 2.79999 osd.23 up 1.00000 1.00000 26 hdd 2.67029 osd.26 up 1.00000 1.00000 -3 23.05193 host ceph02 2 hdd 2.67029 osd.2 up 1.00000 1.00000 3 hdd 2.00000 osd.3 up 1.00000 1.00000 7 hdd 2.67029 osd.7 up 1.00000 1.00000 9 hdd 2.67029 osd.9 up 1.00000 1.00000 13 hdd 2.00000 osd.13 up 1.00000 1.00000 16 hdd 1.59999 osd.16 up 1.00000 1.00000 19 hdd 2.38409 osd.19 up 1.00000 1.00000 24 hdd 2.67020 osd.24 up 1.00000 1.00000 25 hdd 1.71649 osd.25 up 1.00000 1.00000 28 hdd 2.67029 osd.28 up 1.00000 1.00000 -4 17.40666 host ceph03 5 hdd 1.71660 osd.5 down 1.00000 1.00000 6 hdd 1.71660 osd.6 down 1.00000 1.00000 10 hdd 2.67029 osd.10 down 1.00000 1.00000 15 hdd 2.00000 osd.15 down 1.00000 1.00000 17 hdd 1.20000 osd.17 down 1.00000 1.00000 20 hdd 1.71649 osd.20 down 1.00000 1.00000 21 hdd 2.00000 osd.21 down 1.00000 1.00000 27 hdd 1.71649 osd.27 down 1.00000 1.00000 32 hdd 2.67020 osd.32 down 1.00000 1.00000
root@ceph03:~# ceph osd dump | grep require_osd_release require_osd_release nautilus
root@ceph03:~# ceph osd versions { "ceph version 14.2.8 (88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19 }
oot@ceph03:~# ceph mon dump | grep min_mon_release dumped monmap epoch 12 min_mon_release 15 (octopus)
ceph versions { "mon": { "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 }, "mgr": { "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 }, "osd": { "ceph version 14.2.8 (88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19 }, "mds": { "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 }, "overall": { "ceph version 14.2.8 (88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19, "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 9 } }
Why does it complain about map->require_osd_release >= ceph_release_t::mimic ?
Cheers, Michael
This is a little beyond my understanding of Ceph, but let me take a crack at it... I've found that Ceph tends to be fairly logical, mostly. require_osd_release looks like a cluster wide configuration value which controls the minimum required version for an OSD daemon to join the cluster. check_osdmap_features looks like an upgrade validation which checks that the above minimum version is. unknown -> luminous suggests that a) require_osd_release is not set in your cluster, and b) because it is unset it is being assumed to be luminous. This looks like a sane way to check, before upgrading, if the upgrade is likely to complete correctly. From your previous email "`map->require_osd_release >= ceph_release_t::mimic' failed." suggests that the upgrade is looking for the above configuration value needs to be mimic or higher. I would suggest that you set this configuration value to mimic or nautilus, as appropriate for your cluster, and retry the upgrade. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Ml Ml [mailto:mliebherr99@googlemail.com] Sent: Wednesday, March 25, 2020 2:16 PM To: ceph-users Subject: [ceph-users] Re: octopus upgrade stuck: Assertion `map->require_osd_release >= ceph_release_t::mimic' failed. in the logs it says: 2020-03-25T22:10:00.823+0100 7f0bd5320e00 0 <cls> /build/ceph-15.2.0/src/cls/hello/cls_hello.cc:312: loading cls_hello 2020-03-25T22:10:00.823+0100 7f0bd5320e00 0 osd.32 57223 crush map has features 288232576282525696, adjusting msgr requires for clients 2020-03-25T22:10:00.823+0100 7f0bd5320e00 0 osd.32 57223 crush map has features 288232576282525696 was 8705, adjusting msgr requires for mons 2020-03-25T22:10:00.823+0100 7f0bd5320e00 0 osd.32 57223 crush map has features 1008808516661821440, adjusting msgr requires for osds 2020-03-25T22:10:00.823+0100 7f0bd5320e00 1 osd.32 57223 check_osdmap_features require_osd_release unknown -> luminous 2020-03-25T22:10:04.695+0100 7f0bd5320e00 0 osd.32 57223 load_pgs 2020-03-25T22:10:10.907+0100 7f0bcc01d700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 3] Generated table #59886: 2107241 keys, 72886355 bytes 2020-03-25T22:10:10.907+0100 7f0bcc01d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1585170610911598, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 59886, "file_size": 72886355, "table_properties": {"data_size": 67112666, "index_size": 504659, "filter_size": 5268165, "raw_key_size": 38673953, "raw_average_key_size": 18, "raw_value_size": 35746098, "raw_average_value_size": 16, "num_data_blocks": 16488, "num_entries": 2107241, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} 2020-03-25T22:10:13.047+0100 7f0bd5320e00 0 osd.32 57223 load_pgs opened 230 pgs 2020-03-25T22:10:13.047+0100 7f0bd5320e00 -1 osd.32 57223 log_to_monitors {default=true} 2020-03-25T22:10:13.107+0100 7f0bd5320e00 0 osd.32 57223 done with init, starting boot process 2020-03-25T22:10:13.107+0100 7f0bd5320e00 1 osd.32 57223 start_boot does the line: check_osdmap_features require_osd_release unknown -> luminous mean it thinks the local osd itself is luminous? On Wed, Mar 25, 2020 at 8:12 PM Ml Ml <mliebherr99@googlemail.com> wrote:
Hello List,
i followed: https://ceph.io/releases/v15-2-0-octopus-released/
I came from a healthy nautilus and i am stuck at: 5.) Upgrade all OSDs by installing the new packages and restarting the ceph-osd daemons on all OSD host
When i try to start an osd like this, i get: /usr/bin/ceph-osd -f --cluster ceph --id 32 --setuser ceph --setgroup ceph ... 2020-03-25T20:11:03.292+0100 7f2762874e00 -1 osd.32 57223 log_to_monitors {default=true} ceph-osd: /build/ceph-15.2.0/src/osd/PeeringState.cc:109: void PGPool::update(ceph::common::CephContext*, OSDMapRef): Assertion `map->require_osd_release >= ceph_release_t::mimic' failed. ceph-osd: /build/ceph-15.2.0/src/osd/PeeringState.cc:109: void PGPool::update(ceph::common::CephContext*, OSDMapRef): Assertion `map->require_osd_release >= ceph_release_t::mimic' failed. *** Caught signal (Aborted) ** in thread 7f274854f700 thread_name:tp_osd_tp Aborted
My current status:
root@ceph03:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 60.70999 root default -2 20.25140 host ceph01 0 hdd 1.71089 osd.0 up 1.00000 1.00000 8 hdd 2.67029 osd.8 up 1.00000 1.00000 11 hdd 1.59999 osd.11 up 1.00000 1.00000 12 hdd 1.59999 osd.12 up 1.00000 1.00000 14 hdd 2.79999 osd.14 up 1.00000 1.00000 18 hdd 1.59999 osd.18 up 1.00000 1.00000 22 hdd 2.79999 osd.22 up 1.00000 1.00000 23 hdd 2.79999 osd.23 up 1.00000 1.00000 26 hdd 2.67029 osd.26 up 1.00000 1.00000 -3 23.05193 host ceph02 2 hdd 2.67029 osd.2 up 1.00000 1.00000 3 hdd 2.00000 osd.3 up 1.00000 1.00000 7 hdd 2.67029 osd.7 up 1.00000 1.00000 9 hdd 2.67029 osd.9 up 1.00000 1.00000 13 hdd 2.00000 osd.13 up 1.00000 1.00000 16 hdd 1.59999 osd.16 up 1.00000 1.00000 19 hdd 2.38409 osd.19 up 1.00000 1.00000 24 hdd 2.67020 osd.24 up 1.00000 1.00000 25 hdd 1.71649 osd.25 up 1.00000 1.00000 28 hdd 2.67029 osd.28 up 1.00000 1.00000 -4 17.40666 host ceph03 5 hdd 1.71660 osd.5 down 1.00000 1.00000 6 hdd 1.71660 osd.6 down 1.00000 1.00000 10 hdd 2.67029 osd.10 down 1.00000 1.00000 15 hdd 2.00000 osd.15 down 1.00000 1.00000 17 hdd 1.20000 osd.17 down 1.00000 1.00000 20 hdd 1.71649 osd.20 down 1.00000 1.00000 21 hdd 2.00000 osd.21 down 1.00000 1.00000 27 hdd 1.71649 osd.27 down 1.00000 1.00000 32 hdd 2.67020 osd.32 down 1.00000 1.00000
root@ceph03:~# ceph osd dump | grep require_osd_release require_osd_release nautilus
root@ceph03:~# ceph osd versions { "ceph version 14.2.8 (88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19 }
oot@ceph03:~# ceph mon dump | grep min_mon_release dumped monmap epoch 12 min_mon_release 15 (octopus)
ceph versions { "mon": { "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 }, "mgr": { "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 }, "osd": { "ceph version 14.2.8 (88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19 }, "mds": { "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3 }, "overall": { "ceph version 14.2.8 (88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19, "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 9 } }
Why does it complain about map->require_osd_release >= ceph_release_t::mimic ?
Cheers, Michael
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
DHilsbos＠performair.com
-
Ml Ml