octopus upgrade stuck: Assertion `map->require_osd_release >= ceph_release_t::mimic' failed.
25 Mar
2020
25 Mar
'20
11:42 p.m.
Hello List,
i followed:
https://ceph.io/releases/v15-2-0-octopus-released/
I came from a healthy nautilus and i am stuck at:
5.) Upgrade all OSDs by installing the new packages and restarting
the ceph-osd daemons on all OSD host
When i try to start an osd like this, i get:
/usr/bin/ceph-osd -f --cluster ceph --id 32 --setuser ceph --setgroup ceph
...
2020-03-25T20:11:03.292+0100 7f2762874e00 -1 osd.32 57223
log_to_monitors {default=true}
ceph-osd: /build/ceph-15.2.0/src/osd/PeeringState.cc:109: void
PGPool::update(ceph::common::CephContext*, OSDMapRef): Assertion
`map->require_osd_release >= ceph_release_t::mimic' failed.
ceph-osd: /build/ceph-15.2.0/src/osd/PeeringState.cc:109: void
PGPool::update(ceph::common::CephContext*, OSDMapRef): Assertion
`map->require_osd_release >= ceph_release_t::mimic' failed.
*** Caught signal (Aborted) **
in thread 7f274854f700 thread_name:tp_osd_tp
Aborted
My current status:
root@ceph03:~# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 60.70999 root default
-2 20.25140 host ceph01
0 hdd 1.71089 osd.0 up 1.00000 1.00000
8 hdd 2.67029 osd.8 up 1.00000 1.00000
11 hdd 1.59999 osd.11 up 1.00000 1.00000
12 hdd 1.59999 osd.12 up 1.00000 1.00000
14 hdd 2.79999 osd.14 up 1.00000 1.00000
18 hdd 1.59999 osd.18 up 1.00000 1.00000
22 hdd 2.79999 osd.22 up 1.00000 1.00000
23 hdd 2.79999 osd.23 up 1.00000 1.00000
26 hdd 2.67029 osd.26 up 1.00000 1.00000
-3 23.05193 host ceph02
2 hdd 2.67029 osd.2 up 1.00000 1.00000
3 hdd 2.00000 osd.3 up 1.00000 1.00000
7 hdd 2.67029 osd.7 up 1.00000 1.00000
9 hdd 2.67029 osd.9 up 1.00000 1.00000
13 hdd 2.00000 osd.13 up 1.00000 1.00000
16 hdd 1.59999 osd.16 up 1.00000 1.00000
19 hdd 2.38409 osd.19 up 1.00000 1.00000
24 hdd 2.67020 osd.24 up 1.00000 1.00000
25 hdd 1.71649 osd.25 up 1.00000 1.00000
28 hdd 2.67029 osd.28 up 1.00000 1.00000
-4 17.40666 host ceph03
5 hdd 1.71660 osd.5 down 1.00000 1.00000
6 hdd 1.71660 osd.6 down 1.00000 1.00000
10 hdd 2.67029 osd.10 down 1.00000 1.00000
15 hdd 2.00000 osd.15 down 1.00000 1.00000
17 hdd 1.20000 osd.17 down 1.00000 1.00000
20 hdd 1.71649 osd.20 down 1.00000 1.00000
21 hdd 2.00000 osd.21 down 1.00000 1.00000
27 hdd 1.71649 osd.27 down 1.00000 1.00000
32 hdd 2.67020 osd.32 down 1.00000 1.00000
root@ceph03:~# ceph osd dump | grep require_osd_release
require_osd_release nautilus
root@ceph03:~# ceph osd versions
{
"ceph version 14.2.8 (88c3b82e8bc76d3444c2d84a30c4a380d6169d46)
nautilus (stable)": 19
}
oot@ceph03:~# ceph mon dump | grep min_mon_release
dumped monmap epoch 12
min_mon_release 15 (octopus)
ceph versions
{
"mon": {
"ceph version 15.2.0
(dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3
},
"mgr": {
"ceph version 15.2.0
(dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3
},
"osd": {
"ceph version 14.2.8
(88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19
},
"mds": {
"ceph version 15.2.0
(dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3
},
"overall": {
"ceph version 14.2.8
(88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19,
"ceph version 15.2.0
(dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 9
}
}
Why does it complain about map->require_osd_release >= ceph_release_t::mimic ?
Cheers,
Michael
26 Mar
26 Mar
1:45 a.m.
in the logs it says:
2020-03-25T22:10:00.823+0100 7f0bd5320e00 0 <cls>
/build/ceph-15.2.0/src/cls/hello/cls_hello.cc:312: loading cls_hello
2020-03-25T22:10:00.823+0100 7f0bd5320e00 0 osd.32 57223 crush map
has features 288232576282525696, adjusting msgr requires for clients
2020-03-25T22:10:00.823+0100 7f0bd5320e00 0 osd.32 57223 crush map
has features 288232576282525696 was 8705, adjusting msgr requires for
mons
2020-03-25T22:10:00.823+0100 7f0bd5320e00 0 osd.32 57223 crush map
has features 1008808516661821440, adjusting msgr requires for osds
2020-03-25T22:10:00.823+0100 7f0bd5320e00 1 osd.32 57223
check_osdmap_features require_osd_release unknown -> luminous
2020-03-25T22:10:04.695+0100 7f0bd5320e00 0 osd.32 57223 load_pgs
2020-03-25T22:10:10.907+0100 7f0bcc01d700 4 rocksdb:
[db/compaction_job.cc:1332] [default] [JOB 3] Generated table #59886:
2107241 keys, 72886355 bytes
2020-03-25T22:10:10.907+0100 7f0bcc01d700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1585170610911598, "cf_name": "default",
"job": 3,
"event": "table_file_creation", "file_number": 59886,
"file_size":
72886355, "table_properties": {"data_size": 67112666,
"index_size":
504659, "filter_size": 5268165, "raw_key_size": 38673953,
"raw_average_key_size": 18, "raw_value_size": 35746098,
"raw_average_value_size": 16, "num_data_blocks": 16488,
"num_entries":
2107241, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
2020-03-25T22:10:13.047+0100 7f0bd5320e00 0 osd.32 57223 load_pgs
opened 230 pgs
2020-03-25T22:10:13.047+0100 7f0bd5320e00 -1 osd.32 57223
log_to_monitors {default=true}
2020-03-25T22:10:13.107+0100 7f0bd5320e00 0 osd.32 57223 done with
init, starting boot process
2020-03-25T22:10:13.107+0100 7f0bd5320e00 1 osd.32 57223 start_boot
does the line:
check_osdmap_features require_osd_release unknown -> luminous
mean it thinks the local osd itself is luminous?
On Wed, Mar 25, 2020 at 8:12 PM Ml Ml <mliebherr99(a)googlemail.com> wrote:
>
> Hello List,
>
> i followed:
> https://ceph.io/releases/v15-2-0-octopus-released/
>
> I came from a healthy nautilus and i am stuck at:
> 5.) Upgrade all OSDs by installing the new packages and restarting
> the ceph-osd daemons on all OSD host
>
> When i try to start an osd like this, i get:
> /usr/bin/ceph-osd -f --cluster ceph --id 32 --setuser ceph --setgroup ceph
> ...
> 2020-03-25T20:11:03.292+0100 7f2762874e00 -1 osd.32 57223
> log_to_monitors {default=true}
> ceph-osd: /build/ceph-15.2.0/src/osd/PeeringState.cc:109: void
> PGPool::update(ceph::common::CephContext*, OSDMapRef): Assertion
> `map->require_osd_release >= ceph_release_t::mimic' failed.
> ceph-osd: /build/ceph-15.2.0/src/osd/PeeringState.cc:109: void
> PGPool::update(ceph::common::CephContext*, OSDMapRef): Assertion
> `map->require_osd_release >= ceph_release_t::mimic' failed.
> *** Caught signal (Aborted) **
> in thread 7f274854f700 thread_name:tp_osd_tp
> Aborted
>
>
>
> My current status:
>
> root@ceph03:~# ceph osd tree
> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
> -1 60.70999 root default
> -2 20.25140 host ceph01
> 0 hdd 1.71089 osd.0 up 1.00000 1.00000
> 8 hdd 2.67029 osd.8 up 1.00000 1.00000
> 11 hdd 1.59999 osd.11 up 1.00000 1.00000
> 12 hdd 1.59999 osd.12 up 1.00000 1.00000
> 14 hdd 2.79999 osd.14 up 1.00000 1.00000
> 18 hdd 1.59999 osd.18 up 1.00000 1.00000
> 22 hdd 2.79999 osd.22 up 1.00000 1.00000
> 23 hdd 2.79999 osd.23 up 1.00000 1.00000
> 26 hdd 2.67029 osd.26 up 1.00000 1.00000
> -3 23.05193 host ceph02
> 2 hdd 2.67029 osd.2 up 1.00000 1.00000
> 3 hdd 2.00000 osd.3 up 1.00000 1.00000
> 7 hdd 2.67029 osd.7 up 1.00000 1.00000
> 9 hdd 2.67029 osd.9 up 1.00000 1.00000
> 13 hdd 2.00000 osd.13 up 1.00000 1.00000
> 16 hdd 1.59999 osd.16 up 1.00000 1.00000
> 19 hdd 2.38409 osd.19 up 1.00000 1.00000
> 24 hdd 2.67020 osd.24 up 1.00000 1.00000
> 25 hdd 1.71649 osd.25 up 1.00000 1.00000
> 28 hdd 2.67029 osd.28 up 1.00000 1.00000
> -4 17.40666 host ceph03
> 5 hdd 1.71660 osd.5 down 1.00000 1.00000
> 6 hdd 1.71660 osd.6 down 1.00000 1.00000
> 10 hdd 2.67029 osd.10 down 1.00000 1.00000
> 15 hdd 2.00000 osd.15 down 1.00000 1.00000
> 17 hdd 1.20000 osd.17 down 1.00000 1.00000
> 20 hdd 1.71649 osd.20 down 1.00000 1.00000
> 21 hdd 2.00000 osd.21 down 1.00000 1.00000
> 27 hdd 1.71649 osd.27 down 1.00000 1.00000
> 32 hdd 2.67020 osd.32 down 1.00000 1.00000
>
> root@ceph03:~# ceph osd dump | grep require_osd_release
> require_osd_release nautilus
>
> root@ceph03:~# ceph osd versions
> {
> "ceph version 14.2.8 (88c3b82e8bc76d3444c2d84a30c4a380d6169d46)
> nautilus (stable)": 19
> }
>
> oot@ceph03:~# ceph mon dump | grep min_mon_release
> dumped monmap epoch 12
> min_mon_release 15 (octopus)
>
>
> ceph versions
> {
> "mon": {
> "ceph version 15.2.0
> (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3
> },
> "mgr": {
> "ceph version 15.2.0
> (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3
> },
> "osd": {
> "ceph version 14.2.8
> (88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19
> },
> "mds": {
> "ceph version 15.2.0
> (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3
> },
> "overall": {
> "ceph version 14.2.8
> (88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19,
> "ceph version 15.2.0
> (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 9
> }
> }
>
>
> Why does it complain about map->require_osd_release >= ceph_release_t::mimic ?
>
> Cheers,
> Michael
8:08 p.m.
This is a little beyond my understanding of Ceph, but let me take a crack at it...
I've found that Ceph tends to be fairly logical, mostly.
require_osd_release looks like a cluster wide configuration value which controls the
minimum required version for an OSD daemon to join the cluster.
check_osdmap_features looks like an upgrade validation which checks that the above minimum
version is.
unknown -> luminous suggests that a) require_osd_release is not set in your cluster,
and b) because it is unset it is being assumed to be luminous.
This looks like a sane way to check, before upgrading, if the upgrade is likely to
complete correctly.
From your previous email "`map->require_osd_release >=
ceph_release_t::mimic' failed." suggests that the upgrade is looking for the
above configuration value needs to be mimic or higher.
I would suggest that you set this configuration value to mimic or nautilus, as appropriate
for your cluster, and retry the upgrade.
Thank you,
Dominic L. Hilsbos, MBA
Director - Information Technology
Perform Air International Inc.
DHilsbos(a)PerformAir.com
www.PerformAir.com
-----Original Message-----
From: Ml Ml [mailto:mliebherr99@googlemail.com]
Sent: Wednesday, March 25, 2020 2:16 PM
To: ceph-users
Subject: [ceph-users] Re: octopus upgrade stuck: Assertion `map->require_osd_release
>= ceph_release_t::mimic' failed.
in the logs it says:
2020-03-25T22:10:00.823+0100 7f0bd5320e00 0 <cls>
/build/ceph-15.2.0/src/cls/hello/cls_hello.cc:312: loading cls_hello
2020-03-25T22:10:00.823+0100 7f0bd5320e00 0 osd.32 57223 crush map
has features 288232576282525696, adjusting msgr requires for clients
2020-03-25T22:10:00.823+0100 7f0bd5320e00 0 osd.32 57223 crush map
has features 288232576282525696 was 8705, adjusting msgr requires for
mons
2020-03-25T22:10:00.823+0100 7f0bd5320e00 0 osd.32 57223 crush map
has features 1008808516661821440, adjusting msgr requires for osds
2020-03-25T22:10:00.823+0100 7f0bd5320e00 1 osd.32 57223
check_osdmap_features require_osd_release unknown -> luminous
2020-03-25T22:10:04.695+0100 7f0bd5320e00 0 osd.32 57223 load_pgs
2020-03-25T22:10:10.907+0100 7f0bcc01d700 4 rocksdb:
[db/compaction_job.cc:1332] [default] [JOB 3] Generated table #59886:
2107241 keys, 72886355 bytes
2020-03-25T22:10:10.907+0100 7f0bcc01d700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1585170610911598, "cf_name": "default",
"job": 3,
"event": "table_file_creation", "file_number": 59886,
"file_size":
72886355, "table_properties": {"data_size": 67112666,
"index_size":
504659, "filter_size": 5268165, "raw_key_size": 38673953,
"raw_average_key_size": 18, "raw_value_size": 35746098,
"raw_average_value_size": 16, "num_data_blocks": 16488,
"num_entries":
2107241, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
2020-03-25T22:10:13.047+0100 7f0bd5320e00 0 osd.32 57223 load_pgs
opened 230 pgs
2020-03-25T22:10:13.047+0100 7f0bd5320e00 -1 osd.32 57223
log_to_monitors {default=true}
2020-03-25T22:10:13.107+0100 7f0bd5320e00 0 osd.32 57223 done with
init, starting boot process
2020-03-25T22:10:13.107+0100 7f0bd5320e00 1 osd.32 57223 start_boot
does the line:
check_osdmap_features require_osd_release unknown -> luminous
mean it thinks the local osd itself is luminous?
On Wed, Mar 25, 2020 at 8:12 PM Ml Ml <mliebherr99(a)googlemail.com> wrote:
Hello List,
i followed:
https://ceph.io/releases/v15-2-0-octopus-released/
I came from a healthy nautilus and i am stuck at:
5.) Upgrade all OSDs by installing the new packages and restarting
the ceph-osd daemons on all OSD host
When i try to start an osd like this, i get:
/usr/bin/ceph-osd -f --cluster ceph --id 32 --setuser ceph --setgroup ceph
...
2020-03-25T20:11:03.292+0100 7f2762874e00 -1 osd.32 57223
log_to_monitors {default=true}
ceph-osd: /build/ceph-15.2.0/src/osd/PeeringState.cc:109: void
PGPool::update(ceph::common::CephContext*, OSDMapRef): Assertion
`map->require_osd_release >= ceph_release_t::mimic' failed.
ceph-osd: /build/ceph-15.2.0/src/osd/PeeringState.cc:109: void
PGPool::update(ceph::common::CephContext*, OSDMapRef): Assertion
`map->require_osd_release >= ceph_release_t::mimic' failed.
*** Caught signal (Aborted) **
in thread 7f274854f700 thread_name:tp_osd_tp
Aborted
My current status:
root@ceph03:~# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 60.70999 root default
-2 20.25140 host ceph01
0 hdd 1.71089 osd.0 up 1.00000 1.00000
8 hdd 2.67029 osd.8 up 1.00000 1.00000
11 hdd 1.59999 osd.11 up 1.00000 1.00000
12 hdd 1.59999 osd.12 up 1.00000 1.00000
14 hdd 2.79999 osd.14 up 1.00000 1.00000
18 hdd 1.59999 osd.18 up 1.00000 1.00000
22 hdd 2.79999 osd.22 up 1.00000 1.00000
23 hdd 2.79999 osd.23 up 1.00000 1.00000
26 hdd 2.67029 osd.26 up 1.00000 1.00000
-3 23.05193 host ceph02
2 hdd 2.67029 osd.2 up 1.00000 1.00000
3 hdd 2.00000 osd.3 up 1.00000 1.00000
7 hdd 2.67029 osd.7 up 1.00000 1.00000
9 hdd 2.67029 osd.9 up 1.00000 1.00000
13 hdd 2.00000 osd.13 up 1.00000 1.00000
16 hdd 1.59999 osd.16 up 1.00000 1.00000
19 hdd 2.38409 osd.19 up 1.00000 1.00000
24 hdd 2.67020 osd.24 up 1.00000 1.00000
25 hdd 1.71649 osd.25 up 1.00000 1.00000
28 hdd 2.67029 osd.28 up 1.00000 1.00000
-4 17.40666 host ceph03
5 hdd 1.71660 osd.5 down 1.00000 1.00000
6 hdd 1.71660 osd.6 down 1.00000 1.00000
10 hdd 2.67029 osd.10 down 1.00000 1.00000
15 hdd 2.00000 osd.15 down 1.00000 1.00000
17 hdd 1.20000 osd.17 down 1.00000 1.00000
20 hdd 1.71649 osd.20 down 1.00000 1.00000
21 hdd 2.00000 osd.21 down 1.00000 1.00000
27 hdd 1.71649 osd.27 down 1.00000 1.00000
32 hdd 2.67020 osd.32 down 1.00000 1.00000
root@ceph03:~# ceph osd dump | grep require_osd_release
require_osd_release nautilus
root@ceph03:~# ceph osd versions
{
"ceph version 14.2.8 (88c3b82e8bc76d3444c2d84a30c4a380d6169d46)
nautilus (stable)": 19
}
oot@ceph03:~# ceph mon dump | grep min_mon_release
dumped monmap epoch 12
min_mon_release 15 (octopus)
ceph versions
{
"mon": {
"ceph version 15.2.0
(dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3
},
"mgr": {
"ceph version 15.2.0
(dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3
},
"osd": {
"ceph version 14.2.8
(88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19
},
"mds": {
"ceph version 15.2.0
(dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3
},
"overall": {
"ceph version 14.2.8
(88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19,
"ceph version 15.2.0
(dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 9
}
}
Why does it complain about map->require_osd_release >= ceph_release_t::mimic ?
Cheers,
Michael
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1513
days inactive
1514
days old
2 comments
2 participants
participants (2)
-
DHilsbos＠performair.com -
Ml Ml